In T191081#4197687, @matthiasmullie wrote:

I added a whole lot of tracking to UW, for pretty much every element that can be interacted with (full list below)
All of these would go into the UploadWizardFlowEvent schema, where only the flow id, flow position & event name are logged.
This should be enough to figure out *where* people drop out.

I'll wait for further input from @Tbayer before pushing to get this code-reviewed, in case we want to change things (log more context, ...)

Another question: with ~85000 monthly UploadWizard uses, logging every action will generate an enormous amount of data.
We should probably only log a fraction; what would a good sampling ratio be? 10%? 5%?

In T203498#4571941, @Neil_P._Quinn_WMF wrote:

@mpopov, I'm actually confused now. I'm looking at Hive downloads page, which has the best version history I could find, and neither Hive 1.3 nor 1.13 seems to exist. Are you sure you put the right version? 🤔

Sorry, I haven't checked my Phabricator emails in a while! Thanks so much @Ottomata! The upgrade has fixed the chart that wasn't working and has revealed that there's an issue with the data:

In T203498#4565957, @elukey wrote:

Hi! Thanks a lot for this task, it triggered some useful discussions. A couple of notes after checking the CDH release details:

On a more general note, we are currently thinking if it would be worth to change distribution and move away from Cloudera (more details will appear in the parent task, T203693), but it would of course require a ton of time :)

After this looong post, I just want to say that we support this request and that we'll try to do everything that we can to upgrade asap, but it might require a couple (or more) quarters before we'll be able to hit it.

Whoops, realized I was missing a digit in the version.

Dmitry and I have gone over the schemas I proposed and he gave them a thumbs up for instrumentation:

I updated the existing ToC interactions schema for the redesigned ToC: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppToCInteraction

We'll discuss with Josh and Charlotte (once she's back from vacation)

@Nuria: if we go with the first method (which is the one @chelsyx and I have a preference for), what do you think should happen to the global query currently in production?

In T186828#4505784, @chelsyx wrote:

1, Any devices that send requests and have a uuid, regardless of intentional requests or unintentional request (e.g. when the app is in the background, request could be send to fetch the feed and the reading list)

WHERE access_method = 'mobile app'
AND COALESCE(x_analytics_map['wmfuuid'],
          parse_url(concat('http://bla.org/woo/', uri_query), 'QUERY', 'appInstallID')) IS NOT NULL
AND webrequest_source IN ('text')

I prefer the first one, but I agree that we need to compare with historical data sometimes. What do you think if we keep both 1 and 4 in two tables?

Motivation (beyond that it's just nice to have the latest and greatest): I'm trying to add a filter to a slice (which usually works) but when the filter is added, the slice goes from working totally fine to unorderable types: str() < int():

@Charlotte: is this still relevant or can we close it?

Proof of concept dashboard up using the new test_gsc_all datasource in Druid: https://superset.wikimedia.org/superset/dashboard/wikipediagoogledemo/

We did this.

Done :)

Done

Correct, this is to add that to the webrequest class in refinery core that tagging relies on.

Current plan is for me to work on this after I finish T172581 in July, and will most likely be able to begin in August.

Thank you!

In T196761#4296594, @Charlotte wrote:

@mpopov - Did the user ever get back to you on the ticket? If not let's close. I think @Dbrant has correctly pointed out the issue.

In T197896#4313361, @Ottomata wrote:

oauth2client and oauthlib were easy because they already have .deb packages in Debian. We'll have to make a .deb package for google-auth.

In T197897#4308523, @mforns wrote:

Hey @mpopov, sure I can try.

I don't know the details of the data, but both scripts look good to me!

In T197897#4306395, @mforns wrote:

Did you mean to tag this task with Product-Analytics or with Analytics?

There haven't been any updates to this in over a year and it's not entirely clear what needs to be done, nor how my team should be involved (if at all).

In T190767#4092654, @Ottomata wrote:

Done!

Yay! Everything is working correctly now. We ran into a problem with file ownership but the ever-excellent @Ottomata fixed that up and we are now fully puppetized.

Everything is all good now! Sorry for the interruption.

Okay, theoretically things should start working again tomorrow.

Sorry about that! These datasets and others are currently not updated because T170494 did not go as smoothly as I had hoped. Andrew and I are on it.

Excellent work, @Sharvaniharan!!!

According to T194424#4298147 this is resolved by 440969

Disabled the cron job in my account's crontab on stat1005. Will check tomorrow if everything works as it should and the metric calculation by the analytics-search (system) user goes without problems then this task will be done.

In T190931#4276351, @Sharvaniharan wrote:

• (B) Or send just one event with multilingual data. Since enabledList and orderList can be arbitrary strings, they could be JSON representations with the lists of values for each language. For example, orderList could have the value of {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]}. Then languages would be the JSON array of languages in the order they're set – e.g. ["de","en"] (notice that is where we have to make sure to have the correct order, the array in the enabledList & orderList fields can have any order) -- I would prefer if the instrumentation implemented this approach.

Currently, the enabledList & orderList are just plain strings. Do you need them to be valid json? in which case, should it be a string:string hashmap looking like {"en":[0,1,2,3,4,5,6,7,8],"de":[8,7,6,5,4,3,2,1,0]} , or JSONArray of JSONArrays, looking like this {"en":["0", "1" , "2" , "3" , "4" , "5" , "6" , "7" , "8" ] , "de":[ "8" , "7" , "6" , "5" , "4" , "3" , "2" , "1" , "0"]}?

1. Funnels part of multilingual analytics

In T190931#4273933, @Sharvaniharan wrote:

@mpopov For the cards that do not have different languages, like the FeaturedImageCard, should it be null?

@Sharvaniharan: I just remembered the analytics for the feed! I've updated the MobileWikiAppFeed schema (Revision 18115458) to include a language field, which should contain a language code for action = 'cardShown' and action = 'cardClicked' events. Also ,per T196783, the following field names have been renamed:

@Bstorm: thank you very much!!!

In T196783#4268407, @Sharvaniharan wrote:

@mpopov also the timeSpent property please. Shoudl it be 'duration' from now on?

In T196668#4267637, @Bstorm wrote:

Good news! Asking for more memory from gridengine did succeed in not getting a memory error so far (got past the whole JSON deserialization).

I wonder if we can make it so that a user can export and then share over email/SMS so when someone else opens it it goes to the app and there's a prompt to import the list. Or maybe this is the start of Shared/Subscribe-able Reading Lists discussion???

What do you think, @RHo @Charlotte?

In T190931#4266665, @Sharvaniharan wrote:

1. Is one delete one interaction count or one count per language deleted

@Sharvaniharan: the schemas have been updated according to recommendations above. Please verify that the schemas and the instrumentation agree.