Moving to "paused" as we wait for feedback from research.

@Nuria let's go over this at our next 1:1 since it goes beyond Erik's involvement (wikistats transition) and we need to scope it, it's unclear at this stage if we'll be able to help in the coming weeks.

I learned from @Neil_P._Quinn_WMF yesterday that the data lake doesn't know about redirects. If indeed that is the case, I'm curious: how do we discern countable pages (aka articles) from all the rest in so called 'content' namespaces? (In dumps there is the redirect tag, which isn't so easy to set, as #REDIRECT tag can be localized into many language versions). Thx

The data lake knows about the redirect status of a page today, the page_is_redirect field. It doesn't know how page_is_redirect changed over time because we're not yet parsing wiki text. When we do that, we'll add that field and until then we'll use page_is_redirect_latest as a hopefully decent substitute.

@Milimetric Ah OK, that's already as good as the current solution. The dumps don't contain redirect on a per revision level, only the current status. Thanks

pinging @Neil_P._Quinn_WMF so he is aware that indeed redirects are taken care of, maybe this deserves an entry in some kind of FAQ?

What were the plans for looking at this? We have some new numbers from the public labs import that are just slightly off from our production numbers. So it's even more interesting to look at now :)

Research is short on resources this quarter, thus we were planning on tackling this end of quarter on early next quarter. cc @DarTar @leila

Ping @ezachte for after wikimanina is done

Ping @ezachte hello?

Hey @Nuria I spoke to Erik last Friday and he told me he was going to pick this up this week. I'll drop him an email if for whatever reason he missed these pings.

Excellent, let's sync up on this once you are back, Erik.

@Nuria Ah I missed these. Sorry about that. A good example of why restructuring my mailbox was dearly needed, so I finally fine-tuned Gmail filters this weekend. So I could start vetting later this week. Shall we do a hangout?

Will set up meeting

Please do. Tomorrow any time till 12 AM PDT works for me. Preferably a bit earlier.

Three vettings:

• import from mediawiki into hadoop, analytics vetted that fairly well
• metric calculation on metrics calculated on top of data lake versus wikistats metrics
• vetting usage of raw data in its current form by community as our goal is to put this data in labs for easy community access (is structure understandable? what would be helpful examples of usage?). The goal is to have a low bar to have questions answered, community members shouldn't have to know alot about the inners of how mediawiki works in order to answer questions about edits on a given project.

Some links:
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits

I collected feedback in https://phabricator.wikimedia.org/T178591 (I don't know how to link it here as a subtask, I never did such)

@Erik_Zachte I just added it as a subtask. For the future, you can click on Edit Related Task on the top right-hand-side of your page and then click on Edit Subtasks and add the phabricator ticket number to be added as a subtask. If the subtask didn't exit, you would click on Edit Related Tasks and then Create Subtask. Please ping me off-thread if you'd like to indulge in more phabricator tips and tricks. ;)