@EBernhardson, actually this is quite close to being done! If you look at the wiki segmentation spreadsheet, the database code is in column Z and the language name in English is in column AD. It doesn't include private wikis at all, and the data past row 663 is messed up (T199266), but it wouldn't be that much work to fix it up.

As Kate said, closing this because this parent task doesn't have any value. I've separated out the reverts issue as T216297.

@kzimmerman based on my discussions with Analytics, I decided that I should move my calculations away the slowly dying dbstore1002 immediately rather than working around the locking of the staging database (T215589#4958363). That means I'll need to do this before we can run the February metrics.

In T215589#4957707, @Nuria wrote:

Per irc conversation we are good to proceed with migration Monday the 18th. As @JAllemandou mentioned the next scooping should include the change tag table even if the change tag data is not yet in the denormalized tables.

It's not clear that porting these queries to reportupdater would be worth the effort, so we have no particular plans to actually do this right now.

In T202148#4955096, @Neil_P._Quinn_WMF wrote:

@DLynch, if you look at the full list of logged features we already have a table, window-open event being logged. Will this be different from that?

In T202148#4954914, @gerritbot wrote:

Change 490661 had a related patch set uploaded (by DLynch; owner: DLynch):
[VisualEditor/VisualEditor@master] Track feature-use activity for tables

https://gerrit.wikimedia.org/r/490661

In T215589#4946535, @elukey wrote:

Message to everybody:

Analytics and the Data Persistence team are planning to schedule the official cut off date for the staging database on dbstore1002 for Monday 18th (US holiday) during the EU morning. This is the plan:

• set the staging database as read only on dbstore1002
• dump the database and import it to dbstore1005 (takes hours to complete)
• set dbstore1005 as read/write and official supported staging database
  
  Important considerations:
• dbstore1002 will not be decommissioned or stopped during this maintenance window, all the database/tables will be available. The only difference will be that the staging database will be read-only.
• we are doing this to allow people to migrate their (stateful) scripts to the new system and avoid loosing data. Basically we want to have a clean date/time of the last snapshot taken for staging.
  
  Please let us know if this is too soon or impossible to coordinate/sustain, we'll find another timing. Going also to send an email to people to alert about this change.

Thanks to @DLynch's bugfixes, we are now properly collecting mobile feature use data (P8074). We will have 2 weeks of data after 22 February, so at that point I will be able to calculate the corresponding numbers for the mobile editor.

In T212386#4943549, @elukey wrote:

@Neil_P._Quinn_WMF Thanks a lot! Mind if I add your snipped to https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas as example?

For the record, he needs this access so he can inspect EventLogging data related to the Editing team's products in Hadoop (one of the most important data streams, EditAttemptStep, is not available in the MariaDB EventLogging store).

In T212386#4942636, @elukey wrote:

SRV records deployed and documentation updated! If anybody could give it a try, I'd be really glad :)

In T209087#4938610, @mforns wrote:

@mpopov Just a heads-up that we'll be turning on the deletion script that will delete unsanitized EL data very soon.
If there are no changes to EL sanitization whitelist, the data that this task refers will be irrecoverable after the script runs.
If you want me to backfill (thus fix) any renamed fields that are missing today, now would be the time! :]

@JAllemandou, @Ottomata, I'd like to get this deployed 😁

In T212386#4927285, @elukey wrote:

I had the chance to have a chat with a lot of people during all hands, this is a summary:

• Mikhail/Chelsy mostly use R for their work, and it seems that the move to a multi dbstore hosts setup should not impact their work too much.
• Tilman/Morten/Neil are using mostly python ad-hoc scripts for their daily work, so as starting point we agreed to 1) add snippets of Python code to generate a host:port mysql combination (to create a valid connection) from a dbname 2) have a tool like the one described above to wrap mysql commands for quick and interactive exploratory use cases.
  
  I already started to modify https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas :)

You can do this using the mariadb.run function with the parameter host = "logs"—see https://github.com/neilpquinn/wmfdata/blob/master/wmfdata/mariadb.py#L21 for details.

In T209052#4897647, @matmarex wrote:

Should probably compare before/after numbers as part of QA. I don't know where they are logged.

@JAllemandou, @Ottomata: do I need to do anything to schedule a deploy of the updated whitelist? From what I understand, it doesn't happen automatically.

In T213597#4899804, @mpopov wrote:

Thank you so much, @Neil_P._Quinn_WMF! Really appreciate you catching that and correcting. I had incorrectly assumed that initial metadata would not be included. I'm currently looking into your suggested method of filtering revisions and comparing it to using revision_parent_id > 0, which should theoretically yield the same result but is not the case in practice.

In T213597#4893605, @mpopov wrote:

Here's the query I used, which I would like someone in Product-Analytics (e.g. @chelsyx and @Neil_P._Quinn_WMF) to review:

In T211173#4892533, @Nuria wrote:

That is quite easy, it just will take a few hours to load data with different transformations and see how the data looks in turnilo, probably @mforns will be doing this work

This is actually kind of blocked on T212529: Standardize datetimes/timestamps in the Data Lake, unless we want to code the UDF to deal with four possible formats...

I going to boldly broaden this into a plea to fully standardize datetimes/timestamps in the Data Lake 🙏 There are actually two more formats that Morten didn't mention! I'll update the description.

Perhaps I can convince you to do the dirty work, @DLynch? 😁

As I mentioned at the Editing offsite in December, my hypothesis is that the difference is due to 2010 wikitext editor init events triggered by bots.

@Milimetric How quickly will you be able to set the cube up in Turnilo once we provide the transform spec? We're trying to prioritize this among our other work 😁

In T212386#4886726, @Neil_P._Quinn_WMF wrote:

Product Analytics met with @elukey and @Milimetric on Monday and had a good discussion about this project, which they are planning to work on (full notes).

There were a couple of action items for me:

• Write [Python] pseudo-code demonstrating how I would invoke and use a hypothetical library used to make the queries.
• Test the existing tools discussed here and determine whether that level of functionality works for us (it likely does)

I discussed this with @Nuria in a meeting today and she clarified that T212386 is important work to Analytics Engineering and said the team plans to try installing the production SQL query script on the new hardware once it's set up (probably in mid-February). However, she can't make any commitments about what the team might do after that.

I sent the list to Edward, and now I'm just waiting for him to make the changes (or possibly object) before I close this.