DAG has passed in production, and the data can be seen at published/datasets/wmde/analytics/wd_rest_api_metrics_monthly/.

The DAG its job at this point are ready. The merge request has been updated and can be seen here: gitlab.wikimedia.org/repos/data-engineering/airflow-dags/merge_requests/738. I don't have time now to do the local testing as I tried earlier today and the query was taking too long, but this will be finished quickly after I'm back.

Data from the DAG can be found at published/datasets/wmde/analytics/wd_item_sitelink_segments_weekly/. There's a 0ed out row for 13-5-2024 as we'd gotten the data from the previous week in testing, but didn't for that week. It'll be collected in T363583 when we get the historical segments.

@Manuel, report has been updated. The queries ran fine, but for some reason the data export didn't work even though it did for testing. Checking with WMF about this. Will get to the rest of the tasks for this month today/tomorrow and the other DAG will be deployed once the issues with the current ones are figured out.

Hey @Manuel. Still not up, which might be because of changes to the queries given the new export to published datasets step. I'm deploying the updated DAG now as tests passed, so we'll see if all finishes.

Hi @Manuel: I just ran the query and it's not yet been updated. Likely that wmf.webrequest isn't updated yet and thus the sensor hasn't fired to trigger the DAG. Will check again tomorrow.

MR is already open with the basic structure of the DAG (basically just the first REST API one with minor modifications). I'll finalize this once the REST API metrics one and the sitelinks one are finalized as far as exporting to the published datasets. Only question on those now is where I should be sending test data to (HDFS or the stat machines).

A question on this as I'm writing the basics while I'm waiting on info for the testing of the other DAGs:

Thank you for this, @Michael! Really appreciate you following along with this and helping out :) :)

Thanks for making this! I've marked the wmde-dashboards project as good for deletion as seen here. Please let us know if further information is needed :)

@Manuel and @Lydia_Pintscher, just shared a folder with the two CSVs on Wolke. Let me know if there's anything else needed, and I will set a reminder that they should be deleted on my end in 89 days (they were generated yesterday). Sharing has been disabled on the directory, so if others need access, then let me know :)

Hi @MarcoSwart 👋 Thanks for the communication here :) I guess I'm a bit confused by how the other one would be used. You're roughly talking about:

@Manuel, my assumption was that you could help any non-analytics PMs or go through the results with them as you have the needed access. Using Google for PII is not something we're supposed to do if it can be avoided, but I have no experience with Wolke. Please let me know if you'd like me to look into Wolke or send the files over Drive.

Talked further with WMF about this just now. One basic question for the end users: would it make it more convenient for you all if the exported datasets were per Wiktionary? There are two options here, with missing entries being used as an example:

I can also prepare a notebook with quick functions to load and explore the data, if that would make the option I suggested a bit easier.

Would it be possible to send us a spreadsheet (and schedule it for deletion after 90 days)?

Base queries for all of this are ready :) Let me know on the above and I'll finalize them. Actually running them will take some time.

Checking on the numbers here really quick: the request is for the top 1000 user agents by number of requests and then a sample of 1000 user agents, but the total is 1221. Would an ordered list of all of them make more sense as we're talking a sample of 82%? There really isn't going to be a difference between the first two sets. An ordered list of all of them and another ordered list of all who were active in May and not in April?

Status is open as T364045 has been resolved :)

Thank you for the efforts here, @JAllemandou! Really great to have this back, and glad that it's worked out in a way where others are not adversely effected :)

Unstalled as the plan for the data export has been approved in T365699 :)

Unstalled as the table has been created :)

Hi @MarcoSwart, sorry for changing the status without explanation. Was in a meeting and we were moving things around, but obviously context should have been added. This is stalled for now as we're waiting for WMF to advise us on the best way forward on migrating data from MariaDB to HDFS. The data processes we need to use for this cannot be run directly on MariaDB in a sustainable way that's in line with long term supported data practices, so first we need to migrate the data to the private data cluster, and then our normal workflows take over. This migration is non-standard, and they're looking into how best to support/guide us.

Note, work that will unblock this task is being done in T364045: [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history.

Quick note on this, in discussion, something to check as well would be those user agents that were present in May 2024, but were not active in April 2024 :)

There's now a MR draft for the DAGs open on GitLab. There's still lots to do as WMF wants to sync on suggestions they'll give me on how to do the MariaDB to HDFS data transfer, but the DAGs are mapped out and the hive queries they're calling have been prepared :)

Thanks so much for the support here, @BTullis! I'll update the epic with this being done. So close to being finished with all this :)

Quick note on this, @Htriedman, was talking with y'all's Data Engineering about this and we made the decision that the query will be reverted to what it was before such that the data lake will have the bigint values, and then in the step that generates the CSV for export to the published datasets directories we'll cast the values to strings and make sure that <25 is replaced for those values that call for it!

wmde/analytics/hql/airflow_jobs/wiktionary_cognate on GitLab now has all the needed queries for missing entries, most popular entries and comparing Wiktionaries. Was easier to write all three at once rather than lose some context later. Note that these are Hive queries as the goal is to first migrate them to HDFS.

Table has been updated with the new data from the most recent DAG run. Lots more user agents - almost a 3x increase. Noting this for now as maybe grounds for further investigation later, but IPs are also increasing (just not by as much).

Perfect, @Lucas_Werkmeister_WMDE! Glad to have this all cleared up :)

Sounds good to me! :) Thanks for the help here, @Lucas_Werkmeister_WMDE and @BTullis!

Hi @Bicolino34 👋 Thanks for reaching out :) We are still working on tasks related to this dashboard - at least bringing back some of the data processes.

Moving this to verification given the work in T364965. Thanks for all of this, @Lucas_Werkmeister_WMDE! Maybe we can resolve this and leave T364965 until stat1007 is deprecated, or resolve both?

None of the files listed in your comment above look like things we should worry about, @Lucas_Werkmeister_WMDE. Similarly that there's a different commit for this, as to my knowledge stat1005 was the main server for the related work.

Given comments in T365700 - https://phabricator.wikimedia.org/T365699#9838686 - resolving this task as the metrics are for the underlying dataset and not user/editor/reader activity :)

Thanks for confirming, @Htriedman! I'm not seeing the message in T365700, but will quote the above message when I resolve it :) Will resolve this one post MR merge!

Thanks for the help here, @Htriedman :) I've now changed the SELECT to the following locally as well as switched over the schema to using string:

I've been asking around about the data source and connecting the tables and have yet to get concrete answers. Based on general assumptions of the names of the tables/columns though, the path forward for getting missing entries for a Wiktionary will be to:

Hey @Htriedman! Is very helpful, yes :) Considering the schema creation for the given table, would it then make sense to convert all of the bigint over to string? I'll also add an annotation of what <25 would mean to the schema :)

Thanks for taking care of this, @Lucas_Werkmeister_WMDE! We'll be able to close both this and T351072 after Tuesday next week if/when the Puppet change is deployed :)