Sorry, I responded to this but on slack. (wrong place.) Here's what I said: I would say https://phabricator.wikimedia.org/T422522 (technical build) because it's not really about correctness (2.5.6), but more about determining how performant our replacement for the Blazegraph label service handling is.

Friday update: the WDP engineers are meeting with Data Platform contacts on Tuesday to discuss the project and what they recommend. We've prepared a short doc summarizing the current state that we will share with the rest of the meeting attendees on Monday. Updates from that meeting will be available next week.

Jar successfully built: https://gitlab.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-streaming-consumer/-/packages/2062

Update: the new repository exists at wikidata-platform/rdf-streaming-consumer. I merely cloned the original RDF repo and am working on an MR to delete everything but the consumer. It will depend on an artifact created by the new wdqs-common repo.

This is all done. I added linting in a pipeline for MRs in the Analytics repo. I also added a license and readme. Turns out the "main" branch of the repo was already protected, too. So, all done with this cleanup.

I've added the apache 2.0 license and a very small README. I'll work on the other items next week.

Update: I've added some documentation to help others solve this problem in the future.

Thank you for reporting those @Mbch331 ; I ran the same script on those entities and they should be un-queryable now (or soon, due to eventual consistency).

Hi there! First, sorry for the slow response on this. I've just run a script to trigger RDF reconciliation for all the entities named here, and they should be deleted from WDQS now. If you see that your deleted entry is still queryable, please follow up and let me know.

The DAG to materialize the top user-agents (>1000 qpd) is running. It cleans up data >90 days as well, and I also set up a monthly maintenance DAG to delete old iceberg snapshots. This is in accordance with the standard practices around this type of data, according to a consultation I had with the data engineering advisors.

• The table exists: wikidata.wdqs_external_queries_by_user_agent_daily aggregates the queries by latency time class for each user agent per day, for user-agents with >1000 queries a day. We also store total query counts and the number of successful queries.
• The DAG to populate this table has been deployed.
• Data cleanup after 90 days at the row level happens via the data-populating DAG, too.
• Another DAG to do Iceberg snapshot cleanup runs separately; this MR is under code review.

Project update:

That sounds like the perfect solution. Thank you!

Hi! This was all a mixup on my part; I didn't realize that I needed to submit an access ticket to add myself to my team's analytics user. This ticket is still open and tracks a bigger project, which I needed the analytics access for. Sorry about the mixup. We still want this task open to track my additional work.

Hi again,

Thanks for your request for this data, and we understand this would be a really useful dataset for WMF to release.

Hi there @Pfps ! I'm looking into refreshing the code that was previously used for query anonymization. Would you clarify what exactly you're hoping to get as output? Of course the queries in anonymized form, but what else - timestamps, user agent categories, etc?

Oh, I am closing this, because I found the documentation for getting a Kerberos identity: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Kerberos#Get_a_password_for_Kerberos I'll follow that process.

Hi again! Thanks for granting this access!