Just to have some fun I counted current usernames that start with all of these proposed prefixes (and I added a goofy proposal of my own, 73^^9):

This is temporarily in review to get opinions on the way I handled the delayed daily timetable interacting with our datasets idea. It's kind of hard coded but I think simpler than a more flexible approach. Let me know what you think. (assigned to Sandra but anyone is welcome to comment)

Thanks for this context, @Niharika! I think it's ok to wait and see when the temp user cookies roll out how those numbers compare with these. If we wanted to know ahead of time, we could probably devise some kind of sampled test by setting a simple cookie on edits and incrementing it... But I don't think it's worth the trouble.

In T332004#8696727, @hashar wrote:

What is the replacement to analytics/wikistats? If it exists maybe @Nemo_bis can migrate to it rather than relaying on an abandoned code base?

+1 to archive then, thanks Federico! And maybe let me know if we can do anything in wikistats 2 to help you out.

sorry we left it open so long. I just have to check with Nemo, I'll reply back within a few days.

note to self, look at T331892: Move eventlogging_to_druid_ jobs to airflow in order to rely on cluster spark mechanism instead of client

Quick update from our last conversation:

found a blocker: Druid pageview jobs need pageview_hourly to continue to create _SUCCESS files, but they weren't spelled out in the migration doc so this was missed. I'll update the airflow jobs on monday and resume deployment.

@apaskulin: created as site ID 24 (note: 24 backwards is 42! :P )
you can see the tracking code here: https://piwik.wikimedia.org/index.php?module=CoreAdminHome&action=trackingCodeGenerator&idSite=24&period=day&date=yesterday&updated=false

Thanks for catching that! Merged - it will auto-deploy

I'm putting this in review, but there are three jobs being migrated so I'll send them in separate patches.

I'm working on a merge request for this, testing the jobs (it's going slow 'cause I'm on ops week)

+1 to Timo's suggestion. The change required is fairly contained right now.

dibs! Yaay :)

Oooh, I'd love to work on this

In T309769#8561444, @Snwachukwu wrote:

I ran the UDF on a day's data and extracted the top 1000 referer's for that day to show the impact of the GetRefererDataUDF on referers. You can check the spreadsheet and a little doc on it.

@EChetty: The old Analytics tag should auto-tag Data-Engineering or be archived/deleted so folks can't use it. I've heard a lot of confusion around the team name lately, and I think the phab tags may be a primary source for that.

Note: Per @EChetty, This task is currently deprioritized since there is a plan to reduce tech debt and use the new mediawiki.page-change stream instead of mediawiki_history for building data pipelines. See more T311129

In T312566#8542940, @Antoine_Quhen wrote:

lmkwyt

just to have this on record: the Wikistats annotation display system was thrown together quickly. It can be easily modified to include ranges, and it probably should be in this case and a few other cases. Annotations are great UX and I still think putting them on wiki is a great way to get the community involved.

In T324907#8503754, @Dreamy_Jazz wrote:

@Milimetric: Keeping data engineering in the loop like for T233004. This task will likely affect your analysis in a larger way than T233004 is as it will split data stored in cu_changes into three tables (cu_changes, cu_private_event and cu_log_event).

note for myself: https://github.com/apache/iceberg/pull/6182/files is recent activity about supporting deletes in future Flink / Iceberg APIs

In T95582#8502156, @bd808 wrote:

In T95582#8501818, @Milimetric wrote:

March: we are generating a proof of concept XML dump, similar to the current one, via a Kafka -> Spark -> Iceberg -> XML pipeline. This depends on Event Platform's content-enriched Kafka topic (page-content-change). It also depends on me getting better at all this, some big moving pieces and lots to learn but I'm getting there. If the Kafka -> Iceberg jump is too challenging, the nice thing here is we can fall back on hourly batches, and that's still ok for lots of use cases, right @bd808?

I am sure that there are folks who would be ok with things stopping at a big pile of data inside of the restricted access production Hadoop cluster. That does not actually move anything forward for Quarry and the general public however.

In T95582#8500253, @bd808 wrote:

In T95582#8500240, @Milimetric wrote:

At this point, we are making significant progress on near-real-time dumps generation. If that project continues to work out, we could have an alternate view available for querying, one that would include data from all wikis, identified by perhaps a wiki_db column or similar. Just putting it out there as a possible solution to this very old problem (I haven't forgotten about this, progress here is just... slow)

You are going to end up with a new real-time replicated, redacted RDBMS as a side effect of dumps generation? That sounds awesome, but also not fully believable.

At this point, we are making significant progress on near-real-time dumps generation. If that project continues to work out, we could have an alternate view available for querying, one that would include data from all wikis, identified by perhaps a wiki_db column or similar. Just putting it out there as a possible solution to this very old problem (I haven't forgotten about this, progress here is just... slow)

@Ottomata I don't have any preference here, it just occurred to me that you could also work around the $ref problem like this:

+1 on T288301#8487410, @BPirkle

In T233004#8491382, @Zabe wrote:

In T233004#7796847, @Milimetric wrote:

[...] We (data engineering) would like to be kept in the loop as closely as possible here.

@Milimetric cuc_actor has been fully populated by now and we can now start reading from cuc_actor instead of cuc_user and cuc_user_text.

@taavi: for a little more context, the dashboard basically keeps track of statistics about access to content provided by GLAMs. The long term plan is to make better APIs that can directly serve their needs, but for now they need to parse mediarequest dumps and store the results. This would be easier in more ways than one on our cloud infrastructure.

Works! Not an issue in hive either, @BTullis

@Volans asked me, basically, how come count(distinct ip) gives slightly inaccurate results in superset -> druid queries. I didn't know, but found out that Druid by default has: useApproximateCountDistinct: true. See more at: https://support.imply.io/hc/en-us/articles/360056362993-Getting-exact-count-distinct-results-using-druid-SQL. Here's an example, and how to go about getting exact answers without tuning that setting.

Just for anyone that grabs this, we already define "active wikis" and use it in datasets like public geoeditors, the query for a dataset would be something like:

Deciding against Flink, at least for now. Documenting as a decision record here.

@Michael: to add a little more detail on what Joseph said, querying 5 days of webrequest (only text) means moving 5 * 1.3T = 6.5T over the network. So there are two important points here.

I was wrong to think I'd finish this by the end of the week. It's just been a series of errors with no docs to help. Current state is Iceberg is having trouble reading metadata, seems like somehow it doesn't know how to use HDFS?

In T321707#8346430, @Aklapper wrote:

@Milimetric: See Pageviews-Anomaly and especially T263908. What is the "stats team" exactly?

Got the basics set up in the Flink SQL client. Updating my code from before. I think I'm going to leave Flink SQL here. The problem is that it has pretty bad actual SQL support (like no built-in timestamp functions etc.) So to use it to actually do the kinds of transformations we need we'd have to build timestamp parsing UDFs and stuff like that. I feel that if you're writing Java/Scala anyway, you might as well just stay in Java and write the whole job there. That way at least all the logic is in one place and understanding the code doesn't require understanding multiple environments. Maybe if we do more work to make the Flink SQL environment painless, we can come back to this. For now, a scala or python Flink job seem to me the best way forward.

Ok, looks good, please check and let me know. If any other language is ready, just file a task and let us know. We'd have no way of knowing that on our own, because even if all the messages are done, they may not be ready for use.

Thanks @Aftabuzzaman, I didn't know about Bengali. Releasing a new language is a manual process at the moment. I'm building and deploying now. It should be available in the next half hour or so.

I think we should do this. We can limit the pages we look at with the import log as Neil says, and then just mark all the revisions that have much larger revision ids than their parent (via rev_parent_id as revision_is_probably_imported