Ditto what @xcollazo said above. In order to have the desired behavior for this pipeline job, I think you need:

@Eevans thank you for that MR! You are correct, wiki_id should be TEXT - we've already implemented it in the Hive counterpart for that table: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1206879

Addressed in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1794

OK so I've now officially backfilled the wmf_contributors. and wmf_readership. tables, but the process I had to use in order for the number of files to be small enough is complicated enough that it warrants being documented somewhere:

We definitely already have the maven-checkstyle-plugin set up in the main pom.xml - I know because it's very annoying since the codebase doesn't seem to conform to the style being checked, and on each compile it produces a ton of ERRORs in output.

That specific use-case sounds like what dbt calls a microbatch incremental strategy that replaces time intervals given the event_time column: https://docs.getdbt.com/docs/build/incremental-microbatch

@JMonton-WMF I think we could use this task to include a .dbtignore file that will let dbt commands ignore the .ipynb_checkpoint folders: https://docs.getdbt.com/reference/dbtignore

insert_overwrite is what @JAllemandou is describing, perfect.

Do we want only Druid realtime configs its own repo? Perhaps we want the batch ones in the same place?

My uninformed thought on this is that this should be a "Druid config stuff" repo, which would therefore include both realtime AND batch configs :)

user_id, user_central_id and page_id fields are now available in both the Hive dataset wmf.mediawiki_history_reduced as well as in the corresponding Druid dataset.

In T406766#11297869, @BTullis wrote:

Yes, that's true. But conda-analytics isn't necessarily a long-term solution. I'd be much happier to start out with a container based solution as per: T406636: Create a dbt Docker container but container runtimes are not available to us on the stat servers at the moment.

At least this way, we will have something unform to work with already on the stat servers.

@Ottomata so it sounds like we are ready to accept the mediawiki_history_reduced dataset as it is right now, but with user_central_id and page_id columns added? If so, I'll start backfilling September 2025.

WIP Typo! I do that when i'm creating test tables so I can increment as I make changes. Final one will not have 0, but maybe _v1 if we want to version it!

@BTullis wouldn't this approach introduce a discrepancy between what users use on stat boxes and what is run in GitLab CI/CD and eventually in Airflow? The latter two will run in Docker images, and I wonder how different the two installations will end up being.

@Ottomata Regarding the naming of user_edited_pages as described in T407559 - why do we need the zero at the end of the table name? user_edited_pages_daily0 looks unnecessary to me, but I could be convinced otherwise.

Oh TIL that we can use kokkuri as a GitLab component! Thumbs up.

I've created https://gitlab.wikimedia.org/repos/data-engineering/dbt

In T406392#11247933, @Scott_French wrote:

@amastilovic - Have you subsequently been able to successfully build / push the docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags image?

This all feels very achievable, but I wonder if we might be making things difficult for ourselves by trying to define one operator that can do it all, like a single Swiss Army knife.

This might be quite onerous on ops week duty and/or folks just trying to upgrade or deploy their job.

We have that manual forced cache warmup for precisely this scenario by the way.

Would data sizes be any concern? One of our use cases is a weekly transfer of a few hundred gb spread across ~10k files in a nested directory structure.

Blunderbuss could easily do this for you, with minimal resource usage on the Airflow executor side :-)

The 2025-08 backfill run of the DAG has completed successfully, and judging by data sizes on HDFS I'd say it falls in line with what we've seen in the previous months. @GFontenelle_WMF if you have some basic validation checks to run on this data, now would be a good time. Thank you!

The new SQL query is:

As far as I can see, all these checkers already use DeequColumnAnalysis so the basic plumbing seems to already be there. I'll have to investigate a bit deeper to see if and how it's being used right now, but hopefully the scope creep won't be too big. I'll keep you updated here.

It seems to me that this growth is organic and should be taken into account when doing the error checks, but the MediawikiHistoryChecker unfortunately doesn't provide for fine-tuning the error boundaries. The way it works right now is, it accepts the minimum and maximum boundaries for any kind of growth. These same two boundaries are then used for user growth, page growth, denormalized history growth and reduced history growth checks.

Use the following example of how these values should be saved in the deployment-charts repository: https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/dse-k8s-services/_airflow_common_/values-analytics-production.yaml

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1609

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1604

In T401694#11082769, @bking wrote:

Sorry for the confusion! I've updated the task description to be more clear. The packages will be built with the jdk and jre; see the current jdk8 Debian repo component for an example of what will be available.

Another vote for JRE version, too.

In T397923#11073329, @JAllemandou wrote:

I approved the MR above.
And actually while we're at it, we should review if the list here https://github.com/wikimedia/analytics-refinery/blob/master/bin/refinery-drop-mediawiki-snapshots#L85 matches the list of tables we sqoop!

In T401325#11065897, @Ottomata wrote:

Hi @amastilovic, could you look into this? Priority is high, but it doesn't need to be fixed-fixed until the end of August, before the next monthly snapshot runs.

Thank you!

Thank you, @BTullis !!

OK so I've done some investigation work and here's what I think needs to be done:

Agreed, I think this is WAD.

Sounds good, closing the ticket.

In T382430#11043433, @Ottomata wrote:

I heard @amastilovic say today that GitLab CI components can't be used for manual CI job runs? If so, that is quite a con and maybe a reason not to use them?

I can't explain why it didn't hit the permission errors when I ran it manually.

In T400188#11035241, @BTullis wrote:

In T400188#11035218, @amastilovic wrote:

Could the solution be to mount /var/cache/archiva onto a new volume?

Yes, we can definitely do this. I'll add a new disk of, say, 200 GB and mount this to /var/cache/archiva.
I can sync a copy of the existing data to the new disk ahead of time, which will minimize the time that the archiva service has to be down.

In T400188#11035220, @BTullis wrote:

Oh it's /var/cache/archiva - it even says so in the title of the ticket!