Data SREs have provided a regular output of logs from dumps.wikimedia.org on stat1011 that can be found in srv/log/webrequest/archive/dumps.wikimedia.org. This information has various uses for WMF and WMDE teams on how the dump data for the projects that they work on is being accessed. This task suggests moving these log files to HDFS and creating DAGs to import the data into the data lake and generate monthly request metrics across all dumps.

Current related task: T399808: [Analytics] Baseline metrics for data dumps
Prior related task: T383175: Analyze Dumps Usage Through Apache Logs
Prior related work: GitLab:data-engineering/dumps-log-analysis

Desired Outputs

The desired outputs of this task are listed and confirmed as being finished below.

Please note: WMDE would be happy to do any of the work that is within our capacities for this task. The initial idea is that the DAGs suggested below are shared infrastructure, so with Data Engineering approval the merge request would go to GitLab:data-engineering/airflow-dags/main.

The dump logs are moved to a location in HDFS where they can be accessed by Airflow processes
- Support from Data SREs needed
A daily DAG is written to import the log data from HDFS into a table in the data lake
- Code based on GitLab:data-engineering/dumps-log-analysis is operationalized via a SparkOperator in the DAG
A monthly DAG is written to calculate usage metrics across all dumps and write them to a table in the data lake
- Input table is the above HDFS import table via an ExternalTaskSensor and daily_execution_dates_of_month
- We need this to run on the second day of the month as the log file for the first day has requests from the one for the last day of the month (0 0 2 * * )

NOTE: The first table mentioned above, the data imported from HDFS, is being suggested as WMDE has other processes that we'd like to do with this data. We would use this interim table for those as well.

Deadline

Please make the time sensitivity of this request clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

DD.MM.YYYY

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Open	KCVelaga_WMF	T422880 Dumps user identification monitoring
Resolved	Ahoelzl	T403159 Airflow processes to import dump logs and generate monthly metrics

Event Timeline

AndrewTavis_WMDE created this task.Aug 28 2025, 10:50 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 28 2025, 10:50 AM

AndrewTavis_WMDE updated the task description. (Show Details)Aug 28 2025, 11:00 AM

AndrewTavis_WMDE moved this task from Incoming to Backlog on the WMDE Analytics board.

AndrewTavis_WMDE updated the task description. (Show Details)Aug 28 2025, 11:08 AM

AndrewTavis_WMDE updated the task description. (Show Details)Aug 29 2025, 4:37 PM

There is probably a reason why not, but could we just put dumps.wm.org behind the CDN? Then requests to it would be in the usual webrequest logs. Tagging Traffic for an answer :)

Ahoelzl moved this task from Incoming (new tickets) to Needs Clarification on the Data-Engineering board.Sep 10 2025, 4:28 PM

GGoncalves-WMF added a subscriber: HCoplin-WMF.Sep 10 2025, 4:28 PM

CDanis subscribed.Sep 10 2025, 4:29 PM

In T403159#11168454, @Ottomata wrote:

There is probably a reason why not, but could we just put dumps.wm.org behind the CDN? Then requests to it would be in the usual webrequest logs.

T306550: Move dumps.wikimedia.org HTTP service behind CDN edge, the rsync traffic needs to move to a different hostname first.

TIL you can point rsync at dumps.wikmedia.org (if we allow you to).

https://phabricator.wikimedia.org/T306550#7968059

Thanks.

This (move dumps to the CDN) has been brought up recently in other discussions as well and Traffic is happy to triage it if required but this does need input and work from the team that manages Dump, so that sync needs to happen before we can do that on the Traffic end. [Speaking with my Traffic interim manager hat on here.]

How about if we did something like this?

(1) nginx -> (2) rsyslog -> (3) kafka-logging               (5) kafka-jumbo -> (6) sqoop? -> (7) hdfs
                                       \> (4) mirror-maker -^

(1)->(2) : nginx logging to rsyslog should be quite easy to set up.
We have (2) and (3) set up already.
We are already starting to set up a mirror-maker instance (4) to mirror some topics from (3) kafka-logging to (5) kafka-jumbo - as part of T304373

I'm not sure if sqoop would be the right tool for (6) - but maybe there would be other easy choices. (e.g. benthos, flink)

That way, we wouldn't need to change the CNAME for dumps.wikimedia.org and put it behind the CDN.

BTullis added a project: Data-Platform-SRE.Sep 10 2025, 5:13 PM

Alternatively, we could just use an hdfs-rsync (or similar) process on the clouddumps100[1-2] servers to have them periodically copy the files from the local file system to HDFS directly.
It's not as nice as either of the options involving kafka (in my opinion), but it could work.

I mean, we want the Kafka logging mirror maker piece anyway for T304373, but I think we were considering only mirroring Event Platform (compatible) stream topics from there. If they are other topics, we'll also have to set up custom gobblin ingestion.

Changing priority to low and there isn't an immediate action. We need to continue the discussion and have a clear general direction for Dumps and Dumps infrastructure first. In particular, there are some ideas of exposing dumps through Ceph, which would require a different kind of solution to get metrics.

AndrewTavis_WMDE mentioned this in T399808: [Analytics] Baseline metrics for data dumps.Sep 23 2025, 1:34 PM

Ottomata moved this task from Needs Clarification to Backlog on the Data-Engineering board.Oct 8 2025, 5:23 PM

AndrewTavis_WMDE mentioned this in T411872: [Analytics] [Request] [Repeating] Monthly manual metric generation processes.Jan 7 2026, 4:15 PM

Lena_WMDE moved this task from Backlog to Watching on the WMDE Analytics board.Apr 10 2026, 11:29 AM

BTullis added a parent task: T422880: Dumps user identification monitoring.Apr 16 2026, 12:51 PM

Ahoelzl mentioned this in T425128: Ingest JSON access logs for dumps.wikimedia.org in the data lake.May 1 2026, 4:27 PM

Ahoelzl claimed this task.Jun 8 2026, 9:06 PM

Ahoelzl edited projects, added: Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed: Data-Engineering.

Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.Jun 15 2026, 3:29 PM

Ahoelzl edited projects, added: Data-Engineering (Q1 FS26/27 July 1st - September 30th); removed: Data-Engineering (Q4 FS25/26 April 1st - June 30st).Jul 8 2026, 12:45 PM

Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.

taavi unsubscribed.Jul 13 2026, 12:02 PM