Page MenuHomePhabricator
Create Task
Maniphest T352650

WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes
Closed, ResolvedPublic
Actions

Description

Update Dec 2024

As per the APP FY24-25: OKRs and Hypotheses spreadsheet, this work has now been associated with a hypothesis in the WE 5.4 KR. The hypothesis has the ID of 5.4.4 and the wording is as follows:

If we decouple the legacy dumps processes from their current bare-metal hosts and instead run them as workloads on the DSE Kubernetes cluster, this will bring about demonstrable benefit to the maintainability of these data pipelines and facilitate the upgrade of PHP to version 8.1 by using shared mediawiki containers.

The ideal solution is still to schedule the dumps with Airflow, but this wording of the hypothesis deliberately makes no mention of this, since we do not wish to place Airflow integration on the critical path to achieving the stated objective by April 1st 2025. The option of using Kubernetes CronJob objects (or similar) would meet the objective.

Original ticket description below

While there's ongoing work to create a new architecture for dumps, we still need to support the current version. In concise terms, what happens now is that a set python scripts launch periodically some mediawiki maintenance scripts and the (very large) output of those is written to an NFS volume that is mounted from the dumpsdata host.

After some consideration of other plans that were outlined in this task originally, we've decided to actually port the dumps to run on k8s, as outlined .

In short (for the longer version see Ben's document) we want to do the following:

  • Create a new MediaWiki image that includes a python interpreter and the dumps code
  • Allow the MediaWiki chart to include a PersistentVolume mounted under /mnt/dumpsdata and the dumps configuration (now in puppet) under /etc/dumps/
  • Run a MediaWiki deployment on the DSE k8s cluster using a specific ceph volume to write the dumps to, substituting the function of NFS, running the python scripts as CronJob in a first implementation, moving to airflow later
  • Run a dependent job to copy the dumps files over to the server that will serve them to the public

Details

Due Date
Apr 1 2025, 8:00 AM
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add another 10 TB to the legacy dumps pvcoperations/deployment-chartsmaster+1 -1
mediawiki: Use the servergroup to configure the dumps feature flagoperations/deployment-chartsmaster+1 -3
mediawiki: Update the dumps job template to support write accessoperations/deployment-chartsmaster+4 -1
mediawiki: bump versionoperations/deployment-chartsmaster+1 -1
mediwiki-dumps-legacy: Create helmfile deployment of a suspended joboperations/deployment-chartsmaster+459 -0
mediawiki: Add support for dumps suspended joboperations/deployment-chartsmaster+113 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add data-engineering/sync-utils to the trusted runnersrepos/releng/gitlab-trusted-runner!111btullisadd_sync_utilsmain
Add curl to the mediawiki-debug image for use with dumpsrepos/releng/release!154btullisadd_curl_mwdebugmain
Add the bzip2 package to the mediawiki-debug image for legacy dumpsrepos/releng/release!153btullisadd_bzip2_mwdebugmain
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT88728 Improve Wikimedia dumping infrastructure
 ResolvedBTullisT352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes
 ResolvedJoeT381473 Generate a dumps-enabled mediawiki image
 ResolvedClement_GoubertT389484 Create a mediawiki-cli image
 ResolvedScott_FrenchT389499 Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds
 ResolvedBTullisT382489 Create a namespace on dse-k8s and suitable tokens for testing/running legacy dumps
 ResolvedBTullisT382490 Allow the cephfs csi plugin on dse-k8s to mount the dumps cephfs volume
 ResolvedbrouberolT386189 Allow the dse-k8s cluster to pull images from the protected docker registry
 ResolvedbrouberolT388378 Orchestrate dumps v1 from an airflow instance
 ResolvedbrouberolT389762 Monitor the free space in the dumps v1 Ceph volume
 ResolvedbrouberolT389770 Adjust the dumps v1 task retry policy
 ResolvedBTullisT389773 Adjust maximum number of mapped tasks in the airflow instance
 ResolvedbrouberolT389777 Monitor the resource usage in the mediawiki-dumps-legacy namespace
 ResolvedBTullisT389778 Monitor the load on the Kubernetes API masters
 ResolvedbrouberolT389782 Adjust dump tasks resources after observing real-life data
 ResolvedBTullisT389784 Synchronize the dumps files to the clouddumps hosts
 ResolvedBTullisT390736 Create a container image containing the necessary sync utilities for publishing dump files
 ResolvedBTullisT390738 [dumps] Configure rsyncd authentication - or a suitable alternative
 ResolvedbrouberolT389786 Integrate mediawiki-dumps-legacy with the regular MW scap deployments
 ResolvedbrouberolT389931 Prototype the airflow DAG responsible for running the xml-sql v1 dumps jobs
 ResolvedBTullisT389962 Differentiate full and partial dumps in the DAGs
 ResolvedbrouberolT389759 Document the dumps v1 on airflow infrastructure
 ResolvedbrouberolT390059 Large wiki dump is getting stuck when running in airflow
 ResolvedbrouberolT390110 Make it possible to run manual dumps command from the toolbox pod
 ResolvedbrouberolT390186 Integrate recent contributions to the KubernetesPodExecutor
 ResolvedbrouberolT390852 Split wiki dump DAG into multiple tasks
 ResolvedbrouberolT390945 Run an experimental dump of 200 regular sized wikis
 ResolvedbrouberolT391332 Create a dashboard displaying airflow pool, scheduler, executor and operator activity
 ResolvedbrouberolT391481 Disable XCOMs for dump tasks
 ResolvedbrouberolT391480 Experiment with absolute priority weights
 ResolvedbrouberolT391466 Add more failure tolerance to the dumps DAGs
 ResolvedbrouberolT391497 Adjust the scheduler liveness checks to account for periods of high load
 ResolvedbrouberolT391510 Tolerate dump task pods being OOM killed
 ResolvedbrouberolT391556 Adjust the scheduler resource requirements
 ResolvedbrouberolT391483 Experiment with disabling dynamic task mapping
 ResolvedbrouberolT391678 Adjust the file processing interval
 ResolvedBTullisT391564 Adjust the database resource requirements
 ResolvedbrouberolT391669 Investigate how to run more than 16 parallel dump task instances
 ResolvedbrouberolT391684 Experiment with running multiple concurrent dump DAGs to keep each graph relatively low in size
 ResolvedbrouberolT391744 Increase file processing timeout
 ResolvedbrouberolT391745 Write a DAG checking that the statically defined list of DAGs matches what NOC returns
 ResolvedBTullisT391791 Adjust database retention settings
 ResolvedBTullisT391837 Investigate running the largest wikis with a higher priority weight
 ResolvedbrouberolT392461 Batch dump jobs within a single airflow task to speed up DAG execution
 ResolvedbrouberolT392470 Increase mediawiki-dumps-legacy namespace quotas to enable running 32 parallel pods
 ResolvedbrouberolT392987 Adjust webserver resources
 ResolvedbrouberolT392995 Completed dump pods take a long time to get terminated
 ResolvedbrouberolT391492 Infer the --date parameter from the DAG scheduling date
 ResolvedBTullisT391087 Investigate an incident where the airflow-test-k8s database was wiped
 ResolvedbrouberolT391348 airflow: Restrict the rights for airflow deployers to destroy postgresql clusters
 ResolvedBTullisT392915 Batch dump jobs in a way that makes business sense
 ResolvedbrouberolT393407 Untangle dump DAG dependencies
 ResolvedBTullisT393687 Skip sync tasks for private wikis, or allow them to fail gracefully
 ResolvedbrouberolT393689 Create a sync-toolbox pod that is similar to the dump-toolbox pod
 ResolvedbrouberolT397039 Set a proper DAG run timeout on the dump DAGs
 ResolvedBTullisT388707 Render the dumps configuration files into configmaps in the mediawiki-dumps-legacy namespace
 ResolvedBTullisT388761 scap needs to be k8s-cluster aware
 ResolvedBTullisT390067 Deploy the dumps monitor script, which generates the html status files, to the dse-k8s cluster
 ResolvedBTullisT394389 Migrate the additional dump types from snapshot1016 to Airflow
 ResolvedbrouberolT394647 Repurpose spare snapshot servers as dse-k8s-workers
 ResolvedBTullisT397858 Pre-load the 20250601 dumps to the cephfs volume to use as prefetch data for the 20250701 dumps
 ResolvedBTullisT397848 Switch over the Dumps_v1 system to run from Airflow instead of snapshot servers
 ResolvedBTullisT398438 Decommission or recommission all snapshot and dumpsdata servers
 ResolvedRequestVRiley-WMFT401182 decommission snapshot1010.eqiad.wmnet
 ResolvedRequestNoneT401181 decommission dumpsdata1003.eqiad.wmnet
 Resolvedbd808T401174 Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1
 ResolvedVRiley-WMFT401299 Investigate whether we can add RAM to dumpsdata100[4-7] from any decommissioned hosts
 ResolvedBTullisT401425 Investigate memcache errors during wikidata and commons dumps runs

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Dec 16 2024, 11:09 AM
Joe added a comment.Dec 16 2024, 11:12 AM

I have added a patch to the mediawiki chart to allow claiming persistent volumes for dumps specifically. @BTullis I guess when both this patch and the one to add the needed stuff to the debug image are ready, I think you'll be ready to create a deployment on the DSE cluster.

I think it might make sense to also link the files under helmfile.d/services/_mediawiki-common_ in your space, as we definitely want to keep the settings in sync.

Let me know if you need further assistance in setting that up.

BTullis added a comment.EditedDec 16 2024, 11:27 AM

Thanks @Joe - I think that looks useful.

I'd also like to catch up on the current work with regard to how this chart is being used to deploy Job objects, if possible.
I see various references to mwscript and mercurius here, so if you (or someone) could let me know the current direction of travel regarding these two things, I'd be grateful.

Clement_Goubert subscribed.Dec 16 2024, 12:04 PM
In T352650#10405691, @BTullis wrote:

[...]

I see various references to mwscript and mercurius here, so if you (or someone) could let me know the current direction of travel regarding these two things, I'd be grateful.

Not sure that's the right file, but Mercurius and mwscript are two Job-type deployments of mediawiki that don't include httpd. Yours would be a third one. You'd also have to add the command that's going to be ran to the MediaWiki container part of the templates/lamp/deployment.yaml.tpl file, and exclude dumps from the liveness probe in that same file.

The rest of the sidecars etc. can be configured from the values.yaml files.

I'm in the process of refactoring the mediawiki chart, but I don't want to block you moving forwards, so I'll port any changes you make when I have something that works.

Clement_Goubert added a comment.EditedDec 16 2024, 12:49 PM

You probably also want to if-guard the service definition in templates/service.yaml.tpl

dr0ptp4kt unsubscribed.Dec 16 2024, 4:28 PM
Scott_French closed subtask T381473: Generate a dumps-enabled mediawiki image as Resolved.Dec 16 2024, 5:38 PM
BTullis renamed this task from Migrate current-generation dumps to run on kubernetes to WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes.Dec 19 2024, 12:30 PM
BTullis updated the task description. (Show Details)
BTullis set Due Date to Apr 1 2025, 8:00 AM.
BTullis removed subscribers: • AeinBagheri, • lbowmaker, • WDoranWMF.
BTullis mentioned this in T382489: Create a namespace on dse-k8s and suitable tokens for testing/running legacy dumps.Dec 19 2024, 1:14 PM
BTullis added a subtask: T382489: Create a namespace on dse-k8s and suitable tokens for testing/running legacy dumps.
BTullis added a subtask: T382490: Allow the cephfs csi plugin on dse-k8s to mount the dumps cephfs volume.Dec 19 2024, 1:21 PM
BTullis closed subtask T382489: Create a namespace on dse-k8s and suitable tokens for testing/running legacy dumps as Resolved.Dec 19 2024, 5:22 PM
BTullis mentioned this in T368098: Dumps generation cause disruption to the production environment.Dec 20 2024, 2:15 PM
BTullis mentioned this in T382947: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]).Jan 3 2025, 1:21 PM
BTullis added a comment.Jan 3 2025, 3:53 PM

For reference, here is the full list of jobs that are understood by the dumps worker process.

  • noop (runs no job but rewrites checksums files andresets latest links)
  • latestlinks (runs no job but resets latest links)
  • createdirs (runs no job but creates dump dirs for the given date)
  • tables (includes all items below that end in 'table')
    • sitestatstable
    • imagetable
    • pagelinkstable
    • categorylinkstable
    • imagelinkstable
    • templatelinkstable
    • linktargettable
    • externallinkstable
    • langlinkstable
    • usergroupstable
    • userformergroupstable
    • categorytable
    • pagetable
    • pagerestrictionstable
    • pagepropstable
    • protectedtitlestable
    • redirecttable
    • iwlinkstable
    • geotagstable
    • changetagstable
    • changetagdeftable
    • flaggedpagestable
    • flaggedrevstable
    • flaggedpageconfigtable
    • flaggedrevspromotetable
    • flaggedrevsstatisticstable
    • flaggedrevstrackingtable
    • sitestable
    • wbcentityusagetable
    • babeltable
  • namespaces
  • namespacesv2
  • pagetitlesdump
  • allpagetitlesdump
  • abstractsdump
  • abstractsdumprecombine
  • xmlstubsdump
  • xmlstubsdumprecombine
  • articlesdump
  • articlesdumprecombine
  • metacurrentdump
  • metacurrentdumprecombine
  • xmlpagelogsdump
  • xmlpagelogsdumprecombine
  • metahistorybz2dump
  • metahistory7zdump
  • articlesmultistreamdump
  • articlesmultistreamdumprecombine

I want to make sure that I understand the logic of when these get called by the worker and/or scheduler process, so we can work out an efficient command to call for each of the deployments orjobs that we create.

BTullis added a comment.Jan 3 2025, 5:25 PM

There is some useful background information here, too: https://wikitech.wikimedia.org/wiki/Dumps/Phases_of_a_dump_run

• VirginiaPoundstone edited projects, added Data-Engineering; removed Test Kitchen.Jan 6 2025, 9:21 PM
BTullis closed subtask T382490: Allow the cephfs csi plugin on dse-k8s to mount the dumps cephfs volume as Resolved.Jan 7 2025, 12:09 PM
BTullis renamed this task from WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes to WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes.Jan 7 2025, 4:11 PM
BTullis moved this task from Epics to Quarterly Goals on the Data-Platform-SRE board.Jan 9 2025, 12:34 PM
gerritbot added a comment.Jan 24 2025, 3:27 PM

Change #1114001 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] mediwiki-dumps-legacy: Create helmfile deployment of a suspended job

https://gerrit.wikimedia.org/r/1114001

BTullis added a comment.EditedJan 24 2025, 5:56 PM

I have had some more thoughts about how to get this to work and I have made a few changes to this mediawiki chart patch to support it.

Also, I started work on a helmfile.d/dse-k8s-services/mediwiki-dumps-legacy release, to accompany it.

The overall idea is this:

  • We do a standard helmfile -e dse-k8s-eqiad apply in the dse-k8s-services/mediawiki-dumps-legacy directory.
  • This creates the following objects from the mediawiki chart in the mediawiki-dumps-legacy namespace:
    • networkpolicy: mediawiki-production
    • configmaps:
      • mediawiki-production-envoy-config-volume
      • mediawiki-production-wikimedia-cluster-config
      • mediawiki-production-wmerrors
      • mediawiki-production-rsyslog-config
      • (Probably the mcrouter config as well)
    • persistentvolumeclaim: mediawiki-dumps-legacy-fs
    • job: mediawiki-production-dumps-job-template
  • We also need to get the files under /etc/dumps/confs and /etc/dumps/dblists into the container, so maybe we need a dumps chart for this.

The mediawiki-production-dumps-job-template job will be a suspended job, so it will not actually run anything, but it will contain a pod spec for running jobs.

Then, when we start the airflow DAG to carry out the dumps, the first task will be to use kubectl or curl to describe the mediawiki-production-dumps-job-template and extract the spec.template from it.
This pod template will then be saved as an XCom, which will be used by subsequent tasks using the KubernetesPodOperator.

We can also generate an XCom which is a list of wikis, then we can create a task group, which is a worker process for each of the wikis,

If we do this, then I think that we can avoid having to use the custom looping over all wikis in worker.py.

Ottomata moved this task from Incoming (new tickets) to Tag with Roadmap on the Data-Engineering board.Feb 11 2025, 7:35 PM
Ottomata edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.
brouberol closed subtask T386189: Allow the dse-k8s cluster to pull images from the protected docker registry as Resolved.Feb 17 2025, 9:13 AM
gerritbot added a comment.Feb 21 2025, 9:36 AM

Change #1104605 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Add support for dumps suspended job

https://gerrit.wikimedia.org/r/1104605

gerritbot added a comment.Feb 21 2025, 9:36 AM

Change #1114001 merged by jenkins-bot:

[operations/deployment-charts@master] mediwiki-dumps-legacy: Create helmfile deployment of a suspended job

https://gerrit.wikimedia.org/r/1114001

gerritbot added a comment.Feb 21 2025, 9:50 AM

Change #1121587 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] mediawiki: bump version

https://gerrit.wikimedia.org/r/1121587

gerritbot added a comment.Feb 21 2025, 9:56 AM

Change #1121587 merged by Brouberol:

[operations/deployment-charts@master] mediawiki: bump version

https://gerrit.wikimedia.org/r/1121587

brouberol added a comment.Feb 21 2025, 10:04 AM

The Cronjob and PVC have been deployed!

brouberol@deploy2002:/srv/deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy$ kube_env mediawiki-dumps-legacy-deploy dse-k8s-eqiad
brouberol@deploy2002:/srv/deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy$ kubectl get pvc -n mediawiki-dumps-legacy mediawiki-dumps-legacy-fs
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
mediawiki-dumps-legacy-fs   Bound    pvc-94490633-4041-4985-bb13-74e8764c71e6   10Gi       RWX            ceph-cephfs-dumps   53s
brouberol@deploy2002:/srv/deployment-charts/helmfile.d/dse-k8s-services/mediawiki-dumps-legacy$ kubectl get all
NAME                                                COMPLETIONS   DURATION   AGE
job.batch/mediawiki-production-dumps-job-template   0/1                      69s

I'll let @BTullis assess whether we can close.

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2025, 10:30 AM
RKemper updated the task description. (Show Details)Mar 1 2025, 2:50 AM
RKemper subscribed.EditedMar 1 2025, 2:54 AM

I believe the configmaps mentioned in T352650#10493191 still need to be implemented, as well as I think these two things:

We also need to get the files under /etc/dumps/confs and /etc/dumps/dblists into the container, so maybe we need a dumps chart for this.
We can also generate an XCom which is a list of wikis, then we can create a task group, which is a worker process for each of the wikis,

I'm basing that off of not seeing associated patches and not seeing the configmaps present in the k8s namespace. We'll have Ben back this next week to confirm if my understanding is correct, and if so we can start work on those pieces.

brouberol mentioned this in T388378: Orchestrate dumps v1 from an airflow instance.Mar 10 2025, 10:33 AM
brouberol changed the status of subtask T388378: Orchestrate dumps v1 from an airflow instance from Open to In Progress.Mar 11 2025, 1:05 PM
BTullis added a subtask: T388761: scap needs to be k8s-cluster aware.Mar 13 2025, 10:20 AM
CodeReviewBot added a project: Patch-For-Review.Mar 13 2025, 3:25 PM

btullis opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/153

Add the bzip2 package to the mediawiki-debug image for legacy dumps

BTullis closed subtask T388707: Render the dumps configuration files into configmaps in the mediawiki-dumps-legacy namespace as Resolved.Mar 13 2025, 5:56 PM
gerritbot added a comment.Mar 14 2025, 12:06 PM

Change #1127879 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] mediawiki: Update the dumps job template to support write access

https://gerrit.wikimedia.org/r/1127879

gerritbot added a comment.Mar 14 2025, 3:09 PM

Change #1127916 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] mediawiki: Use the servergroup to configure the dumps feature flag

https://gerrit.wikimedia.org/r/1127916

CodeReviewBot added a comment.Mar 17 2025, 4:11 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/153

Add the bzip2 package to the mediawiki-debug image for legacy dumps

gerritbot added a comment.Mar 18 2025, 10:17 AM

Change #1127879 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Update the dumps job template to support write access

https://gerrit.wikimedia.org/r/1127879

BTullis mentioned this in T319432: Migrate WMF production from PHP 7.4 to PHP 8.1.Mar 20 2025, 7:10 AM
BTullis mentioned this in T389470: CephOSDClusterInWarning.Mar 20 2025, 10:28 AM
CodeReviewBot added a comment.Mar 20 2025, 10:45 AM

btullis opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/154

Add curl to the mediawiki-debug image for use with dumps

CodeReviewBot added a comment.Mar 20 2025, 12:39 PM

btullis closed https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/154

Add curl to the mediawiki-debug image for use with dumps

CodeReviewBot added a comment.Apr 3 2025, 10:31 AM

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/refinery to the trusted runners

CodeReviewBot added a comment.Apr 3 2025, 10:40 AM

btullis updated https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/sync-utils to the trusted runners

CodeReviewBot added a comment.Apr 3 2025, 2:43 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/sync-utils to the trusted runners

BTullis mentioned this in T391087: Investigate an incident where the airflow-test-k8s database was wiped.Apr 4 2025, 10:41 AM
BTullis closed subtask T390067: Deploy the dumps monitor script, which generates the html status files, to the dse-k8s cluster as Resolved.May 12 2025, 1:35 PM
BTullis added a parent task: T88728: Improve Wikimedia dumping infrastructure.May 13 2025, 12:14 PM
BTullis mentioned this in T237361: Discuss common needs in a job manager/scheduler.May 14 2025, 6:05 PM
BTullis mentioned this in T149894: do proof of concept dump of one wiki via analytics infra.May 14 2025, 6:09 PM
BTullis mentioned this in T143205: Draft usage scenarios for job/workflow manager.
BTullis mentioned this in T143206: List requirements needed for task/job/workflow manager..May 14 2025, 6:11 PM
BTullis mentioned this in T143207: Evaluate software packages for job/task/workflow management.
BTullis mentioned this in T146070: Review progress of dumps rewrite.
BTullis mentioned this in T128520: What already available software can we build on for a job scheduler?.
BTullis mentioned this in T128755: What monitoring of dumps can we provide to users?.May 14 2025, 6:14 PM
BTullis mentioned this in T128752: What sort of internal storage do we want for the new dumps sytem?.May 14 2025, 6:16 PM
BTullis mentioned this in T394389: Migrate the additional dump types from snapshot1016 to Airflow.May 15 2025, 10:08 AM
brouberol changed the status of subtask T394647: Repurpose spare snapshot servers as dse-k8s-workers from Open to In Progress.May 19 2025, 12:05 PM
brouberol closed subtask T394647: Repurpose spare snapshot servers as dse-k8s-workers as Resolved.May 19 2025, 6:17 PM
brouberol reopened subtask T394647: Repurpose spare snapshot servers as dse-k8s-workers as In Progress.May 19 2025, 6:24 PM
brouberol closed subtask T394647: Repurpose spare snapshot servers as dse-k8s-workers as Resolved.May 20 2025, 9:53 AM
BTullis closed subtask T388761: scap needs to be k8s-cluster aware as Resolved.Jun 25 2025, 2:54 PM
BTullis mentioned this in T397848: Switch over the Dumps_v1 system to run from Airflow instead of snapshot servers.Jun 25 2025, 3:13 PM
BTullis closed subtask T394389: Migrate the additional dump types from snapshot1016 to Airflow as Resolved.Jun 26 2025, 10:52 AM
gerritbot added a comment.Jun 27 2025, 1:38 PM

Change #1127916 abandoned by Btullis:

[operations/deployment-charts@master] mediawiki: Use the servergroup to configure the dumps feature flag

Reason:

No longer required.

https://gerrit.wikimedia.org/r/1127916

BTullis closed subtask T397858: Pre-load the 20250601 dumps to the cephfs volume to use as prefetch data for the 20250701 dumps as Resolved.Jun 30 2025, 3:08 PM
BTullis mentioned this in T398436: Future improvements to the Dumps on Airflow configuration - tracking task.Jul 2 2025, 12:19 PM
BTullis closed subtask T388378: Orchestrate dumps v1 from an airflow instance as Resolved.Jul 2 2025, 12:47 PM
BTullis closed this task as Resolved.Jul 2 2025, 1:12 PM
BTullis claimed this task.
BTullis edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Patch-For-Review, Data-Platform-SRE.

I have created a sub-ticket of T398438: Decommission or recommission all snapshot and dumpsdata servers - although I don't propose that we let it dfer our closing of this epic.

Similarly, I created T398436: Future improvements to the Dumps on Airflow configuration - tracking task as a new tracking task for improvements that we were not able to make to the Dumps on Airflow system before go-live. I moved most of the outstanding tickets from this tree to that new one.
So I think that we can close this ticket now.

BTullis moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jul 2 2025, 1:12 PM
BTullis closed subtask T397848: Switch over the Dumps_v1 system to run from Airflow instead of snapshot servers as Resolved.Jul 2 2025, 6:29 PM
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.Jul 4 2025, 3:57 PM
BTullis mentioned this in T398756: No wikidata dumps this week (20250630).Jul 7 2025, 9:37 AM
BTullis mentioned this in T389199: Fix a performance regression affecting wikibase dumps when using mediawiki analytics replica of s8 - dbstore1009.Jul 18 2025, 11:08 AM
brouberol closed subtask T398438: Decommission or recommission all snapshot and dumpsdata servers as Resolved.Aug 6 2025, 2:00 PM
BTullis reopened subtask T398438: Decommission or recommission all snapshot and dumpsdata servers as Open.Aug 6 2025, 3:49 PM
gerritbot added a comment.Aug 18 2025, 8:37 AM

Change #1179639 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add another 10 TB to the legacy dumps pvc

https://gerrit.wikimedia.org/r/1179639

gerritbot added a project: Patch-For-Review.Aug 18 2025, 8:37 AM
gerritbot added a comment.Aug 18 2025, 8:42 AM

Change #1179639 merged by jenkins-bot:

[operations/deployment-charts@master] Add another 10 TB to the legacy dumps pvc

https://gerrit.wikimedia.org/r/1179639

BTullis closed subtask T401425: Investigate memcache errors during wikidata and commons dumps runs as Resolved.Sep 8 2025, 9:31 AM
BTullis mentioned this in T384381: Airflow jobs to do monthly XML dumps.Sep 9 2025, 1:13 PM
BTullis mentioned this in T404084: Move the dumps_v1 DAGs from the Airflow test_k8s instance to the main instance.Sep 9 2025, 1:29 PM
BTullis mentioned this in T403793: 20250901 enwiki dump is triple the normal size.Sep 12 2025, 1:02 PM
BTullis closed subtask T398438: Decommission or recommission all snapshot and dumpsdata servers as Resolved.Sep 30 2025, 10:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits