Page MenuHomePhabricator
Create Task
Maniphest T433363

🏖️ Southward Datacenter Switchover (Sept. 2026)
Open, MediumPublic
Actions

Description

This is an umbrella task for the upcoming Sept 2026 Southward (eqiad to codfw) DC Switchover.

Switchovers take place at predictable dates; the work week of the Solar Equinox.

Important Dates

If you have any related work, please file your tasks below.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
mw-web: upsize for single-DC servingSlyngshedeoperations/deployment-chartsmaster+8 -3
mw-api-ext: upsize for single-DC servingSlyngshedeoperations/deployment-chartsmaster+7 -2
wmnet: update CNAME records for DB masters to codfwSlyngshedeoperations/dnsmaster+29 -29
switchdc: remove parsoidSlyngshedeoperations/cookbooksmaster+0 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenSLyngshede-WMFT433363 🏖️ Southward Datacenter Switchover (Sept. 2026)
 ResolvedTrizek-WMFT433364 MoveComms support for Southward Datacentre Switchover (Sept. 2026)
 DeclinedSLyngshede-WMFT435443 Day 1 - Source datacenter depooling (Tue. Sept. 22nd - 2026)
 DeclinedSLyngshede-WMFT435462 Day 2 - Source datacenter depooling (Wed. Sept. 23nd - 2026)
 DeclinedSLyngshede-WMFT436238 Day 8 - Source datacenter pooling (Thu. Sept. 28nd - 2026)
 ResolvedFCeratto-WMFT436496 Pre switchover: Database preparations
 ResolvedFCeratto-WMFT436500 Enable codfw -> eqiad replication
 OpenFCeratto-WMFT438831 sre.mysql.depool cookbook not committing changes for es7
 OpenNoneT436501 Post DC switchover: Disable eqiad -> codfw replication
 DeclinedSLyngshede-WMFT436630 Deployment server switch over
 ResolvedSLyngshede-WMFT436781 Live-Test - September 2nd 2026
 ResolvedSLyngshede-WMFT438299 Upsize codfw for single-DC serving - Sept. 21th
 ResolvedCWilliams-WMFT438833 Improve handling of Preparing state in DC switchover
 OpenNoneT438896 Investigate high saturation in codfw
 OpenNoneT439052 October Southward Switchover Prep
 ResolvedNoneT439552 DC Switchover - MW requests to EventGate cause long queuing waiting time
 OpenEjeggT439239 CentralNotice must batch campaign mixin lookup & use stampede protection
 ResolvedMarosteguiT439653 Consider upsizing s7 replicas in codfw (vs. 13 in eqiad) following codfw high saturation
 DuplicateNoneT439654 Upsize memcached cluster in codfw following high saturation
 Openjasmine_T438970 sre.discovery.service-route traceback during depool

Event Timeline

SLyngshede-WMF created this task.Jul 28 2026, 11:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2026, 11:30 AM
SLyngshede-WMF triaged this task as Medium priority.Jul 28 2026, 11:31 AM
SLyngshede-WMF updated the task description. (Show Details)Jul 28 2026, 11:33 AM
SLyngshede-WMF created subtask T433364: MoveComms support for Southward Datacentre Switchover (Sept. 2026).Jul 28 2026, 11:41 AM
SLyngshede-WMF updated the task description. (Show Details)Jul 28 2026, 11:59 AM
SLyngshede-WMF renamed this task from 🚀 Southward Datacenter Switchover (Sept. 2026) to 🏖️ Southward Datacenter Switchover (Sept. 2026).Jul 28 2026, 12:12 PM
jasmine_ subscribed.Jul 28 2026, 9:46 PM
gerritbot added a comment.Jul 31 2026, 11:22 AM

Change #1319815 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] wmnet: update CNAME records for DB masters to codfw

https://gerrit.wikimedia.org/r/1319815

gerritbot added a project: Patch-For-Review.Jul 31 2026, 11:22 AM
gerritbot added a comment.Jul 31 2026, 11:24 AM

Change #1319816 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] geo-maps: update map default to list codfw first

https://gerrit.wikimedia.org/r/1319816

gerritbot added a comment.Jul 31 2026, 11:59 AM

Change #1319825 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] wmnet: update deployment CNAME record to deploy2003

https://gerrit.wikimedia.org/r/1319825

gerritbot added a comment.Jul 31 2026, 12:03 PM

Change #1319826 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] hieradata: update deployment_server to deploy2002

https://gerrit.wikimedia.org/r/1319826

MLechvien-WMF added a project: ServiceOps.Jul 31 2026, 1:17 PM
MLechvien-WMF moved this task from Inbox to In Progress on the ServiceOps board.
SLyngshede-WMF created subtask T435443: Day 1 - Source datacenter depooling (Tue. Sept. 22nd - 2026).Aug 20 2026, 9:00 AM
SLyngshede-WMF created subtask T435462: Day 2 - Source datacenter depooling (Wed. Sept. 23nd - 2026).Aug 20 2026, 11:23 AM
gerritbot added a comment.Aug 21 2026, 11:49 AM

Change #1319818 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/mediawiki-config@master] debug.json: order codfw (primary) DC backends first

https://gerrit.wikimedia.org/r/1319818

SLyngshede-WMF created subtask T436238: Day 8 - Source datacenter pooling (Thu. Sept. 28nd - 2026).Aug 27 2026, 11:20 AM
SLyngshede-WMF added a comment.Aug 28 2026, 8:07 AM

Dry-run test - To be fixed:

00-reduce-ttl:

DRY-RUN: No task specified, skipping task update with comment: slyngshede@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw - finished with status: **FAILURE** elapsed time: 0:00:00.014131
DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.00-reduce-ttl:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 229, in resolve
    answer = dns_resolver.query(record_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1262, in query
    return self.resolve(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1190, in resolve
    (request, answer) = resolution.next_request()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 691, in next_request
    raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: mw-parsoid.discovery.wmnet.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
    ret = self.action()
          ^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/00-reduce-ttl.py", line 25, in action
    old_ttl_sec = max(record.ttl for record in discovery.resolve())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/00-reduce-ttl.py", line 25, in <genexpr>
    old_ttl_sec = max(record.ttl for record in discovery.resolve())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 231, in resolve
    raise DiscoveryError(f"Unable to resolve {record_name} from {nameserver}") from e
spicerack.dnsdisc.DiscoveryError: Unable to resolve mw-parsoid.discovery.wmnet from dns1004.wikimedia.org

01-stop-maintenance

DRY-RUN: No task specified, skipping task update with comment: slyngshede@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw - finished with status: **FAILURE** elapsed time: 0:00:00.580888
DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.01-stop-maintenance:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
    ret = self.action()
          ^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/01-stop-maintenance.py", line 55, in action
    _wait_for_jobs_to_stop(batch_api, namespace=namespace, dry_run=self.spicerack.dry_run)
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 244, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/01-stop-maintenance.py", line 71, in _wait_for_jobs_to_stop
    raise SpicerackCheckError(f'{len(still_running)} {namespace} {jobs} still running: {names}')
spicerack.exceptions.SpicerackCheckError: 3 mw-script jobs still running: mw-script.eqiad.1w6c7bq0, mw-script.eqiad.l9oy281u, mw-script.eqiad.m2lj8b5z

04-switch-mediawiki

DRY-RUN: No task specified, skipping task update with comment: slyngshede@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw - finished with status: **FAILURE** elapsed time: 0:00:10.401518
DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.04-switch-mediawiki:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 105, in resolve_address
    return next(iter(self._resolvers.values())).query(name)[0].address
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1262, in query
    return self.resolve(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1190, in resolve
    (request, answer) = resolution.next_request()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 691, in next_request
    raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: mw-parsoid.svc.codfw.wmnet.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
    ret = self.action()
          ^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/04-switch-mediawiki.py", line 39, in action
    dnsdisc_records.check_record(record, '{name}.svc.{dc_to}.wmnet'.format(name=record, dc_to=self.dc_to))
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 244, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 180, in check_record
    expected_address = self.resolve_address(expected_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 107, in resolve_address
    raise DiscoveryError(f"Unable to resolve {name}") from e
spicerack.dnsdisc.DiscoveryError: Unable to resolve mw-parsoid.svc.codfw.wmnet

09-restore-ttl

DRY-RUN: No task specified, skipping task update with comment: slyngshede@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw - finished with status: **FAILURE** elapsed time: 0:00:00.081563
DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.09-restore-ttl:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 229, in resolve
    answer = dns_resolver.query(record_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1262, in query
    return self.resolve(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1190, in resolve
    (request, answer) = resolution.next_request()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 691, in next_request
    raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: mw-parsoid.discovery.wmnet.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
    ret = self.action()
          ^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/09-restore-ttl.py", line 19, in action
    dnsdisc_records.update_ttl(300)
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 123, in update_ttl
    self.check_ttl(ttl)
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 244, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 146, in check_ttl
    for record in self.resolve():
  File "/usr/lib/python3/dist-packages/spicerack/dnsdisc.py", line 231, in resolve
    raise DiscoveryError(f"Unable to resolve {record_name} from {nameserver}") from e
spicerack.dnsdisc.DiscoveryError: Unable to resolve mw-parsoid.discovery.wmnet from dns1004.wikimedia.org
DRY-RUN: Releasing lock for key sre.switchdc.mediawiki.09-restore-ttl with ID 685e3ecb-7cde-4273-9c8b-b1668bdfac20

Overall result

#--- Switch Datacenter for MediaWiki args=['eqiad', 'codfw'] ---#
[PASS] 00-downtime-db-readonly-checks: Downtime read-only checks on MariaDB primaries changed in Phase 3 so they don't page.
[PASS] 00-lock-scap: Lock scap.
[FAIL] 00-optional-warmup-caches: Warmup MediaWiki caches.
[FAIL] 00-reduce-ttl: Reduce TTL for various DNS Discovery entries.
[FAIL] 01-stop-maintenance: Stop MediaWiki maintenance and periodic jobs.
[PASS] 02-set-readonly: Set MediaWiki in read-only mode.
[PASS] 03-set-db-readonly: Set the old-site core DB primaries in read-only mode and check replication.
[FAIL] 04-switch-mediawiki: Switch MediaWiki active datacenter.
[PASS] 06-set-db-readwrite: Set the new-site core DB primaries in read-write mode.
[PASS] 07-set-readwrite: Set MediaWiki in read-write mode.
[PASS] 08-restart-mw-jobrunner: Restart pods in mw-jobrunner on kubernetes and envoy on jobrunner hosts in DC_FROM.
[PASS] 08-start-maintenance: Start MediaWiki maintenance jobs.
[FAIL] 09-restore-ttl: Restore TTL for various DNS Discovery entries.
[PASS] 09-run-puppet-on-db-masters: Run Puppet on all core DB primaries and remove read-only check downtimes.
[PASS] 09-unlock-scap: Unlock scap.
q - Quit
h - Help
gerritbot added a comment.Aug 28 2026, 8:18 AM

Change #1331446 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/cookbooks@master] switchdc: remove parsoid

https://gerrit.wikimedia.org/r/1331446

SLyngshede-WMF added a comment.Aug 28 2026, 8:49 AM

Errors from dry-run is solely related to parsoid, with the expection of 04-switch-mediawiki which simply doesn't handle dry-run and simply notices that the jobs remain running and therefor raise an exception.

gerritbot added a comment.Aug 31 2026, 7:07 AM

Change #1331446 merged by jenkins-bot:

[operations/cookbooks@master] switchdc: remove parsoid

https://gerrit.wikimedia.org/r/1331446

SLyngshede-WMF created subtask T436496: Pre switchover: Database preparations.Aug 31 2026, 7:20 AM
Marostegui created subtask T436500: Enable codfw -> eqiad replication.Aug 31 2026, 8:17 AM
Marostegui created subtask T436501: Post DC switchover: Disable eqiad -> codfw replication.Aug 31 2026, 8:20 AM
SLyngshede-WMF added a comment.Aug 31 2026, 12:25 PM

Removing parsoid fixed the cookbook issues:

#--- Switch Datacenter for MediaWiki args=['eqiad', 'codfw'] ---#
[PASS] 00-downtime-db-readonly-checks: Downtime read-only checks on MariaDB primaries changed in Phase 3 so they don't page.
[PASS] 00-lock-scap: Lock scap.
[PASS] 00-optional-warmup-caches: Warmup MediaWiki caches.
[PASS] 00-reduce-ttl: Reduce TTL for various DNS Discovery entries.
[FAIL] 01-stop-maintenance: Stop MediaWiki maintenance and periodic jobs.
[PASS] 02-set-readonly: Set MediaWiki in read-only mode.
[PASS] 03-set-db-readonly: Set the old-site core DB primaries in read-only mode and check replication.
[PASS] 04-switch-mediawiki: Switch MediaWiki active datacenter.
[PASS] 06-set-db-readwrite: Set the new-site core DB primaries in read-write mode.
[PASS] 07-set-readwrite: Set MediaWiki in read-write mode.
[PASS] 08-restart-mw-jobrunner: Restart pods in mw-jobrunner on kubernetes and envoy on jobrunner hosts in DC_FROM.
[PASS] 08-start-maintenance: Start MediaWiki maintenance jobs.
[PASS] 09-restore-ttl: Restore TTL for various DNS Discovery entries.
[PASS] 09-run-puppet-on-db-masters: Run Puppet on all core DB primaries and remove read-only check downtimes.
[PASS] 09-unlock-scap: Unlock scap.

Again 01-stop-maintenance checks if the jobs are running, even in DRY-RUN, causing the check to fail.

gerritbot added a comment.Aug 31 2026, 1:20 PM

Change #1332725 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/deployment-charts@master] mw-web: upsize for single-DC serving

https://gerrit.wikimedia.org/r/1332725

SLyngshede-WMF created subtask T436630: Deployment server switch over.Sep 1 2026, 11:29 AM
Jdforrester-WMF subscribed.Sep 1 2026, 11:53 AM
gerritbot added a comment.Sep 2 2026, 7:41 AM

Change #1333603 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/deployment-charts@master] mw-api-ext: upsize for single-DC serving

https://gerrit.wikimedia.org/r/1333603

SLyngshede-WMF created subtask T436781: Live-Test - September 2nd 2026.Sep 2 2026, 12:17 PM
SLyngshede-WMF closed subtask T436781: Live-Test - September 2nd 2026 as Resolved.Sep 2 2026, 4:12 PM
Trizek-WMF subscribed.Sep 4 2026, 9:42 AM
Trizek-WMF changed the status of subtask T433364: MoveComms support for Southward Datacentre Switchover (Sept. 2026) from Open to In Progress.Sep 4 2026, 2:26 PM
JMeybohm mentioned this in T437273: Regression: CentralAuth doesn't support Read-Only mode any more.Sep 8 2026, 10:32 AM
SLyngshede-WMF created subtask T438299: Upsize codfw for single-DC serving - Sept. 21th.Thu, Sep 17, 8:38 AM
FCeratto-WMF closed subtask T436496: Pre switchover: Database preparations as Resolved.Mon, Sep 21, 8:32 AM
Marostegui mentioned this in T436082: Compile and package MariaDB 10.11.19.Mon, Sep 21, 10:22 AM
gerritbot added a comment.Mon, Sep 21, 2:55 PM

Change #1333603 merged by jenkins-bot:

[operations/deployment-charts@master] mw-api-ext: upsize for single-DC serving

https://gerrit.wikimedia.org/r/1333603

gerritbot added a comment.Mon, Sep 21, 3:05 PM

Change #1332725 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web: upsize for single-DC serving

https://gerrit.wikimedia.org/r/1332725

SLyngshede-WMF closed subtask T438299: Upsize codfw for single-DC serving - Sept. 21th as Resolved.Mon, Sep 21, 3:07 PM
FCeratto-WMF changed the status of subtask T436500: Enable codfw -> eqiad replication from Open to In Progress.Tue, Sep 22, 8:56 AM
FCeratto-WMF closed subtask T436500: Enable codfw -> eqiad replication as Resolved.Tue, Sep 22, 10:15 AM
CWilliams-WMF added a subtask: T438833: Improve handling of Preparing state in DC switchover.Tue, Sep 22, 12:20 PM
bking subscribed.Tue, Sep 22, 3:35 PM
bking mentioned this in T438877: Enable Performance Governor on Cirrussearch hosts.Tue, Sep 22, 3:57 PM
CWilliams-WMF changed the status of subtask T438833: Improve handling of Preparing state in DC switchover from Open to In Progress.Tue, Sep 22, 4:49 PM
jasmine_ added a subtask: T438896: Investigate high saturation in codfw.Tue, Sep 22, 6:13 PM
Marostegui created subtask T439063: Enable codfw -> eqiad replication Act 2.Thu, Sep 24, 5:16 AM
Marostegui created subtask T439214: Pre switchover: Database preparations Act 2.Fri, Sep 25, 8:27 AM
CWilliams-WMF closed subtask T438833: Improve handling of Preparing state in DC switchover as Resolved.Fri, Sep 25, 10:40 AM
MLechvien-WMF added a subtask: T438970: sre.discovery.service-route traceback during depool.Fri, Sep 25, 12:21 PM
jasmine_ added a subtask: T439052: October Southward Switchover Prep.Tue, Sep 29, 7:41 PM
jasmine_ removed a subtask: T439052: October Southward Switchover Prep.Tue, Sep 29, 7:46 PM
Trizek-WMF closed subtask T433364: MoveComms support for Southward Datacentre Switchover (Sept. 2026) as Resolved.Wed, Sep 30, 12:12 PM
SLyngshede-WMF closed subtask T435443: Day 1 - Source datacenter depooling (Tue. Sept. 22nd - 2026) as Declined.Wed, Sep 30, 1:30 PM
SLyngshede-WMF closed subtask T435462: Day 2 - Source datacenter depooling (Wed. Sept. 23nd - 2026) as Declined.
SLyngshede-WMF closed subtask T436238: Day 8 - Source datacenter pooling (Thu. Sept. 28nd - 2026) as Declined.
SLyngshede-WMF closed subtask T436630: Deployment server switch over as Declined.
hnowlan mentioned this in T439010: codfw power outage.Thu, Oct 1, 11:37 AM
SLyngshede-WMF removed a subtask: T439063: Enable codfw -> eqiad replication Act 2.Tue, Oct 6, 7:06 AM
SLyngshede-WMF removed a subtask: T439214: Pre switchover: Database preparations Act 2.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits