Page MenuHomePhabricator
Create Task
Maniphest T419976

Upgrade redis_misc hosts to Debian Trixie (Redis 8.0)
Closed, ResolvedPublic
Actions

Description

Redis recommends upgrading one major version at a time, which would mean upgrading the servers twice;

  • Debian Bookworm and Redis 7.0
  • Debian Trixie and Redis 8.0

However, given that we have active hardware refreshes in both datacenters T418918: rdb101[56] implementation tracking T418924: rdb201[34] implementation tracking, we have an opportunity to skip Bookworm entirely and move directly to Trixie, provided that there are no concerns from teams currently using redis_misc.

If possible we should take the chance to T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets during the OS upgrade reimage.

Services using redis_misc

Pair 1

PortDBUsagePoC/tag
63780IDP (CAS-SSO) Production (rdb2011 only) Infrastructure-Foundations
63781IDP (CAS-SSO) Test (rdb2011 only)
63790changeprop / cpjobqueue MediaWiki-API-Platform-Team
63791rest-gatewayMediaWiki-API-Platform-Team
63800redioscopeMediaWiki-API-Platform-Team
63800Ratelimit MediaWiki-API-Platform-Team
63810filebackend.php (redisLockManager)MediaWiki-Core-Platform-Team
63820filebackend.php (redisLockManager)

Pair 2

PortDBUsagePoC/tag
63782, 3Netbox tasks (db 2) and Netbox caching (db 3)- *(rdb1015 only)netbox Infrastructure-Foundations
63790changeprop / cpjobqueue MediaWiki-API-Platform-Team
63791rest-gatewayMediaWiki-API-Platform-Team
63800redioscopeMediaWiki-API-Platform-Team
63800Ratelimit MediaWiki-API-Platform-Team
63810filebackend.php (redisLockManager)MediaWiki-Core-Platform-Team
63820docker-registryServiceOps
How?

The new hosts will be reimaged directly to Trixie (Redis 8.0) and services migrated one by one (or more). If any issues come up, we can simply revert the service back to the previous servers (point it back at the old rdb hosts).

Open Questions to owners before proceeding
  • Shall we migrate the data?
    • We need to decide whether to migrate existing Redis data to the new hosts or start fresh. Given that Redis is generally considered to be ephemeral storage, not migrating the data should be an acceptable risk.
    • How does data persistence affect your service?
  • Application behavior under server unavailability
    • It is currently unknown how each of the services above behaves if their Redis storage becomes unavailable. Maybe this could be a good opportunity to test this in a controlled manner, since rolling back to existing hosts would be easy to do.
    • Can your service tolerate a brief Redis unavailability?
Dashboard improvements
  • improve grafana dashboards: https://grafana-rw.wikimedia.org/d/000000174/redis
    • Role-aware filtering via $role
    • cache hit ratio, memory total, connected clients
    • new panels:
      • connected clients & replicas
      • replication lag
      • cache hits vs misses
    • reorganise layout
Wikitech Updates

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ProductionServices.php: switch filebackend.php back to rdb1013Effie Mouzelioperations/mediawiki-configmaster+2 -2
mediawiki_common: update IP for rdb1014Effie Mouzelioperations/deployment-chartsmaster+2 -2
alias.yaml: retire the old codfw redis serversEffie Mouzelioperations/puppetproduction+0 -0
ProductionServices.php: switch filebackend.php to rdb2013:6381Effie Mouzelioperations/mediawiki-configmaster+1 -1
ProductionServices.php: switch filebackend.php to rdb2011:6382Effie Mouzelioperations/mediawiki-configmaster+1 -1
changeprop-jobqueue: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+1 -1
ProductionServices.php: switch filebackend.php to rdb2011:6381Effie Mouzelioperations/mediawiki-configmaster+1 -1
changeprop: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+1 -1
rest-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+1 -1
redioscope: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+2 -1
api-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+1 -1
ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+2 -2
ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)Effie Mouzelioperations/deployment-chartsmaster+1 -1
mediawiki-common: add rdb2011 and rdb2012 IPsEffie Mouzelioperations/deployment-chartsmaster+6 -0
IDP: Redis migrationSlyngshedeoperations/dnsmaster+1 -1
idp: migrate IDP to Redis 8Effie Mouzelioperations/puppetproduction+1 -1
redis::master: Remove obsolete code only used for old ferm serviceMuehlenhoffoperations/puppetproduction+0 -5
idp_test: switch to rdb2011Effie Mouzelioperations/puppetproduction+1 -1
site.pp: add role for rdb2011 and rdb2012Effie Mouzelioperations/puppetproduction+16 -4
redis::master: Remove obsolete code only used for old ferm serviceMuehlenhoffoperations/puppetproduction+0 -5
redis::master: Pass ports as an array, not a stringMuehlenhoffoperations/puppetproduction+2 -3
site.pp: add role for rdb2011Effie Mouzelioperations/puppetproduction+15 -9
redis::master: Move to firewall::serviceMuehlenhoffoperations/puppetproduction+14 -6
redis::slave: Move to firewall::serviceMuehlenhoffoperations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Apr 21 2026, 11:04 AM

Change #1275752 merged by Muehlenhoff:

[operations/puppet@production] redis::slave: Move to firewall::service

https://gerrit.wikimedia.org/r/1275752

MoritzMuehlenhoff added a comment.Apr 21 2026, 11:21 AM

The current Redis slaves have been migrated to use firewall::service by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275752 which was confirmed to be a NOP on existing hosts. This unblocks the use of nftables for the new Redis trixie hosts:

jmm@puppetdb1003:~$ nftables-compat-check.py rdb1012.eqiad.wmnet
Warning: This server uses defs_from_etcd, which isn't implemented for nft yet

All firewall services are compatible with nftables. The full list is:
{'full_monitoring_metrics_access_udp', 'ssh_from_cumin_masters', 'redis_slave_role', 'ssh_from_bastion', 'full_monitoring_metrics_access_tcp'}
gerritbot added a comment.Apr 21 2026, 1:52 PM

Change #1275757 merged by Muehlenhoff:

[operations/puppet@production] redis::master: Move to firewall::service

https://gerrit.wikimedia.org/r/1275757

MoritzMuehlenhoff added a comment.Apr 21 2026, 2:10 PM

The current Redis masters have been migrated to use firewall::service by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275757 which was confirmed to be a NOP on existing hosts. This unblocks the use of nftables for the new Redis trixie hosts as well:

jmm@puppetdb1003:~$ nftables-compat-check.py rdb1013.eqiad.wmnet
Warning: This server uses defs_from_etcd, which isn't implemented for nft yet

All firewall services are compatible with nftables. The full list is:
{'ssh_from_cumin_masters', 'redis_master_role', 'ssh_from_bastion', 'full_monitoring_metrics_access_tcp', 'full_monitoring_metrics_access_udp'}
Jhancock.wm closed subtask Unknown Object (Task) as Resolved.Apr 21 2026, 4:57 PM
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Apr 24 2026, 12:53 PM
gerritbot added a comment.Apr 27 2026, 8:49 AM

Change #1275502 abandoned by Effie Mouzeli:

[operations/puppet@production] site.pp: add role for rdb2011

Reason:

rebase mumbo jumbo

https://gerrit.wikimedia.org/r/1275502

gerritbot added a comment.Apr 27 2026, 8:55 AM

Change #1277429 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] site.pp: add role for rdb2011

https://gerrit.wikimedia.org/r/1277429

Ladsgroup added a comment.Apr 29 2026, 10:30 PM

I'm updating the code for redis lock manager right now (for T366938: Reduce relying on database locks ) and if you give me like a week or two, I make the system much more robust and safer towards issues.

gerritbot added a comment.May 4 2026, 11:13 AM

Change #1282308 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] redis::master: Remove obsolete code only used for old ferm service

https://gerrit.wikimedia.org/r/1282308

gerritbot added a comment.May 4 2026, 11:36 AM

Change #1282315 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] redis::master: Pass ports as an array, not a string

https://gerrit.wikimedia.org/r/1282315

gerritbot added a comment.May 4 2026, 1:00 PM

Change #1282315 abandoned by Muehlenhoff:

[operations/puppet@production] redis::master: Pass ports as an array, not a string

Reason:

1282311 was merged instead

https://gerrit.wikimedia.org/r/1282315

gerritbot added a comment.May 4 2026, 1:04 PM

Change #1282353 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] redis::master: Remove obsolete code only used for old ferm service

https://gerrit.wikimedia.org/r/1282353

gerritbot added a comment.May 4 2026, 1:09 PM

Change #1282308 abandoned by Muehlenhoff:

[operations/puppet@production] redis::master: Remove obsolete code only used for old ferm service

Reason:

Replaced by 1282353

https://gerrit.wikimedia.org/r/1282308

gerritbot added a comment.May 6 2026, 11:22 AM

Change #1277429 merged by Effie Mouzeli:

[operations/puppet@production] site.pp: add role for rdb2011 and rdb2012

https://gerrit.wikimedia.org/r/1277429

ops-monitoring-bot added a comment.May 6 2026, 11:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb2011.codfw.wmnet with OS trixie

gerritbot added a comment.May 6 2026, 11:58 AM

Change #1283733 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] idp_test: switch to rdb2011

https://gerrit.wikimedia.org/r/1283733

ops-monitoring-bot added a comment.May 6 2026, 12:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb2012.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.May 6 2026, 12:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb2011.codfw.wmnet with OS trixie completed:

  • rdb2011 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061157_jiji_582750_rdb2011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jijiki added a subscriber: Blake.May 6 2026, 12:42 PM

@Blake and I reimaged rdb2011 and rdb2012 today, and submitted an idp test patch. @SLyngshede-WMF please merge the patch and let us know how it went, the soonest the better :)

ops-monitoring-bot added a comment.May 6 2026, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb2012.codfw.wmnet with OS trixie completed:

  • rdb2012 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061224_jiji_609364_rdb2012.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MLechvien-WMF moved this task from Scheduled (this Q) to In Progress on the ServiceOps board.May 7 2026, 2:13 PM
gerritbot added a comment.May 8 2026, 8:49 AM

Change #1283733 merged by Slyngshede:

[operations/puppet@production] idp_test: switch to rdb2011

https://gerrit.wikimedia.org/r/1283733

jijiki updated the task description. (Show Details)May 8 2026, 8:57 AM
jijiki updated the task description. (Show Details)May 8 2026, 9:00 AM
jijiki added a comment.May 8 2026, 10:19 AM

@SLyngshede-WMF as discussed, since we accidentally wired IDP to a non primary redis host, we can migrate idp to rdb2011 directly, and wrap up the migration

jijiki updated the task description. (Show Details)May 8 2026, 10:25 AM
gerritbot added a comment.May 8 2026, 10:28 AM

Change #1285324 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] idp: migrate IDP to Redis 8

https://gerrit.wikimedia.org/r/1285324

gerritbot added a comment.May 8 2026, 11:41 AM

Change #1285336 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mediawiki-common: add rdb2011 and rdb2012 IPs

https://gerrit.wikimedia.org/r/1285336

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285339 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] redioscope: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285339

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285340 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] api-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285340

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285341 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] changeprop-jobqueue: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285341

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285342 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] changeprop: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285342

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285343 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285343

gerritbot added a comment.May 8 2026, 11:49 AM

Change #1285344 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] rest-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285344

gerritbot added a comment.May 11 2026, 5:32 AM

Change #1282353 merged by Muehlenhoff:

[operations/puppet@production] redis::master: Remove obsolete code only used for old ferm service

https://gerrit.wikimedia.org/r/1282353

gerritbot added a comment.May 11 2026, 7:49 AM

Change #1285324 merged by Slyngshede:

[operations/puppet@production] idp: migrate IDP to Redis 8

https://gerrit.wikimedia.org/r/1285324

gerritbot added a comment.May 11 2026, 7:55 AM

Change #1285547 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] IDP: Redis migration

https://gerrit.wikimedia.org/r/1285547

gerritbot added a comment.May 11 2026, 8:08 AM

Change #1285547 merged by Slyngshede:

[operations/dns@master] IDP: Redis migration

https://gerrit.wikimedia.org/r/1285547

• elukey added a comment.EditedMay 11 2026, 9:39 AM

I would do this simple step for netbox:

  • ssh on netbox1003
  • disable puppet
  • Modify the following configuration bit to use DATABASE 4 and 5
REDIS = {
    'tasks': {
        'HOST': 'rdb1013.eqiad.wmnet',
        'PORT': 6378,
        'DATABASE': 2,
    },
    'caching': {
        'HOST': 'rdb1013.eqiad.wmnet',
        'PORT': 6378,
        'DATABASE': 3,
        'DEFAULT_TIMEOUT': 300,
    },
}
  • Restart netbox and verify
  • Rollback
• elukey added a comment.May 11 2026, 1:31 PM

Checked on netbox1003 and starting from a cold cache is not an issue!

gerritbot added a comment.May 12 2026, 10:06 AM

Change #1285336 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-common: add rdb2011 and rdb2012 IPs

https://gerrit.wikimedia.org/r/1285336

gerritbot added a comment.May 13 2026, 10:54 AM

Change #1285343 merged by jenkins-bot:

[operations/deployment-charts@master] ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285343

gerritbot added a comment.May 13 2026, 11:48 AM

Change #1286875 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1286875

gerritbot added a comment.May 14 2026, 8:53 AM

Change #1286875 merged by jenkins-bot:

[operations/deployment-charts@master] ratelimit: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1286875

gerritbot added a comment.May 14 2026, 9:21 AM

Change #1285340 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285340

gerritbot added a comment.May 14 2026, 10:33 AM

Change #1285339 merged by jenkins-bot:

[operations/deployment-charts@master] redioscope: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285339

gerritbot added a comment.May 14 2026, 12:20 PM

Change #1285344 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285344

jijiki updated the task description. (Show Details)May 14 2026, 12:32 PM
jijiki updated the task description. (Show Details)May 14 2026, 12:39 PM
jijiki updated the task description. (Show Details)May 19 2026, 8:42 AM
gerritbot added a comment.May 21 2026, 9:50 AM

Change #1285342 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285342

jijiki updated the task description. (Show Details)May 21 2026, 10:01 AM
jijiki updated the task description. (Show Details)May 21 2026, 10:18 AM
gerritbot added a comment.May 21 2026, 10:26 AM

Change #1290709 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2011:6381

https://gerrit.wikimedia.org/r/1290709

gerritbot added a comment.May 21 2026, 10:35 AM

Change #1290709 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2011:6381

https://gerrit.wikimedia.org/r/1290709

Stashbot added a comment.May 21 2026, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-21T10:36:06Z] <jiji@deploy1003> Started scap sync-world: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]]

Stashbot added a comment.May 21 2026, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-21T10:37:52Z] <jiji@deploy1003> jiji: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.May 21 2026, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-21T10:44:08Z] <jiji@deploy1003> Finished scap sync-world: Backport for [[gerrit:1290709|ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s)

gerritbot added a comment.May 25 2026, 10:27 AM

Change #1285341 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1285341

gerritbot added a comment.May 25 2026, 11:53 AM

Change #1293095 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] changeprop-jobqueue: codfw: replace rdb2007 with rdb2011 (Redis 8)

https://gerrit.wikimedia.org/r/1293095

gerritbot added a comment.May 26 2026, 10:44 AM

Change #1293095 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2011:6382

https://gerrit.wikimedia.org/r/1293095

Stashbot added a comment.May 26 2026, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T10:44:49Z] <jiji@deploy1003> Started scap sync-world: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]]

Stashbot added a comment.May 26 2026, 10:46 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T10:46:42Z] <jiji@deploy1003> jiji: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.May 26 2026, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T11:00:40Z] <jiji@deploy1003> Finished scap sync-world: Backport for [[gerrit:1293095|ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s)

jijiki closed subtask T418261: rdb20[11-12] implementation tracking as Resolved.May 26 2026, 11:46 AM
MLechvien-WMF mentioned this in T418918: rdb101[56] implementation tracking.May 28 2026, 2:08 PM
gerritbot added a comment.Jun 3 2026, 11:52 AM

Change #1297110 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2013:6381

https://gerrit.wikimedia.org/r/1297110

gerritbot added a comment.Jun 3 2026, 12:42 PM

Change #1297110 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2013:6381

https://gerrit.wikimedia.org/r/1297110

Stashbot added a comment.Jun 3 2026, 12:43 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-03T12:43:13Z] <jiji@deploy1003> Started scap sync-world: Backport for [[gerrit:1297110|ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]]

Stashbot added a comment.Jun 3 2026, 12:45 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-03T12:45:14Z] <jiji@deploy1003> jiji: Backport for [[gerrit:1297110|ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

gerritbot added a comment.Jun 3 2026, 12:50 PM

Change #1297124 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] alias.yaml: retire the old codfw redis servers

https://gerrit.wikimedia.org/r/1297124

Stashbot added a comment.Jun 3 2026, 12:50 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-03T12:50:57Z] <jiji@deploy1003> Finished scap sync-world: Backport for [[gerrit:1297110|ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s)

jijiki mentioned this in T428561: decommission rdb2007-rdb2010.codfw.wmnet.Jun 9 2026, 9:58 AM
gerritbot added a comment.Jun 9 2026, 10:52 AM

Change #1297124 abandoned by Effie Mouzeli:

[operations/puppet@production] alias.yaml: retire the old codfw redis servers

Reason:

wrong

https://gerrit.wikimedia.org/r/1297124

jijiki updated the task description. (Show Details)Jun 9 2026, 10:55 AM
jijiki updated the task description. (Show Details)Jun 9 2026, 2:49 PM
jijiki added a comment.Jun 9 2026, 2:51 PM

In order to upgrade rdb1013 (and rdb1014) to Trixie, since they are not up for retirement, I will temporarily switch wmf-config/ProductionServices.php to use rdb1011 (which is up for retirement), while rdb1013 and rdb1014 are being reimaged.

All other services using rdb1013, are doing so via nutcracker, so nutcracker will take care of things during rdb1013s unavailability.

gerritbot added a comment.Jun 10 2026, 12:32 PM

Change #1300114 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mediawiki_common: update IP for rdb1014

https://gerrit.wikimedia.org/r/1300114

gerritbot added a comment.Jun 11 2026, 6:24 AM

Change #1300114 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki_common: update IP for rdb1014

https://gerrit.wikimedia.org/r/1300114

jijiki added a comment.Jun 11 2026, 7:26 AM
In T419976#12000190, @jijiki wrote:

In order to upgrade rdb1013 (and rdb1014) to Trixie, since they are not up for retirement, I will temporarily switch wmf-config/ProductionServices.php to use rdb1011 (which is up for retirement), while rdb1013 and rdb1014 are being reimaged.

All other services using rdb1013, are doing so via nutcracker, so nutcracker will take care of things during rdb1013s unavailability.

rdb1014 has been successfully reimaged to Trixie.

ops-monitoring-bot added a comment.Jun 11 2026, 7:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb1013.eqiad.wmnet with OS trixie

gerritbot added a comment.Jun 11 2026, 7:42 AM

Change #1300580 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php back to rdb1013

https://gerrit.wikimedia.org/r/1300580

jijiki mentioned this in T428858: decommission rdb101[12].eqiad.wmnet.Jun 11 2026, 8:03 AM
ops-monitoring-bot added a comment.Jun 11 2026, 8:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb1013.eqiad.wmnet with OS trixie completed:

  • rdb1013 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606110754_jiji_3160377_rdb1013.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jun 11 2026, 9:42 AM

Change #1300580 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php back to rdb1013

https://gerrit.wikimedia.org/r/1300580

Stashbot added a comment.Jun 11 2026, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-11T09:43:17Z] <jiji@deploy1003> Started scap sync-world: Backport for [[gerrit:1300580|ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]]

Stashbot mentioned this in T291916: Tracking task for Bullseye migrations in production.Jun 11 2026, 9:43 AM
Stashbot added a comment.Jun 11 2026, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-11T09:45:37Z] <jiji@deploy1003> jiji: Backport for [[gerrit:1300580|ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Jun 11 2026, 9:59 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-11T09:58:59Z] <jiji@deploy1003> Finished scap sync-world: Backport for [[gerrit:1300580|ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s)

jijiki closed this task as Resolved.Jun 11 2026, 10:31 AM
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
KineticPelagic subscribed.Jun 18 2026, 6:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits