Page MenuHomePhabricator
Create Task
Maniphest T418918

rdb101[56] implementation tracking
Closed, ResolvedPublic
Actions

Description

rdb101[56] implementation tracking

This task is to track the service implementation of serviceops host(s) rdb101[56]

Pair 2

PortDBUsagePoC/tag
63782, 3Netbox tasks (db 2) and Netbox caching (db 3) - *(rdb1015 only)netbox Infrastructure-Foundations
63790changeprop / cpjobqueue MW-Interfaces-Team
63791rest-gatewayMW-Interfaces-Team
63800redioscope MW-Interfaces-Team
63800Ratelimit MW-Interfaces-Team
63810filebackend.php (redisLockManager)MediaWiki-Platform-Team
63820docker-registry ServiceOps new

IP addresses

HostnameIPv4IPv6
rdb1011.eqiad.wmnet 10.64.0.362620:0:861:101:10:64:0:36
rdb1012.eqiad.wmnet 10.64.48.492620:0:861:107:10:64:48:49
rdb1015.eqiad.wmnet10.64.0.92620:0:861:101:10:64:0:9
rdb1016.eqiad.wmnet10.64.133.62620:0:861:10c:10:64:133:6

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Mar 3 2026, 7:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2026, 7:32 PM
RobH unsubscribed.Mar 3 2026, 7:32 PM
Raine changed the task status from Open to Stalled.Mar 4 2026, 12:30 PM
Raine triaged this task as Medium priority.
Raine subscribed.

Un-stall when racking task done.

Raine moved this task from Inbox to Needs Info / Blocked on the ServiceOps new board.Mar 4 2026, 12:31 PM
MLechvien-WMF mentioned this in T419212: Upgrade ServiceOps roles from Bullseye to Debian Trixie.Mar 6 2026, 10:53 AM
jijiki mentioned this in T419976: Upgrade redis_misc hosts to Debian Trixie (Redis 8.0).Mar 13 2026, 12:42 PM
MLechvien-WMF changed the task status from Stalled to Open.Fri, May 22, 10:34 AM
MLechvien-WMF edited projects, added: ServiceOps new (Next quarter); removed: ServiceOps new.
MLechvien-WMF subscribed.

This is unstalled now and ready to be picked up

MLechvien-WMF removed Clement_Goubert as the assignee of this task.Thu, May 28, 2:08 PM
MLechvien-WMF added a subscriber: jijiki.

@jijiki are you handling that task too as part of T419976: Upgrade redis_misc hosts to Debian Trixie (Redis 8.0) ?

jijiki added a comment.Wed, Jun 3, 12:47 PM
In T418918#11963447, @MLechvien-WMF wrote:

@jijiki are you handling that task too as part of T419976: Upgrade redis_misc hosts to Debian Trixie (Redis 8.0) ?

I will see if I can work on it next week

jijiki mentioned this in T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.Tue, Jun 9, 10:56 AM
gerritbot added a comment.Tue, Jun 9, 11:06 AM

Change #1299455 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] site.pp: reimage rdb1015 and rdb1016 as redis servers

https://gerrit.wikimedia.org/r/1299455

gerritbot added a project: Patch-For-Review.Tue, Jun 9, 11:06 AM
jijiki updated the task description. (Show Details)Tue, Jun 9, 11:07 AM
gerritbot added a comment.Tue, Jun 9, 11:18 AM

Change #1299458 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] changeprop: switch to rdb1015 (staging)

https://gerrit.wikimedia.org/r/1299458

gerritbot added a comment.Tue, Jun 9, 11:22 AM

Change #1299459 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] redioscope: switch to rdb1015

https://gerrit.wikimedia.org/r/1299459

gerritbot added a comment.Tue, Jun 9, 11:24 AM

Change #1299460 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] changeprop: switch to rdb1015 (staging)

https://gerrit.wikimedia.org/r/1299460

gerritbot added a comment.Tue, Jun 9, 11:25 AM

Change #1299462 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] changeprop-jobqueue: switch to rdb1015

https://gerrit.wikimedia.org/r/1299462

gerritbot added a comment.Tue, Jun 9, 11:33 AM

Change #1299455 merged by Effie Mouzeli:

[operations/puppet@production] site.pp: reimage rdb1015 and rdb1016 as redis servers

https://gerrit.wikimedia.org/r/1299455

gerritbot added a comment.Tue, Jun 9, 11:37 AM

Change #1299464 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] ratelimit: switch to rdb1015

https://gerrit.wikimedia.org/r/1299464

gerritbot added a comment.Tue, Jun 9, 11:47 AM

Change #1299466 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] rest-gateway: switch to rdb1015

https://gerrit.wikimedia.org/r/1299466

gerritbot added a comment.Tue, Jun 9, 11:48 AM

Change #1299467 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] docker-registry: switch to rdb1015

https://gerrit.wikimedia.org/r/1299467

jijiki updated the task description. (Show Details)Tue, Jun 9, 11:53 AM
gerritbot added a comment.Tue, Jun 9, 11:54 AM

Change #1299468 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2015:6381

https://gerrit.wikimedia.org/r/1299468

gerritbot added a comment.Tue, Jun 9, 11:59 AM

Change #1299470 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mediawiki-common: add rdb1015 rdb1016

https://gerrit.wikimedia.org/r/1299470

ops-monitoring-bot added a comment.Tue, Jun 9, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, Jun 9, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie

gerritbot added a comment.Tue, Jun 9, 12:33 PM

Change #1299458 abandoned by Effie Mouzeli:

[operations/deployment-charts@master] changeprop: switch to rdb1015 (staging)

https://gerrit.wikimedia.org/r/1299458

ops-monitoring-bot added a comment.Tue, Jun 9, 12:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie completed:

  • rdb1016 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606091240_jiji_2357571_rdb1016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Jun 9, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie completed:

  • rdb1015 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606091245_jiji_2356908_rdb1015.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Tue, Jun 9, 2:39 PM

Change #1299519 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] netbox: switch to rdb1015

https://gerrit.wikimedia.org/r/1299519

jijiki updated the task description. (Show Details)Tue, Jun 9, 2:41 PM
gerritbot added a comment.Tue, Jun 9, 3:01 PM

Change #1299519 merged by Effie Mouzeli:

[operations/puppet@production] netbox: switch to rdb1015

https://gerrit.wikimedia.org/r/1299519

gerritbot added a comment.Tue, Jun 9, 3:16 PM

Change #1299470 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-common: add rdb1015 rdb1016 #1

https://gerrit.wikimedia.org/r/1299470

gerritbot added a comment.Tue, Jun 9, 3:24 PM

Change #1299468 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: switch filebackend.php to rdb2015:6381 #2

https://gerrit.wikimedia.org/r/1299468

Stashbot added a comment.Tue, Jun 9, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-09T15:25:25Z] <jiji@deploy1003> Started scap sync-world: Backport for [[gerrit:1299468|ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]]

Stashbot mentioned this in T291916: Tracking task for Bullseye migrations in production.Tue, Jun 9, 3:25 PM
Stashbot added a comment.Tue, Jun 9, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-09T15:32:47Z] <jiji@deploy1003> Finished scap sync-world: Backport for [[gerrit:1299468|ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s)

gerritbot added a comment.Tue, Jun 9, 4:21 PM

Change #1299460 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: switch to rdb1015 #4

https://gerrit.wikimedia.org/r/1299460

gerritbot added a comment.Tue, Jun 9, 4:28 PM

Change #1299462 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: switch to rdb1015 #5

https://gerrit.wikimedia.org/r/1299462

gerritbot added a comment.Tue, Jun 9, 4:39 PM

Change #1299464 merged by jenkins-bot:

[operations/deployment-charts@master] ratelimit: switch to rdb1015 #6

https://gerrit.wikimedia.org/r/1299464

gerritbot added a comment.Tue, Jun 9, 4:45 PM

Change #1299459 merged by jenkins-bot:

[operations/deployment-charts@master] redioscope: switch to rdb1015 #7

https://gerrit.wikimedia.org/r/1299459

gerritbot added a comment.Tue, Jun 9, 4:56 PM

Change #1299466 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: switch to rdb1015 #8

https://gerrit.wikimedia.org/r/1299466

jijiki updated the task description. (Show Details)Tue, Jun 9, 4:58 PM
gerritbot added a comment.Wed, Jun 10, 11:09 AM

Change #1299467 merged by Effie Mouzeli:

[operations/puppet@production] docker-registry: switch to rdb1015 #3

https://gerrit.wikimedia.org/r/1299467

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 10, 11:31 AM
ops-monitoring-bot added a comment.Wed, Jun 10, 12:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host rdb1014.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Jun 10, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host rdb1014.eqiad.wmnet with OS trixie completed:

  • rdb1014 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606101245_jiji_2797735_rdb1014.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MLechvien-WMF assigned this task to jijiki.Wed, Jun 10, 7:42 PM
MLechvien-WMF edited projects, added: ServiceOps new; removed: ServiceOps new (Next quarter).
MLechvien-WMF moved this task from Needs Info / Blocked to In Progress on the ServiceOps new board.
jijiki closed this task as Resolved.Thu, Jun 11, 11:16 AM
jijiki updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits