Page MenuHomePhabricator
Create Task
Maniphest T434188

eqiad/codfw: 3 trixie VMs for rdb-lock
Closed, ResolvedPublic
Actions

Description

Site/Location: eqiad and codfw
Number of systems: 3 in each datacentre
Service: rdb-lock T427999: Redis solution for LockManager
Networking Requirements: internal IP, access from WikiKube
Processor Requirements: 4vCPU
Memory: 5GB
Disks: 30GB

Rely on client-side quorum, so spread'em out :D

Details

Other Assignee
Clement_Goubert
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
external_services: Just in case, add redis-lockClément Goubertoperations/puppetproduction+13 -0
site.pp: Configure rdb-lock instancesClément Goubertoperations/puppetproduction+3 -5
rdb-lock: Create puppet role redis::lock::instanceClément Goubertoperations/puppetproduction+27 -0
rdb-lock: Prepare adding VMs for new clusterClément Goubertoperations/puppetproduction+22 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Aug 6 2026, 11:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 6 2026, 11:55 AM
Clement_Goubert triaged this task as Medium priority.Aug 6 2026, 11:56 AM
Clement_Goubert updated Other Assignee, added: Clement_Goubert.
Clement_Goubert added projects: ServiceOps, ServiceOps-Datastores, ServiceOps-Mediawiki.
Clement_Goubert moved this task from Inbox to Needs Info / Blocked on the ServiceOps board.
MoritzMuehlenhoff subscribed.Wed, Aug 12, 10:29 AM

Looks good, for eqiad please use A, B, C and for codfw B, C, D

Clement_Goubert changed the task status from Open to In Progress.Wed, Aug 12, 10:54 AM
Clement_Goubert claimed this task.
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert moved this task from Needs Info / Blocked to In Progress on the ServiceOps board.
gerritbot added a comment.Wed, Aug 12, 11:33 AM

Change #1324692 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] rdb-lock: Prepare adding VMs for new cluster

https://gerrit.wikimedia.org/r/1324692

gerritbot added a project: Patch-For-Review.Wed, Aug 12, 11:33 AM

Change #1324693 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] rdb-lock: Create puppet role redis::lock::instance

https://gerrit.wikimedia.org/r/1324693

gerritbot added a comment.Wed, Aug 12, 11:33 AM

Change #1324694 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] site.pp: Configure rdb-lock instances

https://gerrit.wikimedia.org/r/1324694

gerritbot added a comment.Wed, Aug 12, 1:07 PM

Change #1324692 merged by Clément Goubert:

[operations/puppet@production] rdb-lock: Prepare adding VMs for new cluster

https://gerrit.wikimedia.org/r/1324692

ops-monitoring-bot added a comment.Wed, Aug 12, 1:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock1001.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Aug 12, 1:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock2001.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Aug 12, 1:53 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock1001.eqiad.wmnet with OS trixie completed:

  • rdb-lock1001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121338_cgoubert_1012335_rdb-lock1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Aug 12, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock2001.codfw.wmnet with OS trixie completed:

  • rdb-lock2001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121349_cgoubert_1013090_rdb-lock2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Aug 12, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock1002.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Aug 12, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock2002.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Aug 12, 2:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock1002.eqiad.wmnet with OS trixie completed:

  • rdb-lock1002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121423_cgoubert_1026127_rdb-lock1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Aug 12, 2:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock1003.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Aug 12, 2:57 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock2002.codfw.wmnet with OS trixie completed:

  • rdb-lock2002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121439_cgoubert_1029247_rdb-lock2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Aug 12, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock1003.eqiad.wmnet with OS trixie completed:

  • rdb-lock1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608121503_cgoubert_1040420_rdb-lock1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Ladsgroup awarded a token.Thu, Aug 13, 9:42 AM
Ladsgroup subscribed.
ops-monitoring-bot added a comment.Fri, Aug 14, 10:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2003 for host rdb-lock2003.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Fri, Aug 14, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2003 for host rdb-lock2003.codfw.wmnet with OS trixie completed:

  • rdb-lock2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608141044_cgoubert_1614955_rdb-lock2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Clement_Goubert closed this task as Resolved.Fri, Aug 14, 11:05 AM
Clement_Goubert added a parent task: T427999: Redis solution for LockManager.

Resolving, VMs are now created, configuration followup in parent task

gerritbot added a comment.Fri, Aug 14, 11:21 AM

Change #1324693 merged by Clément Goubert:

[operations/puppet@production] rdb-lock: Create puppet role redis::lock::instance

https://gerrit.wikimedia.org/r/1324693

gerritbot added a comment.Fri, Aug 14, 11:30 AM

Change #1325875 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] external_services: Just in case, add redis-lock

https://gerrit.wikimedia.org/r/1325875

gerritbot added a comment.Mon, Aug 17, 2:39 PM

Change #1324694 merged by Clément Goubert:

[operations/puppet@production] site.pp: Configure rdb-lock instances

https://gerrit.wikimedia.org/r/1324694

gerritbot added a comment.Mon, Aug 17, 2:39 PM

Change #1325875 merged by Clément Goubert:

[operations/puppet@production] external_services: Just in case, add redis-lock

https://gerrit.wikimedia.org/r/1325875

Maintenance_bot removed a project: Patch-For-Review.Mon, Aug 17, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits