Page MenuHomePhabricator
Create Task
Maniphest T368874

Productionize dbproxy102[89]
Closed, ResolvedPublic
Actions

Description

These hosts will replace dbproxy1020 and dbproxy1021

  • dbproxy1028 to replace dbproxy1020
  • dbproxy1029 to replace dbproxy1021

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dbproxy: update grants with ip and fqdnoperations/puppetproduction+60 -62
dbproxy: switch CNAMEsoperations/dnsmaster+1 -1
wmnet: Switch m5-master proxyoperations/dnsmaster+1 -1
wmnet: Switchover m3-masteroperations/dnsmaster+1 -1
dbproxy1028: Enable notificationsoperations/puppetproduction+0 -1
production-m5.sql.erb: Replace dbproxy1021 with dbproxy1029operations/puppetproduction+28 -28
report_users.sh: Add dbproxy102[89]operations/softwaremaster+2 -0
production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028operations/puppetproduction+22 -22
dbproxy1028: Update hostsoperations/puppetproduction+2 -3
dbproxy: mirrors hieradataoperations/puppetproduction+12 -8
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjcrespoT376916 Upgrade backup hosts to Debian Bookworm 12.X
 ResolvedjcrespoT383902 Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them
Unknown Object (Task)
 ResolvedJclark-ctrT365485 Q4:rack/setup/install dbproxy102[89]
Unknown Object (Task)
 ResolvedMarosteguiT373579 Productionize db22[21-40]
 ResolvedMarosteguiT374623 Decommission db21[21-40]
 ResolvedMarosteguiT368874 Productionize dbproxy102[89]
 ResolvedRequestJclark-ctrT383025 decommission dbproxy1020.eqiad.wmnet
 ResolvedRequestJclark-ctrT383033 decommission dbproxy1021.eqiad.wmnet
 ResolvedMarosteguiT367380 Productionize dbproxy200[5-8]
 ResolvedMarosteguiT381962 Decommission dbproxy200[1-4]
 ResolvedRequestJhancock.wmT382867 decommission dbproxy2001.codfw.wmnet
 ResolvedRequestJhancock.wmT382868 decommission dbproxy2002.codfw.wmnet
 ResolvedRequestJhancock.wmT382875 decommission dbproxy2003.codfw.wmnet
 ResolvedRequestJhancock.wmT382877 decommission dbproxy2004.codfw.wmnet
 ResolvedRequestJhancock.wmT383388 decommission db2123.codfw.wmnet
 ResolvedRequestJhancock.wmT383395 decommission db2126.codfw.wmnet
 ResolvedRequestJhancock.wmT383426 decommission db2135.codfw.wmnet
 ResolvedRequestJhancock.wmT383572 decommission db2128.codfw.wmnet
 ResolvedRequestJhancock.wmT383697 decommission db2132.codfw.wmnet
 ResolvedRequestJhancock.wmT383766 decommission db2130.codfw.wmnet
 ResolvedJhancock.wmT383971 decommission db2139
 ResolvedRequestJhancock.wmT384001 decommission db2131.codfw.wmnet
 ResolvedRequestJhancock.wmT384343 decommission db2133.codfw.wmnet
 ResolvedRequestJhancock.wmT384476 decommission db2134.codfw.wmnet
 ResolvedRequestJhancock.wmT384479 decommission db2136.codfw.wmnet
 ResolvedRequestJhancock.wmT384480 decommission db2140.codfw.wmnet

Event Timeline

Marostegui created this task.Jul 1 2024, 7:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2024, 7:10 AM
Marostegui moved this task from Triage to Ready on the DBA board.Jul 1 2024, 7:11 AM
ABran-WMF claimed this task.Jul 1 2024, 1:49 PM
ABran-WMF triaged this task as Medium priority.Jul 15 2024, 12:48 PM
gerritbot added a comment.Jul 19 2024, 12:28 PM

Change #1055428 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] dbproxy: mirrors hieradata

https://gerrit.wikimedia.org/r/1055428

gerritbot added a project: Patch-For-Review.Jul 19 2024, 12:28 PM
gerritbot added a comment.Aug 13 2024, 9:24 AM

Change #1055428 merged by Arnaudb:

[operations/puppet@production] dbproxy: mirrors hieradata

https://gerrit.wikimedia.org/r/1055428

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2024, 9:31 AM
ABran-WMF mentioned this in T374355: Productionize pc(1|2)017.Sep 9 2024, 12:45 PM
ABran-WMF added parent tasks: T374623: Decommission db21[21-40], T373579: Productionize db22[21-40].Oct 18 2024, 12:29 PM
gerritbot added a comment.Nov 5 2024, 8:46 AM

Change #1087369 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] dbproxy: update grants with ip and fqdn

https://gerrit.wikimedia.org/r/1087369

gerritbot added a project: Patch-For-Review.Nov 5 2024, 8:46 AM
gerritbot added a comment.Nov 5 2024, 8:50 AM

Change #1087374 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] dbproxy: switch CNAMEs

https://gerrit.wikimedia.org/r/1087374

Marostegui added a comment.Dec 23 2024, 7:01 AM

Seems that these hosts were "productionized" but silenced for 3 months as the grants were not added and were left in that state?. They just showed up in icinga with HAproxy being down for 70 days (I assume the downtime expired). In order not to play with grants in production one day before the holidays, I will silence them until we are back and fix them when we are back.

Marostegui moved this task from Ready to In progress on the DBA board.Jan 2 2025, 6:40 AM
gerritbot added a comment.Jan 2 2025, 6:42 AM

Change #1107883 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1028: Update hosts

https://gerrit.wikimedia.org/r/1107883

gerritbot added a comment.Jan 2 2025, 6:44 AM

Change #1107883 merged by Marostegui:

[operations/puppet@production] dbproxy1028: Update hosts

https://gerrit.wikimedia.org/r/1107883

Marostegui updated the task description. (Show Details)Jan 2 2025, 6:48 AM

dbproxy1028 is now up and running.

Marostegui added a comment.Jan 2 2025, 6:48 AM
In T368874#10426145, @Marostegui wrote:

dbproxy1028 is now up and running.

I will leave it a few days before doing the switchover.

gerritbot added a comment.Jan 2 2025, 6:50 AM

Change #1107884 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028

https://gerrit.wikimedia.org/r/1107884

gerritbot added a comment.Jan 2 2025, 6:53 AM

Change #1107884 merged by Marostegui:

[operations/puppet@production] production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028

https://gerrit.wikimedia.org/r/1107884

Marostegui added a comment.Jan 2 2025, 7:11 AM

dbproxy1029 is up and running - will leave it a few days before doing the switchover.

gerritbot added a comment.Jan 2 2025, 7:13 AM

Change #1107887 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-m5.sql.erb: Replace dbproxy1021 with dbproxy1029

https://gerrit.wikimedia.org/r/1107887

gerritbot added a comment.Jan 2 2025, 7:14 AM

Change #1107888 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] report_users.sh: Add dbproxy102[89]

https://gerrit.wikimedia.org/r/1107888

gerritbot added a comment.Jan 2 2025, 7:16 AM

Change #1107888 merged by jenkins-bot:

[operations/software@master] report_users.sh: Add dbproxy102[89]

https://gerrit.wikimedia.org/r/1107888

gerritbot added a comment.Jan 2 2025, 7:40 AM

Change #1107887 merged by Marostegui:

[operations/puppet@production] production-m5.sql.erb: Replace dbproxy1021 with dbproxy1029

https://gerrit.wikimedia.org/r/1107887

gerritbot added a comment.Jan 6 2025, 7:04 AM

Change #1108315 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1028: Enable notifications

https://gerrit.wikimedia.org/r/1108315

gerritbot added a comment.Jan 6 2025, 7:04 AM

Change #1108316 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Switchover m3-master

https://gerrit.wikimedia.org/r/1108316

gerritbot added a comment.Jan 6 2025, 7:09 AM

Change #1108315 merged by Marostegui:

[operations/puppet@production] dbproxy1028: Enable notifications

https://gerrit.wikimedia.org/r/1108315

Stashbot added a comment.Jan 6 2025, 7:13 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-06T07:13:22Z] <marostegui> dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874

gerritbot added a comment.Jan 6 2025, 7:13 AM

Change #1108316 merged by Marostegui:

[operations/dns@master] wmnet: Switchover m3-master

https://gerrit.wikimedia.org/r/1108316

Marostegui added a subtask: T383025: decommission dbproxy1020.eqiad.wmnet.Jan 6 2025, 7:55 AM
Marostegui added a subtask: T383033: decommission dbproxy1021.eqiad.wmnet.Jan 6 2025, 9:20 AM
Marostegui claimed this task.Jan 7 2025, 5:25 AM
Marostegui added a subscriber: ABran-WMF.
gerritbot added a comment.Jan 7 2025, 5:36 AM

Change #1108539 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Switch m5-master proxy

https://gerrit.wikimedia.org/r/1108539

Marostegui added a subtask: T383068: phabricator long running queries.Jan 7 2025, 5:37 AM
gerritbot added a comment.Jan 7 2025, 5:38 AM

Change #1108539 merged by Marostegui:

[operations/dns@master] wmnet: Switch m5-master proxy

https://gerrit.wikimedia.org/r/1108539

Stashbot added a comment.Jan 7 2025, 5:40 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-07T05:40:41Z] <marostegui> Switchover m5 eqiad proxy dbmaint eqiad T368874

Marostegui updated the task description. (Show Details)Jan 7 2025, 7:55 AM

No more connections from dbproxy1020, they are all routed via dbproxy1028

Marostegui closed this task as Resolved.Jan 7 2025, 12:07 PM

Drained the connections from dbproxy1021 and it is all now being sent through dbproxy1029

Marostegui updated the task description. (Show Details)Jan 7 2025, 12:07 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Jan 7 2025, 12:29 PM
Jclark-ctr closed subtask T383025: decommission dbproxy1020.eqiad.wmnet as Resolved.Jan 8 2025, 4:37 PM
Jclark-ctr closed subtask T383033: decommission dbproxy1021.eqiad.wmnet as Resolved.Jan 8 2025, 4:39 PM
gerritbot added a comment.Jan 29 2025, 9:45 AM

Change #1087374 abandoned by Marostegui:

[operations/dns@master] dbproxy: switch CNAMEs

Reason:

This is not needed anymore

https://gerrit.wikimedia.org/r/1087374

jcrespo removed a subtask: T383068: phabricator long running queries.Feb 4 2025, 1:16 PM
gerritbot added a comment.Feb 27 2025, 8:43 AM

Change #1087369 abandoned by Marostegui:

[operations/puppet@production] dbproxy: update grants with ip and fqdn

Reason:

This was done in a different patch

https://gerrit.wikimedia.org/r/1087369

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2025, 9:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits