Page MenuHomePhabricator
Create Task
Maniphest T331896

upgrade miscweb VMs to bullseye
Closed, ResolvedPublic
Actions

Description

replace miscweb1002 and miscweb2002 (buster) with new VMs on bullseye and switch microsites to them

part of SRE sprint week - sub group bullseye upgrades

this should also include full decom of old VMs and removal from repo of "webserver-misc-static"

update 20230404: all services have been switched to new set of VMs on bullseye with a single exception, https://iegreview.wikimedia.org

one of the 2 remaining buster machines, the inactive one in codfw, has already been decom'ed fully.

this is only open because miscweb1002 is left in eqiad and hosts iegreview and that is waiting for T332918

  • new VMs on bullseye in service
  • (almost) all services moved to new VMs
  • old VM in codfw decom'ed
  • old VM in eqiad decom'ed (not yet, still hosting iegreview)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site/miscweb: remove miscweb1002, switch rsync source to miscweb1003operations/puppetproduction+2 -3
remove miscweb1002->webserver-misc-appsoperations/dnsmaster+0 -1
miscweb/site: remove miscweb2002 from siteoperations/puppetproduction+1 -1
wdqs/wcqs: switch query.wikidata.org and wcqs to bullseye backendsoperations/puppetproduction+4 -4
trafficserver/wdqs: switch query-preview.wikidata.org to new backendoperations/puppetproduction+2 -2
miscweb: remove miscweb2002 from rsync dest hostsoperations/puppetproduction+0 -1
delete webserver-misc-static.discovery.wmnetoperations/dnsmaster+0 -2
bacula: Add miscweb2003 jobs to the list of monitoring-ignored jobsoperations/puppetproduction+2 -0
miscweb: switch static-codereview to bullseye backendoperations/puppetproduction+1 -1
miscweb/static-codereview/httpbb: also test if files were syncedoperations/puppetproduction+5 -0
miscweb: switch design.wikimedia.org to bullseye backendoperations/puppetproduction+1 -1
miscweb: switch wikiworkshop.org to bullseye backendoperations/puppetproduction+2 -2
miscweb: switch research.wikimedia.org to bullseye backendoperations/puppetproduction+1 -1
miscweb: add custom and error log for transparency and archivesoperations/puppetproduction+8 -0
miscweb: switch os-reports.wikimedia.org to bullseye backendoperations/puppetproduction+1 -1
miscweb: add custom and error log for os-reports.wikimedia.orgoperations/puppetproduction+5 -0
miscweb: move os_reports httpd template to profile/microsites/operations/puppetproduction+1 -1
miscweb: switch annual and bienvenida microsites to miscweb2003operations/puppetproduction+2 -2
miscweb: switch tendril and dbtree microsites to miscweb2003operations/puppetproduction+2 -2
miscweb: add miscweb1003/2003 to rsync_dst_hostsoperations/puppetproduction+2 -0
miscweb/iegreview: set custom log, don't log into "other_vhosts" fileoperations/puppetproduction+4 -0
miscweb: move transparency httpd site templates out of role/apacheoperations/puppetproduction+2 -2
miscweb: switch sitemaps, transparency and tr-archives to miscweb2003operations/puppetproduction+3 -3
miscweb/annualreport: use non-generic custom log file nameoperations/puppetproduction+2 -2
miscweb: switch security.wm.org microsite to miscweb2003operations/puppetproduction+1 -1
miscweb: switch 15.wikipedia.org to miscweb2003operations/puppetproduction+1 -1
add webserver-misc-sites and point it to miscweb1003/2003operations/dnsmaster+2 -0
site: add miscweb role to miscweb2003operations/puppetproduction+1 -5
miscweb: use http::mod_conf to add authnz_ldap moduleoperations/puppetproduction+4 -0
miscweb: use php7.4 if on bullseyeoperations/puppetproduction+7 -1
miscweb: remove custom source line from httpd::mpmoperations/puppetproduction+0 -1
miscweb: add httpd::mpm directory to httpd classoperations/puppetproduction+5 -0
site: add miscweb1003 to miscweb roleoperations/puppetproduction+2 -2
set target quarter for miscweb bullseye upgrade to 2023-1operations/puppetproduction+1 -1
site: add miscweb1003/miscweb2003 with role insetup::serviceops_collaboperations/puppetproduction+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 ResolvedDzahnT327068 Bullseye upgrade for remaining Collab hosts
 ResolvedArnoldokothT295416 upgrade/replace VRTS (formerly OTRS) buster to bullseye
 ResolvedArnoldokothT338418 VRTS eqiad replacement
 ResolvedArnoldokothT339253 Decom otrs1001
 Resolved• eoghanT319477 Migrate doc hosts to Bullseye
 Resolvedandrea.denisseT332812 Site: 1 VM request for doc1003
 Resolvedandrea.denisseT332819 Site: 1 VM request for doc2002
 ResolvedhasharT333294 MediaWiki periodic Doxygen Jenkins job fails to publish
 ResolvedDzahnT327405 Decommission Racktables
 ResolvedDzahnT332952 upgrade moscovium (RT) to bullseye
 Resolved• eoghanT334435 upgrade releases hosts to bullseye
 Resolved• eoghanT337349 eqiad+codfw: 1 VM each site request for releases.wikimedia.org
Resolved• eoghanT333452 Replace existing aphlict1001 with puppet-managed bullseye host
 ResolvedDzahnT334519 upgrade phab (phorge) hosts to bullseye
 ResolvedDzahnT354221 automate data syncing between phabricator servers
 ResolvedDzahnT334517 upgrade contint servers to bullseye
 ResolvedNoneT342346 Refresh integration/zuul/deploy to work on Debian Bullseye
 ResolvedhasharT259611 The python-build images regenerate wheels even when matching ones are already available
 ResolvedDzahnT358237 Ganeti VM for contint migration
 ResolvedhasharT361224 Test zuul under Bullseye with contint1003
 DeclinedNoneT157818 zuul-merger fails when repository names overlaps
 ResolvedhasharT158406 operations/software repo commits always getting -1 from Jenkins-bot
 ResolvedDzahnT334521 upgrade gerrit servers to bullseye
 ResolvedDzahnT326368 gerrit1003 service implementation task
 ResolvedDzahnT334524 add new and remove old Gerrit service IP from netbox and DNS
 ResolvedDzahnT336427 decom gerrit1001
 ResolvedJclark-ctrT340077 decommission gerrit1001.wikimedia.org (dcops, netbox)
 ResolvedBUG REPORThasharT336524 IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?)
 ResolvedJeltoT345577 Investigate the phabricator-prod-1001 alert
 ResolvedDzahnT348392 Migrate planet servers to bullseye or bookworm
 ResolvedLegoktmT281219 Find a replacement for RSS aggregator for planet.wikimedia.org
 ResolvedDzahnT351849 Site: 2 VMs %request for planet
 ResolvedMoritzMuehlenhoffT353379 spicerack.ganeti.GanetiError: Error while performing request to RAPI
 InvalidNoneT351333 build python-phabricator package for bullseye (and bookworm?)
 Resolved• eoghanT331706 Migrate Mailman/lists to Bullseye/Bookworm
 Resolved• eoghanT363572 Reimage physical lists hosts to have public IPs
 Resolved• eoghanT283615 Make mailman3 work in the standby host (lists2001.wikimedia.org)
 Resolved• eoghanT365714 PuppetFailure - lists2001
 Resolved• eoghanT365698 SystemdUnitFailed - lists2001
 Resolved• eoghanT365781 SystemdUnitFailed - gitlab1003, gitlab1004, lists2001
 Resolved• eoghanT365804 SystemdUnitFailed - lists2001
 Resolved• eoghanT365789 SystemdUnitFailed - gitlab1003, gitlab1004, lists2001
 Resolved• eoghanT365999 SystemdUnitFailed - lists1004
 Resolved• eoghanT367521 Mailman Downtime: Migrate mailman from lists1001 to lists1004
 ResolvedLadsgroupT367833 Update grants for mailman
 DuplicateNoneT332623 scap deploy fails for iegreview
 ResolvedDzahnT331896 upgrade miscweb VMs to bullseye
 ResolvedDzahnT332102 Site: eqiad/codfw 2 VM request for miscweb
 ResolvedRequestDzahnT334024 decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mar 20 2023, 8:25 PM

Change 901296 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] add webserver-misc-sites and point it to miscweb1003/2003

https://gerrit.wikimedia.org/r/901296

gerritbot added a comment.Mar 20 2023, 8:27 PM

Change 901296 merged by Dzahn:

[operations/dns@master] add webserver-misc-sites and point it to miscweb1003/2003

https://gerrit.wikimedia.org/r/901296

gerritbot added a comment.Mar 20 2023, 8:32 PM

Change 901300 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch 15.wikipedia.org to miscweb2003

https://gerrit.wikimedia.org/r/901300

gerritbot added a comment.Mar 20 2023, 10:10 PM

Change 901300 merged by Dzahn:

[operations/puppet@production] miscweb: switch 15.wikipedia.org to miscweb2003

https://gerrit.wikimedia.org/r/901300

gerritbot added a comment.Mar 20 2023, 11:08 PM

Change 901318 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch annual and bienvenida microsites to miscweb2003

https://gerrit.wikimedia.org/r/901318

gerritbot added a comment.Mar 20 2023, 11:09 PM

Change 901319 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch tendril and dbtree microsites to miscweb2003

https://gerrit.wikimedia.org/r/901319

gerritbot added a comment.Mar 20 2023, 11:10 PM

Change 901320 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch security.wm.org microsite to miscweb2003

https://gerrit.wikimedia.org/r/901320

gerritbot added a comment.Mar 20 2023, 11:12 PM

Change 901321 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch sitemaps, transparency and tr-archives to miscweb2003

https://gerrit.wikimedia.org/r/901321

andrea.denisse subscribed.Mar 21 2023, 3:15 AM
gerritbot added a comment.Mar 21 2023, 4:48 PM

Change 901318 merged by Dzahn:

[operations/puppet@production] miscweb: switch annual and bienvenida microsites to miscweb2003

https://gerrit.wikimedia.org/r/901318

gerritbot added a comment.Mar 21 2023, 4:50 PM

Change 901319 merged by Dzahn:

[operations/puppet@production] miscweb: switch tendril and dbtree microsites to miscweb2003

https://gerrit.wikimedia.org/r/901319

gerritbot added a comment.Mar 21 2023, 10:04 PM

Change 900465 merged by Dzahn:

[operations/puppet@production] miscweb: add miscweb1003/2003 to rsync_dst_hosts

https://gerrit.wikimedia.org/r/900465

gerritbot added a comment.Mar 21 2023, 10:11 PM

Change 901677 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb/iegreview: set custom log, don't log into "other_vhosts" file

https://gerrit.wikimedia.org/r/901677

gerritbot added a comment.Mar 21 2023, 10:11 PM

Change 901677 merged by Dzahn:

[operations/puppet@production] miscweb/iegreview: set custom log, don't log into "other_vhosts" file

https://gerrit.wikimedia.org/r/901677

gerritbot added a comment.Mar 21 2023, 10:13 PM

Change 901678 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb/annualreport: use non-generic custom log file name

https://gerrit.wikimedia.org/r/901678

gerritbot added a comment.Mar 21 2023, 10:17 PM

Change 901678 merged by Dzahn:

[operations/puppet@production] miscweb/annualreport: use non-generic custom log file name

https://gerrit.wikimedia.org/r/901678

gerritbot added a comment.Mar 22 2023, 5:43 PM

Change 901320 merged by Dzahn:

[operations/puppet@production] miscweb: switch security.wm.org microsite to miscweb2003

https://gerrit.wikimedia.org/r/901320

Dzahn changed the task status from Open to In Progress.Mar 22 2023, 6:11 PM
Dzahn triaged this task as High priority.
Dzahn updated the task description. (Show Details)
Stashbot mentioned this in T332101: determine whether https://sitemaps.wikimedia.org still serves a purpose.Mar 22 2023, 6:12 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-22T18:12:38Z] <mutante> rsyncing /srv/org/wikimedia/sitemaps files for https://sitemaps.wikimedia.org from old to new machines. most other things are auto-deployed by puppet or puppet running intial scap or automatic rsync.. this is not. rsync -av /srv/org/wikimedia/sitemaps/ rsync://miscweb2003.codfw.wmnet/miscapps-srv/org/wikimedia/sitemaps/ T331896 - but also see T332101

gerritbot added a comment.Mar 22 2023, 6:20 PM

Change 901321 merged by Dzahn:

[operations/puppet@production] miscweb: switch sitemaps, transparency and tr-archives to miscweb2003

https://gerrit.wikimedia.org/r/901321

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 6:30 PM
gerritbot added a comment.Mar 22 2023, 7:02 PM

Change 902167 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch research.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902167

gerritbot added a project: Patch-For-Review.Mar 22 2023, 7:02 PM

Change 902169 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch wikiworkshop.org to bullseye backend

https://gerrit.wikimedia.org/r/902169

gerritbot added a comment.Mar 22 2023, 7:04 PM

Change 902170 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch design.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902170

gerritbot added a comment.Mar 22 2023, 7:05 PM

Change 902172 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch os-reports.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902172

gerritbot added a comment.Mar 22 2023, 7:06 PM

Change 902174 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: switch static-codereview to bullseye backend

https://gerrit.wikimedia.org/r/902174

Dzahn updated the task description. (Show Details)Mar 22 2023, 7:09 PM
gerritbot added a comment.Mar 23 2023, 12:55 AM

Change 902140 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: move transparency httpd site templates out of role/apache

https://gerrit.wikimedia.org/r/902140

gerritbot added a comment.Mar 23 2023, 12:58 AM

Change 902140 merged by Dzahn:

[operations/puppet@production] miscweb: move transparency httpd site templates out of role/apache

https://gerrit.wikimedia.org/r/902140

gerritbot added a comment.Mar 23 2023, 1:04 AM

Change 902142 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: move os_reports httpd template to profile/microsites/

https://gerrit.wikimedia.org/r/902142

gerritbot added a comment.Mar 23 2023, 1:07 AM

Change 902142 merged by Dzahn:

[operations/puppet@production] miscweb: move os_reports httpd template to profile/microsites/

https://gerrit.wikimedia.org/r/902142

gerritbot added a comment.Mar 23 2023, 1:09 AM

Change 902144 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: add custom and error log for os-reports.wikimedia.org

https://gerrit.wikimedia.org/r/902144

gerritbot added a comment.Mar 23 2023, 1:12 AM

Change 902144 merged by Dzahn:

[operations/puppet@production] miscweb: add custom and error log for os-reports.wikimedia.org

https://gerrit.wikimedia.org/r/902144

gerritbot added a comment.Mar 23 2023, 1:20 AM

Change 902172 merged by Dzahn:

[operations/puppet@production] miscweb: switch os-reports.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902172

gerritbot added a comment.Mar 23 2023, 1:22 AM

Change 902166 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: add custom and error log for transparency and archives

https://gerrit.wikimedia.org/r/902166

gerritbot added a comment.Mar 23 2023, 1:26 AM

Change 902166 merged by Dzahn:

[operations/puppet@production] miscweb: add custom and error log for transparency and archives

https://gerrit.wikimedia.org/r/902166

gerritbot added a comment.Mar 23 2023, 1:35 AM

Change 902167 merged by Dzahn:

[operations/puppet@production] miscweb: switch research.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902167

gerritbot added a comment.Mar 23 2023, 1:42 AM

Change 902169 merged by Dzahn:

[operations/puppet@production] miscweb: switch wikiworkshop.org to bullseye backend

https://gerrit.wikimedia.org/r/902169

gerritbot added a comment.Mar 23 2023, 1:48 AM

Change 902170 merged by Dzahn:

[operations/puppet@production] miscweb: switch design.wikimedia.org to bullseye backend

https://gerrit.wikimedia.org/r/902170

Stashbot added a comment.Mar 23 2023, 2:00 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-23T02:00:00Z] <mutante> rsyncing ~4GB files for static-codereview.wikimedia.org from old to newer VMs for T331896 - no automatic sync / deploy for these

gerritbot added a comment.Mar 23 2023, 2:09 AM

Change 902228 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb/static-codereview/httpbb: also test if files were synced

https://gerrit.wikimedia.org/r/902228

gerritbot added a comment.Mar 23 2023, 2:16 AM

Change 902228 merged by Dzahn:

[operations/puppet@production] miscweb/static-codereview/httpbb: also test if files were synced

https://gerrit.wikimedia.org/r/902228

gerritbot added a comment.Mar 23 2023, 2:20 AM

Change 902174 merged by Dzahn:

[operations/puppet@production] miscweb: switch static-codereview to bullseye backend

https://gerrit.wikimedia.org/r/902174

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 2:30 AM
gerritbot added a comment.Mar 23 2023, 2:42 AM

Change 902229 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] decom miscweb2002

https://gerrit.wikimedia.org/r/902229

gerritbot added a project: Patch-For-Review.Mar 23 2023, 2:42 AM
Dzahn moved this task from Incoming to Work in Progress on the collaboration-services board.Mar 23 2023, 5:00 PM
jcrespo subscribed.Mar 27 2023, 7:24 AM

miscweb2003 bacula backups are failing. As I understand, those hosts are in setup, so this is expected (please warn if not!) so I will remove them from the check so they stop alerting- they should be added back to it before closing this task to make sure newer backups are checked.

gerritbot added a comment.Mar 27 2023, 7:44 AM

Change 903179 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] bacula: Add miscweb2003 jobs to the list of monitoring-ignored jobs

https://gerrit.wikimedia.org/r/903179

gerritbot added a comment.Mar 27 2023, 7:51 AM

Change 903179 merged by Jcrespo:

[operations/puppet@production] bacula: Add miscweb2003 jobs to the list of monitoring-ignored jobs

https://gerrit.wikimedia.org/r/903179

jcrespo added a comment.Mar 28 2023, 10:47 AM
✔️ root@backup1001:~$ check_bacula.py miscweb2003.codfw.wmnet-Monthly-1st-Wed-productionEqiad-rt-static
id: 502485, ts: None, type: I, status: C, bytes: 0
id: 501517, ts: 2023-03-21 05:53:55, type: F, status: f, bytes: 0
id: 501660, ts: 2023-03-22 05:25:05, type: F, status: f, bytes: 0
id: 501796, ts: 2023-03-23 05:11:14, type: F, status: f, bytes: 0
id: 501932, ts: 2023-03-24 05:28:19, type: F, status: f, bytes: 0
id: 502069, ts: 2023-03-25 05:23:13, type: F, status: f, bytes: 0
id: 502207, ts: 2023-03-26 04:56:19, type: F, status: f, bytes: 0
id: 502346, ts: 2023-03-27 04:57:39, type: F, status: f, bytes: 0
✔️ root@backup1001:~$ check_bacula.py miscweb2003.codfw.wmnet-Monthly-1st-Wed-productionEqiad-static-codereview
id: 502486, ts: None, type: I, status: C, bytes: 0
id: 501518, ts: 2023-03-21 05:56:56, type: F, status: f, bytes: 0
id: 501661, ts: 2023-03-22 05:28:06, type: F, status: f, bytes: 0
id: 501797, ts: 2023-03-23 05:14:14, type: F, status: f, bytes: 0
id: 501933, ts: 2023-03-24 05:31:29, type: F, status: f, bytes: 0
id: 502070, ts: 2023-03-25 05:26:23, type: F, status: f, bytes: 0
id: 502208, ts: 2023-03-26 04:59:19, type: F, status: f, bytes: 0
id: 502347, ts: 2023-03-27 05:00:39, type: F, status: f, bytes: 0

The backup of this host is returning f as backup status ("fatal error"). I ignored what was going on because I thought just paths were missing or other issue common of WIP host. But I now assume this is not expected, so digging deeper.

jcrespo added a comment.Mar 28 2023, 10:54 AM

Log says:

27-Mar 05:00 backup1001.eqiad.wmnet JobId 502347: Warning: bsockcore.c:201 Could not connect to Client: miscweb2003.codfw.wmne
t-fd on miscweb2003.codfw.wmnet:9102. ERR=Connection refused
Retrying ...
27-Mar 05:03 backup1001.eqiad.wmnet JobId 502347: Fatal error: bsockcore.c:208 Unable to connect to Client: miscweb2003.codfw.
wmnet-fd on miscweb2003.codfw.wmnet:9102. ERR=Connection refused
27-Mar 05:03 backup1001.eqiad.wmnet JobId 502347: Fatal error: No Job status returned from FD.

While, what looks like a network error can be several things, my guess is we are missing the profile setting up the client and opening the port of the new host- as it is only happening for this server.

@Dzahn I will be around when you start working so ping me so we can have a look together before we revert the above patch.

Dzahn added a comment.Mar 28 2023, 3:16 PM

@jcrespo The bacula-fd service was running and listening on port 9102 but still refusing connections. Restarting the service fixed it though and now a connection from backup1001 to port 9102 on miscweb2003 works.

before:

root@miscweb2003:/home/dzahn# netstat -tulpen | grep bacula
tcp        0      0 127.0.0.1:9102          0.0.0.0:*               LISTEN      0          2781812    559184/bacula-fd    

..
[backup1001:~] $ telnet miscweb2003.codfw.wmnet 9102
Trying 2620:0:860:104:10:192:48:65...
Trying 10.192.48.65...
telnet: Unable to connect to remote host: Connection refused

action taken:

systemctl restart bacula-fd

after:

root@miscweb2003:/home/dzahn# netstat -tulpen | grep bacula
tcp        0      0 0.0.0.0:9102            0.0.0.0:*               LISTEN      0          11164886   1857341/bacula-fd   

..

[backup1001:~] $ telnet miscweb2003.codfw.wmnet 9102
Trying 2620:0:860:104:10:192:48:65...
Trying 10.192.48.65...
Connected to miscweb2003.codfw.wmnet.
Escape character is '^]'.

So should be fine now.

jcrespo added a comment.Mar 28 2023, 3:19 PM

The bacula-fd service was running and listening on port 9102 but still refusing connections

Oh, weird.

I will try to schedule a rerun of the backups to confirm it is now fixed- and we can deploy the revert. Sorry this happened.

Dzahn added a comment.Mar 28 2023, 3:42 PM

@jcrespo Oh, I noticed now before it was listening only on 127.0.0.1 but after it is listening on 0.0.0.0. Must be a race condition. Seems rare though. Thanks for confirming :)

gerritbot added a comment.Mar 28 2023, 6:17 PM

Change 902175 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] delete webserver-misc-static.discovery.wmnet

https://gerrit.wikimedia.org/r/902175

gerritbot added a comment.Mar 28 2023, 6:22 PM

Change 902175 merged by Dzahn:

[operations/dns@master] delete webserver-misc-static.discovery.wmnet

https://gerrit.wikimedia.org/r/902175

jcrespo added a comment.Mar 29 2023, 8:26 AM
root@backup1001:~$ check_bacula.py miscweb2003.codfw.wmnet-Monthly-1st-Wed-productionEqiad-rt-static
id: 502633, ts: None, type: I, status: C, bytes: 0
id: 501517, ts: 2023-03-21 05:53:55, type: F, status: f, bytes: 0
id: 501660, ts: 2023-03-22 05:25:05, type: F, status: f, bytes: 0
id: 501796, ts: 2023-03-23 05:11:14, type: F, status: f, bytes: 0
id: 501932, ts: 2023-03-24 05:28:19, type: F, status: f, bytes: 0
id: 502069, ts: 2023-03-25 05:23:13, type: F, status: f, bytes: 0
id: 502207, ts: 2023-03-26 04:56:19, type: F, status: f, bytes: 0
id: 502346, ts: 2023-03-27 04:57:39, type: F, status: f, bytes: 0
id: 502485, ts: 2023-03-28 22:03:31, type: F, status: T, bytes: 2032
id: 502523, ts: 2023-03-28 22:39:38, type: F, status: T, bytes: 2032
✔️ root@backup1001:~$ check_bacula.py miscweb2003.codfw.wmnet-Monthly-1st-Wed-productionEqiad-static-codereview
id: 502634, ts: None, type: I, status: C, bytes: 0
id: 501518, ts: 2023-03-21 05:56:56, type: F, status: f, bytes: 0
id: 501661, ts: 2023-03-22 05:28:06, type: F, status: f, bytes: 0
id: 501797, ts: 2023-03-23 05:14:14, type: F, status: f, bytes: 0
id: 501933, ts: 2023-03-24 05:31:29, type: F, status: f, bytes: 0
id: 502070, ts: 2023-03-25 05:26:23, type: F, status: f, bytes: 0
id: 502208, ts: 2023-03-26 04:59:19, type: F, status: f, bytes: 0
id: 502347, ts: 2023-03-27 05:00:39, type: F, status: f, bytes: 0
id: 502486, ts: 2023-03-28 22:03:34, type: F, status: T, bytes: 4524720912
id: 502524, ts: 2023-03-28 22:39:41, type: F, status: T, bytes: 4524720912
✔️

Backups are now flowing ok, reverting ignore list.

Dzahn added a comment.Mar 29 2023, 4:13 PM
In T331896#8737477, @jcrespo wrote:

Backups are now flowing ok, reverting ignore list.

Great! Thanks for confirming that.

gerritbot added a comment.Mar 30 2023, 7:11 PM

Change 904619 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: remove miscweb2002 from rsync dest hosts

https://gerrit.wikimedia.org/r/904619

gerritbot added a comment.Mar 30 2023, 7:12 PM

Change 904619 merged by Dzahn:

[operations/puppet@production] miscweb: remove miscweb2002 from rsync dest hosts

https://gerrit.wikimedia.org/r/904619

gerritbot added a comment.Apr 3 2023, 7:06 PM

Change 905292 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] trafficserver/wdqs: switch query-preview.wikidata.org to new backend

https://gerrit.wikimedia.org/r/905292

bking subscribed.Apr 3 2023, 9:27 PM
gerritbot added a comment.Apr 3 2023, 9:31 PM

Change 905317 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] wdqs/wcqs: switch query.wikidata.org and wcqs to bullseye backends

https://gerrit.wikimedia.org/r/905317

gerritbot added a comment.Apr 3 2023, 9:46 PM

Change 905292 merged by Dzahn:

[operations/puppet@production] trafficserver/wdqs: switch query-preview.wikidata.org to new backend

https://gerrit.wikimedia.org/r/905292

Stashbot added a comment.Apr 3 2023, 9:53 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-03T21:52:59Z] <ryankemper> T331896 sudo -E cumin -b 4 'wdqs*' 'sudo run-puppet-agent'

gerritbot added a comment.Apr 4 2023, 7:52 PM

Change 905317 merged by Dzahn:

[operations/puppet@production] wdqs/wcqs: switch query.wikidata.org and wcqs to bullseye backends

https://gerrit.wikimedia.org/r/905317

Stashbot added a comment.Apr 4 2023, 7:55 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T19:55:15Z] <mutante> https://query.wikidata.org and WCQS GUIs are switching to new backend VMs on bullseye in codfw T330090 T331896

Stashbot mentioned this in T330090: Switchover static miscweb services to codfw.Apr 4 2023, 7:55 PM
Stashbot added a comment.Apr 4 2023, 8:00 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:00:15Z] <ryankemper> T331896 Running puppet on wdqs fleet to pickup new miscweb gui_url: ryankemper@cumin1001:~$ sudo -E cumin -b 6 'wdqs*' 'run-puppet-agent'

Stashbot added a comment.Apr 4 2023, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:06:49Z] <ryankemper> T331896 Running puppet on wcqs fleet to pickup new miscweb gui_url: ryankemper@cumin1001:~$ sudo -E cumin -b 2 'wcqs*' 'run-puppet-agent'

Dzahn added a comment.Apr 4 2023, 8:39 PM

The following services have switched to both codfw and bullseye today.

https://query-preview.wikidata.org
https://query.wikidata.org
https://commons-query.wikimedia.org

Dzahn added a subtask: T334024: decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet.Apr 4 2023, 8:47 PM
Dzahn added a subscriber: LSobanski.EditedApr 4 2023, 8:50 PM

@LSobanski Everything is migrated off of buster/eqiad miscweb - except iegreview. so now we can shutdown miscweb2002, inactive buster host (T334024), but to finish this and shutdown miscweb1002, active buster host with ONLY iegreview, we are still blocked on T332918 .

Dzahn closed subtask T334024: decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet as Resolved.Apr 4 2023, 10:07 PM
gerritbot added a comment.Apr 4 2023, 10:07 PM

Change 902229 merged by Dzahn:

[operations/puppet@production] miscweb/site: remove miscweb2002 from site

https://gerrit.wikimedia.org/r/902229

Dzahn changed the task status from In Progress to Stalled.Apr 4 2023, 10:09 PM

now stalled on T332918 (this ticket is High prio but the one it's stalled on is Low prio)

Dzahn updated the task description. (Show Details)Apr 4 2023, 10:16 PM
Dzahn updated the task description. (Show Details)Apr 6 2023, 7:51 PM
Dzahn mentioned this in T334415: decom iegreview.Apr 10 2023, 4:55 PM
gerritbot added a comment.Apr 10 2023, 10:56 PM

Change 907546 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] remove miscweb1002->webserver-misc-apps

https://gerrit.wikimedia.org/r/907546

gerritbot added a comment.Apr 10 2023, 10:59 PM

Change 907547 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/miscweb: remove miscweb1002, switch rsync source to miscweb1003

https://gerrit.wikimedia.org/r/907547

Dzahn reopened subtask T334024: decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet as In Progress.Apr 10 2023, 11:02 PM
gerritbot added a comment.Apr 10 2023, 11:04 PM

Change 907546 merged by Dzahn:

[operations/dns@master] remove miscweb1002->webserver-misc-apps

https://gerrit.wikimedia.org/r/907546

Dzahn changed the task status from Stalled to In Progress.Apr 10 2023, 11:07 PM
Dzahn closed this task as Resolved.
Dzahn closed subtask T334024: decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet as Resolved.
Dzahn updated the task description. (Show Details)

done! remaining miscweb1002 on buster has also been fully decom'ed. This closes the ticket.

gerritbot added a comment.Apr 10 2023, 11:29 PM

Change 907547 merged by Dzahn:

[operations/puppet@production] site/miscweb: remove miscweb1002, switch rsync source to miscweb1003

https://gerrit.wikimedia.org/r/907547

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits