Cloud VPS "redirects" project Buster deprecation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	StrikerBot
	Jun 14 2024, 2:46 PM

Description

Upstream LTS support for Debian Buster ends on June 30, 2024, and soon after that we'll need to start deleting and removing those VMs.

Please either remove Buster VMs from your project or respond here with a proposed plan and timeline.

Buster VMs remaining after July 15th with no activity on their associated phab task will be shut down and/or deleted at the convenience of WMCS administrators.

Remaining Debian Buster instances (live report):

redirects-nginx02.redirects.eqiad1.wikimedia.cloud

Listed administrators are:

wikitech:User:BryanDavis (@bd808)
wikitech:User:Andrew Bogott (Developer account not linked to Phabricator)
wikitech:User:Dzahn (@Dzahn)

Details

Due Date: Jul 1 2024, 11:59 PM
Other Assignee: Dzahn

Event Timeline

StrikerBot created this task.Jun 14 2024, 2:46 PM

StrikerBot triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2024, 2:46 PM

Andrew moved this task from Unsorted to VPS projects on the Cloud-VPS (Debian Buster Deprecation) board.Jun 14 2024, 2:52 PM

I probably shouldn't be admin of this project. I hardly remember why I am in it. Years ago helped fixing something I guess.

In T367550#9893503, @Dzahn wrote:

I probably shouldn't be admin of this project. I hardly remember why I am in it. Years ago helped fixing something I guess.

I probably tricked you into joining the project to set up a redirect for something. No worries if you want to jump out.

• bd808 claimed this task.Jun 14 2024, 4:03 PM

Restricted Application added a project: User-bd808. · View Herald TranscriptJun 14 2024, 4:03 PM

Hmm.. my bash history shows I ran puppet agent twice and that was it.

I see these:

server {
  listen 80;
  server_name phab.wmcloud.org;
  return 301 https://phabricator.wmcloud.org$request_uri;
}

server {
  listen 80;
  server_name phorge.wmcloud.org;
  return 301 https://phabricator.wmcloud.org$request_uri;
}

Yea,I think it was about adding these redirects related to phab/phorge and the phab test instance.

Hmm, just leave me in for now.

Should we create a bookworm instance and slap role::labs::redirector on it and see what breaks?

Mentioned in SAL (#wikimedia-cloud) [2024-06-14T17:00:00Z] <mutante> creating instance redirects-nginx03 (g4.cores1.ram2.disk20) with bookworm to replace buster (T367550)

ah, we had another issue on the existing machine as well.. the puppet 7 migration:

Notice: puppet7 is not available on buster.  forcing this is likely going to cause issue.

@bd808 woah, the puppet role just works on bookworm.. not even a warning! 🤯

dzahn@redirects-nginx03:~$ for host in $(grep server_name /etc/nginx/sites-enabled/redirect | cut -d " " -f4 | cut -d\; -f1); do echo "curl -s -I -H 'Host: ${host}' localhost | grep Location"; done
curl -s -I -H 'Host: _' localhost | grep Location
curl -s -I -H 'Host: codesearch.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: dash.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: discourse-mediawiki.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: graphite.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: hatjitsu.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: horizon.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: living-style-guide.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: livingstyleguide.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: logstash-beta.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: phab.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: phorge.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: php-security-checker.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: reportcard.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: space.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: wcqs-beta.wmcloud.org' localhost | grep Location
curl -s -I -H 'Host: wcqs-beta.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: wdq.wmflabs.org' localhost | grep Location
curl -s -I -H 'Host: wikistream.wmflabs.org' localhost | grep Location

Location: https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/Host_not_configured
Location: https://codesearch.wmcloud.org/
Location: https://outreachdashboard.wmflabs.org/
Location: https://wmds-archive.toolforge.org/
Location: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes/
Location: https://hatjitsu.toolforge.org/
Location: https://horizon.wikimedia.org/
Location: https://design.wikimedia.org/style-guide/
Location: https://design.wikimedia.org/style-guide/
Location: https://beta-logs.wmcloud.org/
Location: https://phabricator.wmcloud.org/
Location: https://phabricator.wmcloud.org/
Location: https://php-security-checker.toolforge.org/
Location: https://analytics.wikimedia.org/dashboards/reportcard/
Location: https://diff.wikimedia.org/
Location: https://commons-query.wikimedia.org/
Location: https://commons-query.wikimedia.org/
Location: https://query.wikidata.org/
Location: https://wikistream.toolforge.org/

16:54 <@bd808> mutante: yeah, that should pretty much be all it takes. Setup a instance with the role, do some localhost curl tests to see that it is emitting the expected redirects, and then switch the proxy config in Horizon to point at the new instance

instance setup done
curl tests done
switching proxy config for phab.wmcloud.org and phorge.wmcloud.org

Everything works from localhost and also from old host to IP of new host.

But in my browser phab.wmcloud.org and phorge.wmcloud.org timeout.

So looks like there is some networking / firewall thing. But I see nothing that wouldn't match in the security group ("web") that exists in this project.

The other proxies are unchanged for now.

Dzahn changed the task status from Open to In Progress.Jun 14 2024, 9:11 PM

old IP: 172.16.0.48

new IP: 172.16.7.2

port 80

• bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Jun 26 2024, 12:34 AM

Mentioned in SAL (#wikimedia-cloud) [2024-06-26T23:08:52Z] <bd808> Added redirects-nginx03 to "web" security group (T367550)

In T367550#9894652, @Dzahn wrote:

So looks like there is some networking / firewall thing. But I see nothing that wouldn't match in the security group ("web") that exists in this project.

I think the Horizon UI tricked you into thinking that the "web" security group had been applied to the new instance when it in fact had not. Now that it is applied the proxies that have been updated to point to the new instance seem to be working as expected. I will switch them all over.

Mentioned in SAL (#wikimedia-cloud) [2024-06-26T23:17:47Z] <bd808> Switched all proxies to point at redirects-nginx03 (T367550)

In T367550#9928937, @bd808 wrote:

I think the Horizon UI tricked you into thinking that the "web" security group had been applied to the new instance when it in fact had not. Now that it is applied the proxies that have been updated to point to the new instance seem to be working as expected. I will switch them all over.

OMG, thanks for that!

$ ssh redirects-nginx03.redirects.eqiad1.wikimedia.cloud
$ for h in $(awk '/server_name/ && $2 !~ /_/ {gsub(/;/,"",$2); print $2}' /etc/nginx/sites-enabled/redirect); do echo == $h ==; curl -sI https://${h}; done
== codesearch.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://codesearch.wmcloud.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== dash.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://outreachdashboard.wmflabs.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== discourse-mediawiki.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://wmds-archive.toolforge.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== graphite.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== hatjitsu.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://hatjitsu.toolforge.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== horizon.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://horizon.wikimedia.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== living-style-guide.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://design.wikimedia.org/style-guide/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== livingstyleguide.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:17 GMT
content-type: text/html
content-length: 169
location: https://design.wikimedia.org/style-guide/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== logstash-beta.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://beta-logs.wmcloud.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== phab.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://phabricator.wmcloud.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== phorge.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://phabricator.wmcloud.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== php-security-checker.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://php-security-checker.toolforge.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== reportcard.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://analytics.wikimedia.org/dashboards/reportcard/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== space.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://diff.wikimedia.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== wcqs-beta.wmcloud.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://commons-query.wikimedia.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== wcqs-beta.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://commons-query.wikimedia.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== wdq.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://query.wikidata.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

== wikistream.wmflabs.org ==
HTTP/2 301
server: nginx/1.18.0
date: Wed, 26 Jun 2024 23:42:18 GMT
content-type: text/html
content-length: 169
location: https://wikistream.toolforge.org/
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

Mentioned in SAL (#wikimedia-cloud) [2024-06-26T23:43:52Z] <bd808> Shutdown redirects-nginx02 (T367550)

Mentioned in SAL (#wikimedia-cloud) [2024-06-26T23:48:32Z] <bd808> Deleted instance redirects-nginx02 (T367550)

• bd808 closed this task as Resolved.Jun 26 2024, 11:49 PM

• bd808 updated Other Assignee, added: Dzahn.

• bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.

Cloud VPS "redirects" project Buster deprecationClosed, ResolvedPublicActions

Description

Details

Event Timeline

Cloud VPS "redirects" project Buster deprecation
Closed, ResolvedPublic
Actions