Ah, right. Sorry, mixed up 2 different sets of hosts.

@Jhancock.wm cc: @RLazarus I depooled the server and set a downtime of 24 hours.

Lucas is right. I can confirm the test passes with any wikimedia.org subdomain as long as the path stays /wiki/Main_Page and starts failing as expected once that path changes.

scap runs httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug.discovery.wmnet --https_port=4444 --retry_on_timeout

Also see T364773

follow-ups are happening on the parent tasks

T364773

Thanks for confirming that @Tarrow Good to know we have a workaround.

@ecarg Your user is now in the deployment group on the deployment server. Give it about 30 minutes and you should have all the access needed for an actual deployment.

@JMeybohm I noticed I can't manually run puppet agent on this host. It says I don't have the sudo privileges for it. So I think puppet never ran to setup the initial users.

In T363415#9778509, @Dzahn wrote:

This still needs https://gerrit.wikimedia.org/r/c/operations/puppet/+/1026193 to be merged to be able to call it resolved.

T364656 will be about upgrading / replacing the production deployment servers.

@akosiaris I made T364656 and suggest seeing that either as a parent task or simply merging this in there.

Looks like the timeout is already fully puppetized and in Hiera:

Current unit file of docker.gs service:

Is there a setting that can be changed to allow more docker-gc.service failures within a particular window before alerting?

Thanks for manager approval. I will upload a patch and assigning to the group approver for consideration :)

Things I think we can delete, at a first glance:

22:54 < jinxer-wm> RESOLVED: SystemdUnitFailed: wmf_auto_restart_envoyproxy.service on contint1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state -

https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed

added envoy to contint1003 to fix T364510

Recently https://gerrit.wikimedia.org/r/c/operations/puppet/+/1028796 was merged which adds auto-restart service for envoyproxy.

This is the test server for releng from T358237

@Tarrow You could try if it works with ssh -o RequiredRSASize=1024 just to debug or until we can fix this.

quoting from a serverfault.com question: "If your getting the "Invalid key length" error, the problem isn't your Ciphers (that may be it's own problem, but if you're getting a key, SSH has agreed to a Cipher)"

Here is the gerrit -> github replication config, it's in hieradata/common/profile/gerrit.yaml in the puppet repo.

fwiw - for the person who will add the production puppet role to this later: This is only possible since just recently but should be mostly unblocked now: details in T363415 - needs one more patch though where your review would be great.

@Mcastro Please confirm if you approve

@thcipriani please consider for approval (https://wikimedia.namely.com/people/eaebb898-01ba-404e-8cf8-2ed33c4e0d04/show/personal/employee-information/)