Steps to replicate the issue (include links if applicable):
What happens?:
504
What should have happened instead?:
Tool should show
Software version (skip for WMF-hosted wikis like Wikipedia):
k8s php7.4 image
Other information (browser name/version, screenshots, etc.):
Stopped out of the blue, restarted as php7.3, then as php7.4 several times, no effect.
$ sudo become glamtools $ webservice status Your webservice of type php7.4 is running on backend kubernetes $ kubectl get pods NAME READY STATUS RESTARTS AGE glamtools-6dd95c9f99-k8cpf 1/1 Running 0 17d inventory-scan-28039870-blfvv 1/1 Running 0 2d19h
The 17 day uptime for the glamtools-6dd95c9f99-k8cpf indicates that whatever restart actions were tried somehow failed.
$ cat service.manifest # This file is used by Toolforge infrastructure. # Please do not edit manually at this time. # Thu Apr 27 21:34:36 2023 backend: kubernetes canonical: true distribution: debian version: 4 web: php7.4
The $HOME/service.manifest is getting touched, but:
$ kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE glamtools 1/1 1 1 2y309d
The deployment itself has not changed in nearly 3 years. I don't have a good story for what has gone so wrong here, but "restarted as php7.3, then as php7.4 several times, no effect." must mean attempted soft restarts (i.e. webservice restart or maybe even webservice php7.4 restart) and not hard restarts (i.e. webservice stop; webservice php7.4 start).
I do see some nano service.manifest invocations in $HOME/.bash_history. Editing that file will not change anything about the runtime, even if webservice restart is invoked after. No idea if that was part of the attempted fixes however or just a muscle memory replacement for less service.manifest or similar to read the file.
I will try to clean things up a bit by doing a hard stop/start cycle. I will also add a $HOME/service.template file first to take care of selecting the php7.4 runtime when no specific runtime is chosen with cli arguments.
Mentioned in SAL (#wikimedia-cloud) [2023-04-27T23:16:38Z] <wm-bot> <root> Hard stop && start cycle to reset Deployment and all dependent objects (T335520)
tools.glamtools@tools-sgebastion-11:~$ cp service.manifest service.template tools.glamtools@tools-sgebastion-11:~$ vim service.template tools.glamtools@tools-sgebastion-11:~$ cat service.template backend: kubernetes web: php7.4 tools.glamtools@tools-sgebastion-11:~$ webservice stop Stopping webservice tools.glamtools@tools-sgebastion-11:~$ webservice status Your webservice is not running tools.glamtools@tools-sgebastion-11:~$ webservice start Starting webservice............... tools.glamtools@tools-sgebastion-11:~$ dologmsg 'Hard stop && start cycle to reset Deployment and all dependent objects (T335520)' tools.glamtools@tools-sgebastion-11:~$ curl -I https://glamtools.toolforge.org/ HTTP/2 200 server: nginx/1.14.2 date: Thu, 27 Apr 2023 23:19:42 GMT content-type: text/html content-length: 2255 accept-ranges: bytes etag: "1918287804" last-modified: Mon, 13 Apr 2020 15:34:25 GMT strict-transport-security: max-age=31622400 x-clacks-overhead: GNU Terry Pratchett content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: *.toolforge.org wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmcloud.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://glamtools.toolforge.org; report-uri https://csp-report.toolforge.org/collect;