Steps:
1. Upgrade images
- [x] inside production-images, switch the php83 image to bookworm: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1295044
- [x] rebuild shellbox with the new base image (by replaying a previous build, NOT by picking up the newest code, see T427820#11979823)
2. rollout shellbox: over the course of 1 day [whole day affair, start in the morning]
- [x] update shellbox-constraints and shellbox-syntaxhighlight flavour in the secondary DC; wait 2 hours: done at **11:14Z**
- [x] update these in eqiad; wait 2 hours: done at **11:14Z**
- [ ] update all the things; wait 2 hours
During/after the rollout, monitor the following:
- shellbox logs in [[ https://logstash.wikimedia.org/goto/aeaf15284beff7e377686d876858e3f7 | logstash ]]
- [[ https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(description:'MediaWiki%20all%20messages%20unfiltered.',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:type,negate:!f,params:(query:mediawiki),type:phrase),query:(match:(type:(query:mediawiki,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:channel.keyword,negate:!t,params:(query:csp-report-only),type:phrase),query:(match_phrase:(channel.keyword:csp-report-only))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:channel.keyword,negate:!t,params:(query:api-feature-usage),type:phrase),query:(match_phrase:(channel.keyword:api-feature-usage))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:channel.keyword,negate:!f,params:(query:exec),type:phrase),query:(match_phrase:(channel.keyword:exec))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:level,negate:!t,params:(query:INFO),type:phrase),query:(match_phrase:(level:INFO)))),fullScreenMode:!f,options:(darkTheme:!f),query:(language:kuery,query:''),timeRestore:!t,title:mediawiki,viewMode:view) | exec channel in MW logs ]] => check this before start so I can see the diff, because this is noisy
- see if 500s go up compared to baseline in [[ https://grafana.wikimedia.org/goto/afo19mqpfn8xsc?orgId=1 | grafana ]]
- [[ https://grafana.wikimedia.org/goto/efo19i00ffpxcc?orgId=1 | mercurius grafana dashboard ]]