$ ssh deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
$ sudo -i run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(0c9d579a79) gitpuppet - [WIP] Allow provisioning MediaWiki with PHP 8.1'
Notice: Applied catalog in 9.10 seconds
$ php --version
Warning: PHP Startup: Unable to load dynamic library 'json.so' (tried: /usr/lib/php/20210902/json.so (/usr/lib/php/20210902/json.so: cannot open shared object file: No such file or directory), /usr/lib/php/20210902/json.so.so (/usr/lib/php/20210902/json.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 8.1.30 (cli) (built: Sep 30 2024 15:09:07) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.30, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.30, Copyright (c), by Zend Technologies

$ cd /etc/php/8.1
$ find . -name '*json*'
./mods-available/json.ini
./fpm/conf.d/20-json.ini
./cli/conf.d/20-json.ini
$ sudo find . -name '*json*' -delete
$ sudo -i run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(e220ee7a8a) gitpuppet - [WIP] Allow provisioning MediaWiki with PHP 8.1'
Notice: Applied catalog in 11.38 seconds
$ php --version
PHP 8.1.30 (cli) (built: Sep 30 2024 15:09:07) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.30, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.30, Copyright (c), by Zend Technologies

$ hostname -f
deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
$ curl --include --head --connect-to ::$HOSTNAME -H 'X-Forwarded-Proto: https' 'http://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page'
HTTP/1.1 503 Service Unavailable
Date: Tue, 05 Nov 2024 16:41:27 GMT
Server: deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
Last-Modified: Thu, 01 Aug 2024 20:41:36 GMT
ETag: "4c9-61ea53cb0c441"
Accept-Ranges: bytes
Content-Length: 1225
Backend-Timing: D=848 t=1730824887173811
Connection: close
Content-Type: text/html

profile::mediawiki::jobrunner::default_php_version: '8.1'
profile::mediawiki::webserver::default_php_version: '8.1'

$ curl --head --include --connect-to ::$HOSTNAME -H 'X-Forwarded-Proto: https' 'http://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page'
HTTP/1.1 200 OK
Date: Tue, 05 Nov 2024 23:54:17 GMT
Server: deployment-mediawiki81.deployment-prep.eqiad1.wikimedia.cloud
X-Powered-By: PHP/8.1.30
X-Content-Type-Options: nosniff
Content-language: en
X-Analytics: ns=0;page_id=6708;rev_id=50307
Content-Security-Policy: script-src 'unsafe-eval' blob: 'self' *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org cxserver-beta.wmcloud.org recommend.wmflabs.org *.wikipedia.beta.wmcloud.org 'unsafe-inline' login.wikimedia.beta.wmflabs.org; default-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org cxserver-beta.wmcloud.org recommend.wmflabs.org *.wikipedia.beta.wmcloud.org wikimedia.org en.wikipedia.beta.wmflabs.org en.wikibooks.beta.wmflabs.org en.wikinews.beta.wmflabs.org en.wikiquote.beta.wmflabs.org en.wikisource.beta.wmflabs.org en.wikiversity.beta.wmflabs.org en.wikivoyage.beta.wmflabs.org en.wiktionary.beta.wmflabs.org api.wikimedia.beta.wmflabs.org commons.wikimedia.beta.wmflabs.org test2.wikipedia.beta.wmcloud.org pl.wikivoyage.beta.wmcloud.org login.wikimedia.beta.wmflabs.org; style-src 'self' data: blob: https://upload.wikimedia.beta.wmflabs.org upload.wikimedia.beta.wmflabs.org https://commons.wikimedia.beta.wmflabs.org https://upload.wikimedia.org https://commons.wikimedia.org *.wikimedia.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wiktionary.beta.wmflabs.org *.wikibooks.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiquote.beta.wmflabs.org wikidata.beta.wmflabs.org m.wikidata.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.mediawiki.beta.wmflabs.org wikifunctions.beta.wmflabs.org m.wikifunctions.beta.wmflabs.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org ws://visualeditor-realtime.wmflabs.org cxserver-beta.wmcloud.org recommend.wmflabs.org *.wikipedia.beta.wmcloud.org wikimedia.org 'unsafe-inline'; object-src 'none'; report-uri /w/api.php?action=cspreport&format=json
Vary: Accept-Encoding,Cookie,Authorization
Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
X-Request-Id: ZyqwKaaASCKb9y20xPmu-wAAAIM
Last-Modified: Tue, 22 Oct 2024 23:54:17 GMT
Backend-Timing: D=282764 t=1730850857294078
Content-Type: text/html; charset=UTF-8

[00:02]  <    bd808> Southparkfan: Do you remember from your work on deployment-prep bullseye stuff what changes are needed to pool a new MediaWiki server? Context is T378752 and my next step desire to route traffic to that PHP 8.1 node.
[00:02]  < stashbot> T378752: Figure out how to install PHP 8.1 on bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752
[00:05]  <    bd808> ah ha, I bet T361387 will tell me a lot
[00:05]  < stashbot> T361387: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387
[00:06]  <Southparkfan> bd808: I'm about to log off to get some sleep, but what I can tell for sure is that I had to provision the new appservers with puppet, then add them to their appropriate dsh groups so that they get the scap syncs
[00:07]  <Southparkfan> However, I don't think all appservers are pooled in ATS for cache_text traffic, iirc only one of the two appservers is pooled right now. Technically, the other one can also serve traffic just fine, though
[00:08]  <    bd808> Southparkfan: good to know. I found some of what you did. I'll ask you more if I end up really stuck. :)
[00:09]  <Southparkfan> I also recall having issues installing new VMs with certain puppet roles right away, at some point I just decided to bootstrap them with the 'base puppet role' (i.e. the default role for Cloud VPS instances, nothing app-specific applied), then changing the puppet role to the appropriate role
[00:10]  <    bd808> I had "fun" with puppetmaster certs on the new instance I built. It felt like there has been some regression in switching to a self-hosted puppetmaster
[00:10]  <Southparkfan> (otherwise the host could get stuck in some unknown state, where it boot# fine, but wouldn't allow me to log in)
[00:11]  <    bd808> I luckily have all the s3cr3t cloud root juice if that happens
[00:12]  <Southparkfan> Oh heh, I didn't have issues with the puppetmaster certificates. I did have issues with incorrect permissions on the git clone of the puppet repo on the puppetmaster in deployment-prep, but I think it has been fixed since
[00:12]  <Southparkfan> Ahaha :D magic sauce for those who shall not depend on working LDAP clients
[00:13]  <    bd808> yeah, I can even enter through the vm's root console if things are very messed up.
[00:14]  <Southparkfan> Feel free to brain dump the installation process somewhere, I can surely take a look later today (or tomorrow I think, with regards to your TZ - it's already past midnight here)
[00:15]  <    bd808> Southparkfan: thanks for the offer. Get some good sleep. :)

- params:
  - '@plugin=/usr/lib/trafficserver/modules/tslua.so'
  - '@pparam=/etc/trafficserver/lua/rb-mw-mangling.lua'
  replacement: http://deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud/w/api.php
  target: http://(.*)/w/api.php
  type: regex_map
- params:
  - '@plugin=/usr/lib/trafficserver/modules/tslua.so'
  - '@pparam=/etc/trafficserver/lua/normalize-path.lua'
  - '@pparam="3A 2F 40 21 24 28 29 2A 2C 3B"'
  - '@pparam="5B 5D 26 27 2B 3D"'
  - '@plugin=/usr/lib/trafficserver/modules/tslua.so'
  - '@pparam=/etc/trafficserver/lua/rb-mw-mangling.lua'
  replacement: http://deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud
  target: /
  type: map

--- Php::Extension[yaml].orig
+++ Php::Extension[yaml]

@@
-    package_overrides => {'7.4': 'php7.4-yaml'}
+    package_overrides => {'7.4': 'php7.4-yaml', '8.1': 'php8.1-yaml'}

$ sudo -i run-puppet-agent
  # no changes
$ php --version
PHP 7.4.33 (cli) (built: Apr 18 2024 14:41:42) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies
$ sudo -i run-puppet-agent
  # saw some warnings about missing packages.
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php8.1-cli' returned 100: Reading package lists...
Building dependency tree...                                                     
Reading state information...
E: Unable to locate package php8.1-cli                                          
E: Couldn't find any package by glob 'php8.1-cli'
E: Couldn't find any package by regex 'php8.1-cli'
...
Notice: /Stage[main]/Php::Fpm/File[/etc/php/8.1/fpm/php-fpm.conf]: Dependency Package[php8.1-fpm] has failures: true
Warning: /Stage[main]/Php::Fpm/File[/etc/php/8.1/fpm/php-fpm.conf]: Skipping because of failed dependencies
  # seems to be a dependency order problem because eventually we see the component added as would have been expected before the first apt run
...
Notice: /Stage[main]/Profile::Mediawiki::Php/Apt::Package_from_component[wikimedia-php81]/Apt::Repository[repository_wikimedia-php81]/File[/etc/apt/sources.list.d/repository_wikimedia-php81.list]/ensure: defined content as '{sha256}f6a9310fa9ca1920c0e7c0630149a6c709c29638d31898f2f860568b3a2f1fb4'
...
Notice: Applied catalog in 55.95 seconds
$ sudo -i run-puppet-agent
  # second run made things look much better
...
Notice: Applied catalog in 77.25 seconds
$ php --version
Warning: Module "tideways_xhprof" is already loaded in Unknown on line 0
PHP 8.1.31 (cli) (built: Nov 21 2024 21:07:42) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.31, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.31, Copyright (c), by Zend Technologies
$ sudo -i run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-snapshot05.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(51a18fb309) gitpuppet - MediaWiki: Only proxy existing .php files, otherwise return nice 404'
Notice: /Stage[main]/Php/File[/etc/php/8.1/cli/conf.d/20-tideways.ini]/ensure: removed
Notice: /Stage[main]/Php/File[/etc/php/8.1/cli/conf.d/20-wmerrors.ini]/ensure: removed
Notice: /Stage[main]/Php/File[/etc/php/8.1/fpm/conf.d/20-tideways.ini]/ensure: removed
Notice: Applied catalog in 9.64 seconds
$ sudo -i run-puppet-agent
  # no changes
Notice: Applied catalog in 10.02 seconds
$ php --version
PHP 8.1.31 (cli) (built: Nov 21 2024 21:07:42) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.31, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.31, Copyright (c), by Zend Technologies

Notice: /Stage[main]/Php/File[/etc/php/8.1/cli/conf.d/20-tideways.ini]/ensure: removed
Notice: /Stage[main]/Php/File[/etc/php/8.1/cli/conf.d/20-wmerrors.ini]/ensure: removed
Notice: /Stage[main]/Php/File[/etc/php/8.1/fpm/conf.d/20-tideways.ini]/ensure: removed