Note: this is a fallback task, written as more or less a verbatim copy of T264991 on grounds of doubts on the feasibility of the approach adopted in T263437 (see T329491#8895594).
Goal
In order to upgrade for all our MediaWiki clusters (including MW-on-K8s ) to Debian Bullseye, we need to upgrade the current Debian Buster installation to use ICU 67. The reason is that Debian ships Bullseye with version 67. This process requires all servers to be upgraded within a short period of time, and after that running the updateCollation.php script on all wikis with a non-standard collation.
Dependencies
Migration day TBD
The transition plan will roughly be as follows:
- Rebuild related packages for stretch by linking them to libicu67 and add them to the component/icu67 repo
- php7.4-bcmath
- php7.4-bz2
- php7.4-cli
- php7.4-common
- php7.4-curl
- php7.4-dba
- php7.4-fpm
- php7.4-gd
- php7.4-gmp
- php7.4-intl
- php7.4-json
- php7.4-mbstring
- php7.4-mysql
- php7.4-opcache
- php7.4-phpdbg
- php7.4-readline
- php7.4-xml
- dh-php
- libxml2
- php-acpu
- php-excimer
- php-geoip
- php-igbinary
- php-imagick
- php-luasandbox
- php-memcached
- php-msgpack
- php-redis
- php-wmerrors
- tideways
- wikidiff2
Process
- Use a feature flag to enable component/icu67 in puppet
- Build container base images containing the new packages (https://gerrit.wikimedia.org/r/954700)
- Define a migration date - 2023-11-02
- Notify the communities, with commrel's help
- Test packages on mwdebug servers
- Update packages on deployment-prep
- On Migration day:
- Upgrade to libicu67 everywhere
- app, api, parsoid, jobrunners
- snapshot hosts
- kubernetes deploymens (scap sync-world --k8s-only -D full_image_build:true)
- deploy hosts (can be done later)
- Run foreachwikiindblist XX.dblist updateCollation.php --force on all wikis where it's needed (~10 days, this is the point of no return, **use screen/tmux)
- foreachwikiindblist s1.dblist updateCollation.php --force
- foreachwikiindblist s2.dblist updateCollation.php --force
- foreachwikiindblist s3.dblist updateCollation.php --force
- foreachwikiindblist s5.dblist updateCollation.php --force
- foreachwikiindblist s6.dblist updateCollation.php --force
- foreachwikiindblist s7.dblist updateCollation.php --force
- Upgrade to libicu67 everywhere
- Include the icu67 upgrade into the general php7.4 images / retag the php7.4-icu67 ones
Notes:
updateCollation.php needs to run for: s1, s2, s3, s5, s6, s7 (according to find_collations.py)
s1: enwiki s2: cswiki fiwiki itwiki nlwiki nowiki plwiki ptwiki svwiki thwiki trwiki s3: mediawikiwiki abwiki azwiki bawiki bawikibooks be_x_oldwiki bewiki bewikisource bnwiki bnwikisource bswiki bswiktionary ckbwiki cswiktionary cywiki cywikibooks cywikiquote cywikisource cywiktionary dewikisource eswikiversity etwiki etwikibooks etwikimedia etwikiquote etwikisource etwiktionary fawikisource fawiktionary fawikibooks fawikinews fawikiquote fiwikibooks fiwikimedia fiwikinews fiwikiquote fiwikisource fiwikiversity fiwikivoyage frwikibooks frwikinews frwikiversity gawiki gdwiki gewikimedia glwiki grwikimedia hewikisource hrwiki hsbwiki idwikimedia id_internalwikimedia ilowiki iswiki iswiktionary ltwiki lvwiki mkwiki napwikisource nnwiki nowikimedia olowiki plwikisource plwikivoyage plwiktionary ptwikibooks romdwikimedia rowikibooks rowikinews rowikiquote rowikisource rowikivoyage rowiktionary rswikimedia ruwikibooks ruwikinews ruwikiquote ruwikisource ruwikiversity ruwikivoyage ruwiktionary sewiki simplewiki slwiki slwikibooks slwikiquote slwikisource slwikiversity slwiktionary sqwiki sqwikibooks sqwikinews sqwikiquote srwikibooks srwikinews srwikiquote srwikisource srwiktionary skwiki svwikisource tawiki tawikibooks tawikinews tawikiquote tawikisource tawiktionary testwiki thwikibooks thwikinews thwikiquote thwikisource thwiktionary trwikiquote trwiktionary uawikimedia ukwikibooks ukwikinews ukwikiquote ukwikisource ukwikivoyage ukwiktionary viwikibooks viwikiquote viwikisource viwikivoyage viwiktionary wbwikimedia s5: azwikimedia shwiki srwiki trwikivoyage s6: frwiki ruwiki s7: eswiki fawiki frwiktionary hewiki huwiki rowiki ukwiki viwiki
For updating app, api, parsoid, jobrunners, we 'll use this command. For updating snapshot* and deploy* we 'll make sure to remove packages that are not installed there.
cumin mwdebug1001* 'depool; sleep 1; run-puppet-agent -e "upgrading ICU67" ; apt-get -q update; export DEBIAN_FRONTEND=noninteractive; apt-get install $PACKAGES -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"; systemctl restart php7.4-fpm ; pool'
I've checked all binary packages in the icu67 component with sudo -i reprepro -C component/icu67 -A amd64 list buster-wikimedia | sed 's/.*amd64: //;s/ .*//' | grep -v '\-dbg' and checked if they are installed with cat icu67_packages.txt | while read pkg; do dpkg-query -W --showformat='${db:Status-Status}' "$pkg" 2>&1| grep ^installed -q && echo -n "$pkg "; done; echo
This looks like:
cloudweb1003.wikimedia.org: libicu67 libxml2 php-common php-ldap php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-ldap php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml deploy1002.eqiad.wmnet: libicu67 libxml2 php-common php-readline php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-xml php7.4-yaml dumpsdata1001.eqiad.wmnet: libicu67 libxml2 htmldumper1001.eqiad.wmnet: libicu67 libxml2 mw1414.eqiad.wmnet (appserver): libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml mw1437.eqiad.wmnet (jobrunner) libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml mwdebug1001.eqiad.wmnet: libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-phpdbg php7.4-readline php7.4-redis php7.4-tideways php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml mwmaint1002.eqiad.wmnet: libicu67 libapache2-mod-php libapache2-mod-php7.4 libxml2 php-common php-readline php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml snapshot1008.eqiad.wmnet: libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-xml php7.4-yaml