For the ICU transition it's crucially important that no machine with write access to the databases gets updated before the date of the migration. So please do not test this on the eqiad mwdebugs for the time being.

Before the 16th we need also to prepare scripts to run as many updateCollation.php instances as we can, meaning 1 per database section.

The easiest way to do this is I think to extract all the wikis that have non-standard collation from mediawiki-config, and divide them by datase section, and run them with foreachwikiindblist, in parallel. We do have scripts I've used previously linked in tasks about the previous migrations - we can start from those.

Change 639751 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add a Hiera option to enable ICU63 component

https://gerrit.wikimedia.org/r/639751

Change 639780 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable ICU 63 in two hosts

https://gerrit.wikimedia.org/r/639780

In T264991#6603285, @jijiki wrote:

@thcipriani we will be upgrading to ICU 63 on the 16th Nov 2020. Since we will be restarting php-fpm across the cluster that day, can we put a note about this on the deployment calendar?

Do you want no deploys on the whole of the 16th? Or just a note?

For the time being I have a no deploy window for the Monday: https://wikitech.wikimedia.org/wiki/Deployments#Monday,_November_16 If you all just want backport/config/mediawiki deploys cancelled for Monday (and not services) I can do that instead.

In T264991#6608410, @Joe wrote:

For the ICU transition it's crucially important that no machine with write access to the databases gets updated before the date of the migration. So please do not test this on the eqiad mwdebugs for the time being.

Before the 16th we need also to prepare scripts to run as many updateCollation.php instances as we can, meaning 1 per database section.

The easiest way to do this is I think to extract all the wikis that have non-standard collation from mediawiki-config, and divide them by datase section, and run them with foreachwikiindblist, in parallel. We do have scripts I've used previously linked in tasks about the previous migrations - we can start from those.

Cool, we will ask for help on the day, thank you!

In T264991#6609215, @thcipriani wrote:

Do you want no deploys on the whole of the 16th? Or just a note?

For the time being I have a no deploy window for the Monday: https://wikitech.wikimedia.org/wiki/Deployments#Monday,_November_16 If you all just want backport/config/mediawiki deploys cancelled for Monday (and not services) I can do that instead.

I had a look on SAL from the last time, there were some config deployments, but that was all. If it is not a problem, it would be lovely to stick to no backport/config/mediawiki deploys on Monday, and make a note on the calendar as to why. Thank you!

On a pointer from @Joe (thanks!) I used the same basic approach as T86096#2325895 to identify wikis we'll need to touch with updateCollation.php.

The results:

s1:
  enwiki
s2:
  cswiki fiwiki itwiki nlwiki nowiki plwiki ptwiki svwiki thwiki trwiki
s3:
  mediawikiwiki abwiki azwiki bawiki bawikibooks be_x_oldwiki bewiki bewikisource bnwiki bnwikisource bswiki bswiktionary ckbwiki cswiktionary cywiki cywikibooks cywikiquote cywikisource cywiktionary dewikisource eswikiversity etwiki etwikibooks etwikimedia etwikiquote etwikisource etwiktionary fawikisource fawiktionary fawikibooks fawikinews fawikiquote fiwikibooks fiwikimedia fiwikinews fiwikiquote fiwikisource fiwikiversity fiwikivoyage frwikibooks frwikinews frwikiversity gdwiki gewikimedia glwiki grwikimedia hewikisource hrwiki hsbwiki idwikimedia id_internalwikimedia ilowiki iswiki iswiktionary ltwiki lvwiki mkwiki napwikisource nnwiki nowikimedia olowiki plwikisource plwikivoyage plwiktionary ptwikibooks romdwikimedia rowikibooks rowikinews rowikiquote rowikisource rowikivoyage rowiktionary rswikimedia ruwikibooks ruwikinews ruwikiquote ruwikisource ruwikiversity ruwikivoyage ruwiktionary sewiki slwiki slwikibooks slwikiquote slwikisource slwikiversity slwiktionary sqwiki sqwikibooks sqwikinews sqwikiquote srwikibooks srwikinews srwikiquote srwikisource srwiktionary skwiki svwikisource tawiki tawikibooks tawikinews tawikiquote tawikisource tawiktionary testwiki thwikibooks thwikinews thwikiquote thwikisource thwiktionary trwikiquote trwiktionary uawikimedia ukwikibooks ukwikinews ukwikiquote ukwikisource ukwikivoyage ukwiktionary viwikibooks viwikiquote viwikisource viwikivoyage viwiktionary wbwikimedia
s5:
  shwiki srwiki
s6:
  frwiki ruwiki
s7:
  eswiki fawiki frwiktionary hewiki huwiki rowiki ukwiki viwiki

So we'll need to run on s1, s2, s3, s5, s6, s7. (s5 is the only diff from last time at T189295: it wasn't necessary then as shwiki and srwiki were part of s3.)

Change 639751 merged by Muehlenhoff:
[operations/puppet@production] Add a Hiera option to enable ICU63 component

https://gerrit.wikimedia.org/r/639751

Change 639780 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable ICU 63 in mwdebug2002

https://gerrit.wikimedia.org/r/639780

Mentioned in SAL (#wikimedia-operations) [2020-11-09T17:33:53Z] <effie> updating mwdebug2002 to ICU 63 - T264991

We run the following on mwdebug2002:

# 'export DEBIAN_FRONTEND=noninteractive; apt-get -y libicu63 libxml2'
# 'export DEBIAN_FRONTEND=noninteractive; apt-get install php7.2-bcmath php7.2-bz2 php7.2-cli php7.2-common php7.2-curl php7.2-dba php7.2-fpm php7.2-gd php7.2-gmp php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php-apcu php-cli php-common php-excimer php-geoip php-igbinary php-luasandbox php-memcached php-mongodb php-msgpack php-redis php-tideways-xhprof php-wmerrors -y'

(comment is edited to reflect the final list)

That's not enough, these are just the binary packages from the php72 source package, but you also need to upgrade php-apcu, php-cli, php-common, php-excimer, php-geoip, php-igbinary, php-luasandbox, php-memcached, php-mongodb, php-msgpack, php-redis, php-tideways-xhprof and php-wmerrors

done

@jijiki reading the task it is not clear to me what's needed from us (DBA). Is it a heads up that you'll be running updateCollation.php against the wikis listed on T264991#6609917?

@Marostegui yes, it is a headsup for your radar, thank you!

Thank you! What's the expected impact of updateCollation.php?

One other sanity check for the rollout (in particular when the whole server batch gets upgraded on the 16th);

cumin foo* 'php -r "var_dump(IntlChar::getUnicodeVersion());"'

should consistently return

array(4) {
  [0]=>
  int(11)
  [1]=>
  int(0)
  [2]=>
  int(0)
  [3]=>
  int(0)
}

I will updating to icu63 in deployment-prep, with Moritz looking on. This will likely happen later today, and I'll post updates about the progress.

In T264991#6615421, @Marostegui wrote:

Thank you! What's the expected impact of updateCollation.php?

Many many categorylinks rows being updated. And to some extent, an increase in reads (based on cl_collation, which is indexed); in batches of 100.

In T264991#6616214, @Reedy wrote:

In T264991#6615421, @Marostegui wrote:

Thank you! What's the expected impact of updateCollation.php?

Many many categorylinks rows being updated. And to some extent, an increase in reads (based on cl_collation, which is indexed); in batches of 100.

We will run on one wiki and see how it looks, we expect the longest job to be enwiki which previously took about a week. Tx @Reedy for the input

Thank you @Reedy!

Upgrade plan on deployment-prep:

• add profile::mediawiki::php::icu63: true to hiera for deployment-prep project prefix; this will only have impact on hosts including the appropriate mediawiki manifest
• apt-get update on deploy*, mediawiki*, jobrunner*, parsoid*, snapshot*
• export DEBIAN_FRONTEND=noninteractive; apt-get install -y libicu63 libxml2 on all of the above
• export DEBIAN_FRONTEND=noninteractive; apt-get install php7.2-bcmath php7.2-bz2 php7.2-cli php7.2-common php7.2-curl php7.2-dba php7.2-fpm php7.2-gd php7.2-gmp php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php-apcu php-cli php-common php-excimer php-geoip php-igbinary php-luasandbox php-memcached php-mongodb php-msgpack php-redis php-tideways-xhprof php-wmerrors -y on all of the above

During the last step, the update of php7.2-fpm will trigger a restart which will get the new libxml and libicu, and we can verify that the update happened by

• php -r "var_dump(IntlChar::getUnicodeVersion());" on all the above and expect 11.0.0.0 instead of 8.0.0.0

At tis point the changes will be live and testing can begin.

Sounds good!

Note that I'm running

cumin  'O{project:deployment-prep  name:^deployment-mediawiki-[0-9]+$ } or O{project:deployment-prep  name:^deployment-parsoid[0-9]+$ } or O{project:deployment-prep  name:^deployment-snapshot[0-9]+$ } or O{project:deployment-prep  name:^deployment-jobrunner[0-9]+$ } or O{project:deployment-prep  name:^deployment-mwmaint[0-9]+$ }'

which targets the following instances:

deployment-jobrunner03.deployment-prep.eqiad1.wikimedia.cloud,deployment-mediawiki-[07,09].deployment-prep.eqiad1.wikimedia.cloud,
deployment-mwmaint01.deployment-prep.eqiad1.wikimedia.cloud,deployment-parsoid11.deployment-prep.eqiad1.wikimedia.cloud,
deployment-snapshot02.deployment-prep.eqiad1.wikimedia.cloud

Can't proceed at the moment, puppet sync to deployment-prep has been broken since Nov 6. Log excerpts from the earliest error:

2020-11-06T20:00:06Z INFO     git.cmd: git rev-parse --abbrev-ref HEAD -> 0; stdout: 'master'
2020-11-06T20:00:06Z INFO     git.cmd: git rev-parse master -> 0; stdout: 'c52d47210553bde2e89735a73637b918911d0226'
2020-11-06T20:00:06Z INFO     git.cmd: git merge-base 63bd567cfb4a5dfbeeb28e1908fb3f8b976d6a4e HEAD -> 0; stdout: '63bd567cfb4a5dfbeeb28e1908fb3f8b976d6a4e'
2020-11-06T20:00:06Z INFO     sync-upstream: Up-to-date: /var/lib/git/labs/private
2020-11-06T20:10:01Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
2020-11-06T20:10:01Z ERROR    sync-upstream: Local diffs detected.  Commit your changes!
2020-11-06T20:20:01Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
2020-11-06T20:20:01Z ERROR    sync-upstream: Local diffs detected.  Commit your changes!

git info:

root@deployment-puppetmaster04:/var/lib/git/operations/puppet/modules/profile/manifests/mediawiki# git status
HEAD detached from 0bd812c4d7
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   ../../../confd/manifests/init.pp

no changes added to commit (use "git add" and/or "git commit -a")

The diff:

root@deployment-puppetmaster04:/var/lib/git/operations/puppet/modules/confd/manifests# git diff init.pp
diff --git a/modules/confd/manifests/init.pp b/modules/confd/manifests/init.pp
index 29be11d873..15898051cb 100644
--- a/modules/confd/manifests/init.pp
+++ b/modules/confd/manifests/init.pp
@@ -29,7 +29,7 @@ class confd(
     Boolean          $running       = true,
     String           $backend       = 'etcd',
     Optional[String] $node          = undef,
-    Stdlib::Fqdn     $srv_dns       = $::domain,
+    String           $srv_dns       = $::domain,
     String           $scheme        = 'https',
     Integer          $interval      = 3,
     Boolean          $monitor_files = true,

Anyone know who was doing this work?

This is apparently from T267439 After some discussion with jbond and dancy in irc, I am going to revert that and hope I'm not making the varnish situation worse. Also adding @thcipriani so he is informed.

Puppet sync back to working. Back on track to continue with the update in deployment-prep.

Update in deployment-prep is now complete, assuming I did not miss any hosts.

root@deployment-cumin:~# cumin  'O{project:deployment-prep  name:^deployment-mediawiki-[0-9]+$ } or O{project:deployment-prep  name:^deployment-parsoid[0-9]+$ } or O{project:deployment-prep  name:^deployment-snapshot[0-9]+$ } or O{project:deployment-prep  name:^deployment-jobrunner[0-9]+$ } or O{project:deployment-prep  name:^deployment-mwmaint[0-9]+$ }'  'php -r "var_dump(IntlChar::getUnicodeVersion());"'
6 hosts will be targeted:
deployment-jobrunner03.deployment-prep.eqiad1.wikimedia.cloud,deployment-mediawiki-[07,09].deployment-prep.eqiad1.wikimedia.cloud,deployment-mwmaint01.deployment-prep.eqiad1.wikimedia.cloud,deployment-parsoid11.deployment-prep.eqiad1.wikimedia.cloud,deployment-snapshot02.deployment-prep.eqiad1.wikimedia.cloud
Confirm to continue [y/n]? y

===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |   0% (0/6) [00:00<?, ?hosts/s]
(6) deployment-jobrunner03.deployment-prep.eqiad1.wikimedia.cloud,deployment-mediawiki-[07,09].deployment-prep.eqiad1.wikimedia.cloud,deployment-mwmaint01.deployment-prep.eqiad1.wikimedia.cloud,deployment-parsoid11.deployment-prep.eqiad1.wikimedia.cloud,deployment-snapshot02.deployment-prep.eqiad1.wikimedia.cloud
                                                                                                                                                                                                                                                                                                                                                                      |   0% (0/6) [00:00<?, ?hosts/s]
----- OUTPUT of 'php -r "var_dump...codeVersion());"' -----                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |   0% (0/6) [00:00<?, ?hosts/s]
array(4) {                                                                                                                                                                                           [0]=>   
  int(11)
  [1]=>
  int(0)
  [2]=>
  int(0)
  [3]=>
  int(0)
}
<snip>

Mentioned in SAL (#wikimedia-releng) [2020-11-10T18:40:25Z] <apergos> deployment-prep: switched deployment-prep to libicu63, see T264991

Mentioned in SAL (#wikimedia-operations) [2020-11-16T16:08:35Z] <effie> disable puppet in appservers canaries to install ICU 63 - T264991

Change 641203 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] hiera: Enable icu63 component on api canaries

https://gerrit.wikimedia.org/r/641203

Change 641200 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: enable icu 63 component on appserver canaries

https://gerrit.wikimedia.org/r/641200

Note that I ran a little dumps test on a non-latin1 wiki in deployment-prep (ruwiki to be precise) and the results look just fine. But I didn't do any sort of comprehensive testing like MW integration tests or whatever they do over there.

Mentioned in SAL (#wikimedia-operations) [2020-11-16T16:16:37Z] <rzl> disable puppet on A:mw-api-canary T264991

Change 641203 merged by RLazarus:
[operations/puppet@production] hiera: Enable icu63 component on api canaries

https://gerrit.wikimedia.org/r/641203

Change 641200 merged by Effie Mouzeli:
[operations/puppet@production] hiera: enable icu 63 component on appserver canaries

https://gerrit.wikimedia.org/r/641200

Change 641213 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] hiera: Enable icu63 component on api servers

https://gerrit.wikimedia.org/r/641213

Change 641214 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: enable icu 63 component on appservers

https://gerrit.wikimedia.org/r/641214

Mentioned in SAL (#wikimedia-operations) [2020-11-16T18:15:15Z] <rzl> disable puppet on 'A:mw-api and not A:mw-api-canary' T264991

Change 641213 merged by RLazarus:
[operations/puppet@production] hiera: Enable icu63 component on api servers

https://gerrit.wikimedia.org/r/641213

Change 641214 merged by Effie Mouzeli:
[operations/puppet@production] hiera: enable icu 63 component on appservers

https://gerrit.wikimedia.org/r/641214

Mentioned in SAL (#wikimedia-operations) [2020-11-16T19:48:57Z] <effie> disable puppet on parsoid servers - T264991

Change 641234 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: enable icu 63 component on appservers

https://gerrit.wikimedia.org/r/641234

Change 641234 merged by Effie Mouzeli:
[operations/puppet@production] hiera: enable icu 63 component on parsoid

https://gerrit.wikimedia.org/r/641234

Change 641249 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] hiera: Enable icu63 component on jobrunners

https://gerrit.wikimedia.org/r/641249

Change 641249 merged by RLazarus:
[operations/puppet@production] hiera: Enable icu63 component on jobrunners

https://gerrit.wikimedia.org/r/641249

Change 641255 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: enable icu 63 component on snapshot

https://gerrit.wikimedia.org/r/641255

Mentioned in SAL (#wikimedia-operations) [2020-11-16T21:07:22Z] <rzl> disable puppet on jobrunners T264991

Change 641255 merged by Effie Mouzeli:
[operations/puppet@production] hiera: enable icu 63 component on snapshot

https://gerrit.wikimedia.org/r/641255

Change 641265 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] hiera: Enable icu63 component on maintenance hosts

https://gerrit.wikimedia.org/r/641265

Change 641265 merged by RLazarus:
[operations/puppet@production] hiera: Enable icu63 component on maintenance hosts

https://gerrit.wikimedia.org/r/641265

We upgraded to ICU 63: appservers, api, parsoid, jobrunners, mwmaint, and snapshot. What is left is deploy*.

We are running 6 updateCollation.php scripts at the same time, each operating on different shards.

Is it blocking the edition on wikis (i.e. does it force them to be shutdown for extended period, beside the short time to upload the new collation tables in the database and restarting it)?
Or does it only affect how categories will be temporarily listed?
Also I don't understand why upgrading ICU on the host OS affects the ICU version used by the database (or its clients running on separate processes on any host, or the MediaWiki plugins, e.g. for PHP or Lua which may also have their own version of ICU).

These should be decoupled and should not block the OS upgrade. There may be several independant versions of ICU deployed, that can be upgraded separately (and not all run on the same host): this may give some quirks only if there are assumptiosn that they should all collate the same way.

Also ICU integrates other components : date/number formatting, new version of CLDR data (that may be upgraded as well, e.g. for names of languages, scripts, countries, or timezones), plus some additions for normalizations when there are new encoded characters (normalization is stable on all existing characters, this only affects the default normalization of characters which were initially not mapped of the standard, so it usually does not affect all the scripts we support, even for the large additions of sinograms, whose normalization is still themselves; this only possibly affects their collation; other additions are for emojis but they don't cause normalization problems)

Other additions like descriptive keywords for emojis have very low impact on wikis. And most other additions are for minority scripts that we initially did not support, such as Hanifi Royinga, which also requires free font support to be available for clients, or provided by webfonts).

In fact most components using ICU only use it for normalization and normalization is stable, so it should not be blocking anything.

@Verdy_p: Feel free to ask general understanding questions on https://www.mediawiki.org/wiki/Talk:Wikimedia_Technical_Operations/ICU_announcement . Thanks.

Change 643694 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: Enable icu63 component on deployment hosts

https://gerrit.wikimedia.org/r/643694

Change 643694 merged by Effie Mouzeli:
[operations/puppet@production] hiera: Enable icu63 component on deployment hosts

https://gerrit.wikimedia.org/r/643694

Change 644248 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: enable icu 63 component on labweb hosts

https://gerrit.wikimedia.org/r/644248

Mentioned in SAL (#wikimedia-operations) [2020-12-03T15:52:02Z] <effie> upgrading labweb* to ICU 63 - T264991

Change 644248 merged by Effie Mouzeli:
[operations/puppet@production] hiera: enable icu 63 component on labweb hosts

https://gerrit.wikimedia.org/r/644248

Change 968616 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Remove deprecated hiera keys from icu63 upgrade

https://gerrit.wikimedia.org/r/968616

Change 968616 merged by JMeybohm:

[operations/puppet@production] Remove deprecated hiera keys from icu63 upgrade

https://gerrit.wikimedia.org/r/968616