Install wikidiff2 1.14.1 deb on deployment-prep & test
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

As mentioned in

In T340087#8958132, @MoritzMuehlenhoff wrote:

I've built a package and for now copied it to https://people.wikimedia.org/~jmm/wikidiff/

If it passes basic functional tests (maybe in deploment-prep?), the next step could be to roll this out to the canaries.

One note: https://github.com/wikimedia/mediawiki-php-wikidiff2/commit/c11002017454cec638517a26af0032202a0c0868 mentions the removal of wikidiff2.ini. It's in fact a required file for dh-php, the Debhelper build integration, I've readded it as a Debian-specific patch under debian/patches. I have no idea if other distros also need/use it.

~~Install php7.4-wikidiff2-dbgsym_1.14.0-0+wmf1+buster1_amd64.deb and php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb on~~;
- deployment-mediawiki11
- deployment-mediawiki12
Testing TBC

Details

	Subject	Repo	Branch	Lines +/-
	Exclude equality when comparing similarity metrics	mediawiki/php/wikidiff2	master	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		akosiaris	T340087 Deploy wikidiff2 1.14.1
		Resolved		TheresNoTime	T340542 Install wikidiff2 1.14.1 deb on deployment-prep & test

Event Timeline

TheresNoTime created this task.Jun 27 2023, 11:40 AM

TheresNoTime added a project: Beta-Cluster-Infrastructure.Jun 27 2023, 11:50 AM

Mentioned in SAL (#wikimedia-releng) [2023-06-27T12:23:21Z] <TheresNoTime> deployment-prep: ran sudo apt install /home/samtar/php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb on deployment-mediawiki[11-12] for T340542, watching logs

nb.

samtar@deployment-mediawiki12:~$ php --ri wikidiff2 | grep version
wikidiff2 version => 1.13.0
samtar@deployment-mediawiki12:~$ sudo apt install ./php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'php7.4-wikidiff2' instead of './php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb'
The following packages were automatically installed and are no longer required:
  fonts-beng-extra fonts-deva-extra fonts-gubbi fonts-guru-extra fonts-kalapi fonts-lohit-beng-assamese fonts-lohit-beng-bengali fonts-lohit-deva fonts-lohit-gujr fonts-lohit-guru fonts-lohit-knda fonts-lohit-mlym fonts-lohit-orya
  fonts-lohit-taml fonts-lohit-taml-classical fonts-lohit-telu fonts-nakula fonts-navilu fonts-orya-extra fonts-sahadeva fonts-samyak-deva fonts-samyak-gujr fonts-samyak-mlym fonts-samyak-taml fonts-smc-anjalioldlipi
  fonts-smc-chilanka fonts-smc-dyuthi fonts-smc-karumbi fonts-smc-keraleeyam fonts-smc-manjari fonts-smc-meera fonts-smc-rachana fonts-smc-raghumalayalamsans fonts-smc-suruma fonts-smc-uroob fonts-telu-extra fonts-tlwg-garuda
  fonts-tlwg-garuda-ttf fonts-tlwg-kinnari fonts-tlwg-kinnari-ttf fonts-tlwg-laksaman fonts-tlwg-laksaman-ttf fonts-tlwg-loma fonts-tlwg-loma-ttf fonts-tlwg-mono fonts-tlwg-mono-ttf fonts-tlwg-norasi fonts-tlwg-norasi-ttf
  fonts-tlwg-purisa fonts-tlwg-purisa-ttf fonts-tlwg-sawasdee fonts-tlwg-sawasdee-ttf fonts-tlwg-typewriter fonts-tlwg-typewriter-ttf fonts-tlwg-typist fonts-tlwg-typist-ttf fonts-tlwg-typo fonts-tlwg-typo-ttf fonts-tlwg-umpush
  fonts-tlwg-umpush-ttf fonts-tlwg-waree fonts-tlwg-waree-ttf fonts-yrsa-rasa libaec0 libcfitsio7 libexif12 libfontenc1 libgfortran5 libgsf-1-114 libgsf-1-common libharfbuzz-icu0 libhdf5-103 libilmbase23 libkpathsea6 libmatio4
  libopenexr23 liborc-0.4-0 libpotrace0 libptexenc1 libquadmath0 libsynctex2 libsz2 libteckit0 libtexlua52 libtexlua53 libtexluajit2 libtidy5deb1 libwoff1 libxaw7 libxmu6 libxt6 libxxhash0 libzzip-0-13 t1utils tex-common
  xfonts-encodings xfonts-utils
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  php7.4-wikidiff2
1 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
Need to get 0 B/77.9 kB of archives.
After this operation, 34.8 kB of additional disk space will be used.
Get:1 /home/samtar/php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb php7.4-wikidiff2 amd64 1.14.0-0+wmf1+buster1 [77.9 kB]
(Reading database ... 75181 files and directories currently installed.)
Preparing to unpack .../php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb ...
Unpacking php7.4-wikidiff2 (1.14.0-0+wmf1+buster1) over (1.13.0-1+wmf1+buster1) ...
Setting up php7.4-wikidiff2 (1.14.0-0+wmf1+buster1) ...

Configuration file '/etc/php/7.4/mods-available/wikidiff2.ini'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
Processing triggers for php7.4-fpm (1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2) ...
NOTICE: Not enabling PHP 7.4 FPM by default.
NOTICE: To enable PHP 7.4 FPM in Apache2 do:
NOTICE: a2enmod proxy_fcgi setenvif
NOTICE: a2enconf php7.4-fpm
NOTICE: You are seeing this message because you have apache2 package installed.
Processing triggers for php7.4-cli (1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2) ...
samtar@deployment-mediawiki12:~$ php --ri wikidiff2 | grep version
wikidiff2 version => 1.14.0
samtar@deployment-mediawiki12:~$ pwd
/home/samtar
samtar@deployment-mediawiki12:~$ ls
php7.4-wikidiff2_1.14.0-0+wmf1+buster1_amd64.deb

TheresNoTime updated the task description. (Show Details)Jun 27 2023, 2:29 PM

TheresNoTime added a subscriber: dom_walden.

TheresNoTime moved this task from Ready 🎬 to In Development 💻 on the Community-Tech (CommTech-Kanban) board.Jun 28 2023, 10:17 AM

TheresNoTime moved this task from In Development 💻 to Review/Feedback 💬 on the Community-Tech (CommTech-Kanban) board.Jun 28 2023, 1:07 PM

I tested API:Compare on most of the beta wikis[1], just checking that it could generate a diff without returning an exception. For each wiki, I tested a few hundred diffs[2] (sometimes less if the wiki did not have many articles). I tested both inline and two-column diffs.

I kept an eye on beta's logstash. I did not see any errors that seemed to be associated with diffs.

There are circumstances where diffs are different with the latest wikidiff2. I believe, with the default settings, wikidiff2 sometimes allows for more permissive matches[3].

For example, compare:

Or compare:

Notes

Every different language wikipedia and every English project plus Commons and Wikidata (see https://commons.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix)
Using this script https://gitlab.wikimedia.org/dwalden/betterdiffs-testing/-/blob/main/random_diffs_stress.py
For more information, see https://tools-static.wmflabs.org/betterdiffsreport/report.html

In T340542#8973199, @dom_walden wrote:

There are circumstances where diffs are different with the latest wikidiff2. I believe, with the default settings, wikidiff2 sometimes allows for more permissive matches

General question; should we (attempt to) tweak the settings to more closely match the current wikidiff2 match behaviour prior to moving forward with T340087: Deploy wikidiff2 1.14.1, or is the matching behaviour change "a universal improvement" which can be kept as-is?

In T340542#8973199, @dom_walden wrote:

Or compare:

https://en.wiktionary.org/w/index.php?title=Category:Japanese%20terms%20spelled%20with%20%E6%95%97%20read%20as%20%E3%81%AF%E3%81%84&oldid=61447301&diff=22165765

https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Category%3AJapanese_terms_spelled_with_%E6%95%97_read_as_%E3%81%AF%E3%81%84&diff=589524&oldid=589523

The similarity between the LHS and RHS is exactly equal to the threshold of 0.2. In e2ead0b14af3b63ab35b5094b90134376d7887b6 I accidentally changed the relevant comparison from > to >=. So the lines are judged to be similar in the new wikidiff2. So it is indeed possible to make the new wikidiff2 work like the old one by increasing wikidiff2.change_threshold from 0.2 to 0.2000001. But it is probably better to just revert the change to the operator. Avoiding user-visible changes will make it easier to detect bugs.

Change 935188 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/php/wikidiff2@master] Exclude equality when comparing similarity metrics

https://gerrit.wikimedia.org/r/935188

gerritbot added a project: Patch-For-Review.Jul 4 2023, 5:41 AM

Change 935188 merged by jenkins-bot:

[mediawiki/php/wikidiff2@master] Exclude equality when comparing similarity metrics

https://gerrit.wikimedia.org/r/935188

Maintenance_bot removed a project: Patch-For-Review.Jul 4 2023, 9:30 AM

@MoritzMuehlenhoff (see above for context) — we've released 1.14.1 to fix a bug, could you rebuild the debs and host them, so I can install the update on the beta cluster?

Sure, I'll update the package later the day.

We also need to rebuild the base php-fpm images for mediawiki on k8s

In T340542#8987623, @TheresNoTime wrote:

@MoritzMuehlenhoff (see above for context) — we've released 1.14.1 to fix a bug, could you rebuild the debs and host them, so I can install the update on the beta cluster?

@TheresNoTime Updated packages are at https://people.wikimedia.org/~jmm/wikidiff/

In T340542#8989804, @MoritzMuehlenhoff wrote:

In T340542#8987623, @TheresNoTime wrote:

@MoritzMuehlenhoff (see above for context) — we've released 1.14.1 to fix a bug, could you rebuild the debs and host them, so I can install the update on the beta cluster?

@TheresNoTime Updated packages are at https://people.wikimedia.org/~jmm/wikidiff/

Many thanks!

1.14.1 now installed on beta

samtar@deployment-mediawiki12:~$ php --ri wikidiff2 | grep version
wikidiff2 version => 1.14.1

akosiaris moved this task from Incoming 🐫 to Stalled 🐌 on the serviceops board.Jul 6 2023, 3:27 PM

@GMikesell-WMF and I have finished our testing here. We have not found any more problems. Moving to Done.

Thank you! Work continues at T340087: Deploy wikidiff2 1.14.1

tstarling mentioned this in T341754: Deploy wikidiff2 paragraph split detection.Jul 13 2023, 2:22 AM

Install wikidiff2 1.14.1 deb on deployment-prep & testClosed, ResolvedPublic3 Estimated Story PointsActions