The Beta-Cluster-Infrastructure MediaWiki databases are updated on an hourly basis by running MediaWiki maintenance/update.php. The job is beta-update-databases-eqiad.
The last known build occured on July 25th at 19:20 UTC. The next one is failing, it started at 20:20 UTC and failed at 20:31 UTC.
The segfault triggers when running the Wikifunctions database update:
+ /usr/local/bin/mwscript update.php --wiki=wikifunctionswiki --quick --skip-config-validation 20:31:09 ...wikilambda_zlanguages table already exists. 20:31:09 ...have wlzl_label_primary field in wikilambda_zobject_labels table. 20:31:09 ...have wlzl_return_type field in wikilambda_zobject_labels table. 20:31:09 /usr/local/bin/mwscript: line 27: 1822 Segmentation fault sudo -u "$MEDIAWIKI_WEB_USER" $PHP "$MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE/multiversion/MWScript.php" "$@"
For the previous (good build) update.php gives:
19:30:18 ...have wlzl_label_primary field in wikilambda_zobject_labels table. 19:30:18 ...have wlzl_return_type field in wikilambda_zobject_labels table. 19:30:18 ❌ Unable to make a page for Z7138: The provided content's label clashes with Object 'Z10138' for the label in 'Z1002'. 19:30:18 ❌ Unable to make a page for Z7139: The provided content's label clashes with Object 'Z10139' for the label in 'Z1002'. 19:30:18 ❌ Unable to make a page for Z7140: The provided content's label clashes with Object 'Z10140' for the label in 'Z1002'. 19:30:18 ...site_stats is populated...done.
Thus it looks like the attempt to add wlzl_return_type field in wikilambda_zobject_labels table triggers the issue.
Four changes got submitted to WikiLambda between 19:20 UTC and 20:20 UTC:
- 19:26 Handle oldid url param to view a particular revision
- 19:43 AUTHORS: Update for July 2023
- 19:59 Update function-schemata sub-module to HEAD (1c01f22) for T335583
- New changes:
- abc4aa6 definitions: Add Z1908/bug-bugi and Z1909/bug-lant ZNaturalLanguages
- 0f1941e definitions: Add Z1910/piu ZNaturalLanguage
- 1c01f22 definitions: Re-label all objects to drop the 'Z' per Amin
- 20:16 PageRenderingHandler: Don't make 'read' selected if we're on the edit tab
The function-schemata update looks suspicious :)
From the instance deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud:
[Mon Jul 24 23:33:55 2023] php[28392]: segfault at 7ffe374f5db8 ip 00007f8dc59fc807 sp 00007ffe374f5da0 error 6 in libpcre2-8.so.0.7.1[7f8dc59b9000+5d000] [Mon Jul 24 23:33:55 2023] Code: ff ff 31 ed e9 74 fb ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 55 48 89 d5 53 44 89 c3 48 81 ec 98 52 00 00 <48> 89 7c 24 18 4c 8b a4 24 d0 52 00 00 48 89 74 24 10 48 89 4c 24 [Mon Jul 24 23:33:55 2023] Core dump to |/usr/lib/systemd/systemd-coredump 28392 33 33 11 1690242166 0 php pipe failed
[Mon Jul 24 23:35:10 2023] php[28479]: segfault at 7ffebb86b898 ip 00007f154a088807 sp 00007ffebb86b880 error 6 in libpcre2-8.so.0.7.1[7f154a045000+5d000] [Mon Jul 24 23:35:10 2023] Code: ff ff 31 ed e9 74 fb ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 55 48 89 d5 53 44 89 c3 48 81 ec 98 52 00 00 <48> 89 7c 24 18 4c 8b a4 24 d0 52 00 00 48 89 74 24 10 48 89 4c 24 [Mon Jul 24 23:35:10 2023] Core dump to |/usr/lib/systemd/systemd-coredump 28479 33 33 11 1690242241 0 php pipe failed
At 23:21:53 another coredump shows an error in the coredump config:
[Tue Jul 25 23:21:53 2023] systemd-coredump[21045]: /etc/systemd/coredump.conf:6: Failed to parse size value, ignoring: 20%
And of course /var/lib/systemd/coredump is empty :]