Page MenuHomePhabricator
Create Task
Maniphest T387276

PHP on Beta cluster fails due to mismatching PCRE
Closed, ResolvedPublic
Actions

Description

Warning: PHP Startup: ^(text/|application/xhtml\+xml) (offset=0): unrecognised compile-time option bit(s) in Unknown on line 0
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/multiversion/MWScript.php on line 118
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/php-master/maintenance/includes/MaintenanceRunner.php on line 291
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/php-master/maintenance/includes/MaintenanceRunner.php on line 314
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php on line 110
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/php-master/includes/libs/http/MultiHttpClient.php on line 696
Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki-staging/php-master/includes/libs/http/MultiHttpClient.php on line 696

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
php: use component/pcre2 when using Php 8.1operations/puppetproduction+5 -0
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Feb 26 2025, 8:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 26 2025, 8:51 AM
hashar triaged this task as Unbreak Now! priority.Feb 26 2025, 8:53 AM

That is due to T386006 and new php 8.1 packages that have been uploaded and automatically installed:

Start-Date: 2025-02-26  06:21:32
Commandline: /usr/bin/unattended-upgrade
Upgrade: php8.1-bcmath:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-opcache:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-mysql:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-curl:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-intl:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-mbstring:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-common:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-readline:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-gd:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-bz2:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-cli:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-dba:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-fpm:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-gmp:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4), php8.1-xml:amd64 (8.1.31-1+wmf11u3, 8.1.31-1+wmf11u4)
End-Date: 2025-02-26  06:22:15

Start-Date: 2025-02-26  06:22:16
Commandline: /usr/bin/unattended-upgrade
Upgrade: php8.1-apcu:amd64 (5.1.23-1+wmf11u3, 5.1.23-1+wmf11u4)
End-Date: 2025-02-26  06:22:18

But they need a matching libpcre2-8 which is in another apt component (component/pcre2).

hashar added subscribers: Scott_French, bd808.Feb 26 2025, 9:04 AM

deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud has:

/etc/apt/sources.list.d/wikimedia-php81.list
deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/php81
deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/php81

From December 17th 17:45. If I remove it:

Notice: /Stage[main]/Profile::Mediawiki::Php/Apt::Repository[wikimedia-php81]/File[/etc/apt/sources.list.d/wikimedia-php81.list]/ensure:
defined content as '{sha256}f6a9310fa9ca1920c0e7c0630149a6c709c29638d31898f2f860568b3a2f1fb4' (corrective)

That is not operations/puppet but is a cherry pick of a pending change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1085471 .

gerritbot added a comment.Feb 26 2025, 9:10 AM

Change #1122901 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] php: use component/pcre2 when using Php 8.1

https://gerrit.wikimedia.org/r/1122901

gerritbot added a project: Patch-For-Review.Feb 26 2025, 9:10 AM
Stashbot added a comment.Feb 26 2025, 9:11 AM

Mentioned in SAL (#wikimedia-releng) [2025-02-26T09:11:38Z] <hashar> deployment-prep: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1122901 "php: use component/pcre2 when using Php 8.1" to fix php # T387276

hashar mentioned this in T386006: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer.Feb 26 2025, 9:13 AM
hashar closed this task as Resolved.Feb 26 2025, 9:51 AM

I have manually upgraded libpcre2-8-0 on:

deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
deployment-jobrunner05.deployment-prep.eqiad1.wikimedia.cloud
deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud
deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud
deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud
deployment-snapshot05.deployment-prep.eqiad1.wikimedia.cloud

I think that is good enough for the purpose of this task: fixing the beta cluster. https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ passes now :)

Scott_French added a comment.Feb 26 2025, 3:05 PM

Ah, thanks for the fix, @hashar! It did not occur to me that these same packages were being used in Beta.

gerritbot added a comment.Mar 4 2025, 10:38 AM

Change #1122901 merged by Clément Goubert:

[operations/puppet@production] php: use component/pcre2 when using Php 8.1

https://gerrit.wikimedia.org/r/1122901

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2025, 11:30 AM
Daimona reopened this task as Open.Mar 18 2025, 5:47 PM
Daimona subscribed.

These seem to be occurring still, but only on deployment-jobrunner05. The rate is ~800 errors per minute. Random example: PHP Warning: preg_match(): Compilation failed: unrecognised compile-time option bit(s) at offset 0 in /srv/mediawiki/php-master/includes/libs/http/MultiHttpClient.php on line 722.

Daimona mentioned this in T389239: Bring beta-logs.wmcloud.org to a readable state.Mar 18 2025, 6:05 PM
Daimona added a comment.Mar 18 2025, 6:09 PM

Also noting that these entries have program: php7.2-fpm which is confusing. Presumably a reference to update somewhere (example)?

Stashbot added a comment.Mar 18 2025, 8:25 PM

Mentioned in SAL (#wikimedia-releng) [2025-03-18T20:25:03Z] <bd808> Rebooting deployment-jobrunner05 because things just seem weird (T387631, T387276)

Stashbot mentioned this in T387631: Are eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees jobs being processed in beta?.Mar 18 2025, 8:25 PM
bd808 closed this task as Resolved.Mar 18 2025, 8:54 PM

Rebooting deployment-jobrunner05 seems to have fixed it's damage. I don't see any smoking gun in the /var/log/apt/history.log* files like Hashar found back in T387276#10581787. The most recent mentions of php and pcre were from Hashar's actions in T387276#10581912. Maybe the php-fpm container didn't get restarted following the libpcre2 upgrade?

BTullis subscribed.Mar 20 2025, 10:05 AM

Just as a note, I also had to upgrade these packages manually on the snapshot hosts, after applying this: Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016

BTullis mentioned this in T319432: Migrate WMF production from PHP 7.4 to PHP 8.1.Mar 20 2025, 10:09 AM
thcipriani added a project: Essential-Work.May 8 2025, 1:45 PM
hashar mentioned this in T400693: PHP package must raise libpcre version requirement.Jul 29 2025, 12:52 PM

We had the same libpcre2 / PHP mismatch with the php 8.3 packages (T398245#11010808). I have filed T400693 to have it sorted out.

hashar mentioned this in T398245: Prepare WMF PHP 8.3 packages for bullseye.Jul 29 2025, 12:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits