Page MenuHomePhabricator
Create Task
Maniphest T395164

Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2)
Open, Needs TriagePublic
Actions

Description

T306066: Cloud VPS "cvn" project Stretch deprecation

Debian 11 is not being phased out yet, but WMF CI is removing PHP 7.4 as we speak, which means in order to maintain useful code coverage of tools like cvn-stillalive, we'll want to be running a newer PHP version, and I'm not about to maintain some local package override, so we might as well do the Debian upgrade a bit earlier than usual and get it over with.

Ref https://gerrit.wikimedia.org/r/c/integration/config/+/1127098/4/zuul/layout.yaml

    name: labs/countervandalism/stillalive
-   - name: composer-test-php74-to-php83
+   - name: composer-test-php81-to-php83

Instances

Useful links

Admin CC:
User:AntiCompositeNumber @AntiCompositeNumber
User:Azariv @Az1568
User:Barras @Barras
User:Cameron11598 @Cameron11598
User:Krinkle @Krinkle
User:Rxy @Rxy

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.May 23 2025, 7:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2025, 7:36 PM
Krinkle updated the task description. (Show Details)May 23 2025, 7:37 PM
Krinkle updated the task description. (Show Details)May 23 2025, 7:39 PM
gerritbot added a comment.May 23 2025, 7:41 PM

Change #1149737 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] docroot: Fix link to Grafana dashboard, remove old Nagf link

https://gerrit.wikimedia.org/r/1149737

gerritbot added a project: Patch-For-Review.May 23 2025, 7:41 PM
Stashbot added a comment.May 23 2025, 7:45 PM

Mentioned in SAL (#countervandalism) [2025-05-23T19:45:52Z] <Krinkle> Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), T395164)

gerritbot added a comment.May 23 2025, 8:02 PM

Change #1149739 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Adopt version-neutral php package aliases, remove unused php-apcu-bc

https://gerrit.wikimedia.org/r/1149739

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149738 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Restore `git clone` of cvn-infrastructure repo

https://gerrit.wikimedia.org/r/1149738

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149738 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Restore `git clone` of cvn-infrastructure repo

https://gerrit.wikimedia.org/r/1149738

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149737 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] docroot: Fix link to Grafana dashboard, remove old Nagf link

https://gerrit.wikimedia.org/r/1149737

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149739 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Adopt version-neutral php package aliases, remove unused php-apcu-bc

https://gerrit.wikimedia.org/r/1149739

gerritbot added a comment.May 23 2025, 8:20 PM

Change #1149744 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Autodiscover the /etc/php/X.Y directory

https://gerrit.wikimedia.org/r/1149744

Stashbot added a comment.May 23 2025, 8:22 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:22:24Z] <Krinkle> Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, T395164

Stashbot added a comment.May 23 2025, 8:29 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:29:04Z] <Krinkle> Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, T395164

Stashbot added a comment.May 23 2025, 8:30 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:30:20Z] <Krinkle> Shut off cvn-apache10, T395164

Krinkle claimed this task.May 23 2025, 8:30 PM

That's enough for today.

gerritbot added a comment.May 26 2025, 5:55 PM

Change #1149744 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Autodiscover the /etc/php/X.Y directory

https://gerrit.wikimedia.org/r/1149744

Stashbot added a comment.May 26 2025, 5:57 PM

Mentioned in SAL (#countervandalism) [2025-05-26T17:57:21Z] <Krinkle> Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref T395164

Krinkle mentioned this in T395274: Temporary quota increase for 'cvn'.May 26 2025, 6:04 PM
gerritbot added a comment.May 26 2025, 6:18 PM

Change #1150745 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Add `sudo mkdir -p /etc/cron.hourly`

https://gerrit.wikimedia.org/r/1150745

gerritbot added a comment.May 26 2025, 6:50 PM

Change #1150745 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Install `cron` by default

https://gerrit.wikimedia.org/r/1150745

Maintenance_bot removed a project: Patch-For-Review.May 26 2025, 7:30 PM
gerritbot added a comment.May 26 2025, 7:43 PM

Change #1150753 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Remove redundant ca-certificates-mono workaround

https://gerrit.wikimedia.org/r/1150753

gerritbot added a project: Patch-For-Review.May 26 2025, 7:43 PM
gerritbot added a comment.May 26 2025, 7:55 PM

Change #1150756 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] mysql_cvnclerkbot.sql: Dump fresh backup

https://gerrit.wikimedia.org/r/1150756

gerritbot added a comment.May 26 2025, 7:56 PM

Change #1150756 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] mysql_cvnclerkbot.sql: Dump fresh backup

https://gerrit.wikimedia.org/r/1150756

gerritbot added a comment.May 26 2025, 8:33 PM

Change #1150764 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly"

https://gerrit.wikimedia.org/r/1150764

gerritbot added a comment.May 26 2025, 8:43 PM

Change #1150764 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly"

https://gerrit.wikimedia.org/r/1150764

gerritbot added a comment.May 26 2025, 8:47 PM

Change #1150768 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13

https://gerrit.wikimedia.org/r/1150768

gerritbot added a comment.May 26 2025, 8:48 PM

Change #1150768 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13

https://gerrit.wikimedia.org/r/1150768

Krinkle updated the task description. (Show Details)May 28 2025, 3:48 PM
gerritbot added a comment.May 28 2025, 10:48 PM

Change #1151809 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

gerritbot added a comment.May 28 2025, 10:49 PM

Change #1151809 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

Krinkle added a comment.EditedMay 28 2025, 10:49 PM
In T395164#10866143, @gerritbot wrote:

Change #1151809 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

Reason for revert: Bot keeps going into a boot loop.

cvn-app13 CVNBot.exe[5403]: ERROR [Thread Pool Worker] CVNBot.Program [CVNBot17] Caught unhandled exception in global catcher
Mono.Data.Sqlite.SqliteException (0x80004005): Attempt to write a read-only database
  at Mono.Data.Sqlite.SQLite3.Reset
  at Mono.Data.Sqlite.SQLite3.Step
  at Mono.Data.Sqlite.SqliteDataReader.NextResult
  at Mono.Data.Sqlite.SqliteDataReader..ctor
  at (wrapper remoting-invoke-with-check)
  at Mono.Data.Sqlite.SqliteCommand.ExecuteReader
  at Mono.Data.Sqlite.SqliteCommand.ExecuteNonQuery
  at CVNBo.ListManager.CollectGarbage (System.Object stateInfo)
  at System.Threading.Timer+Scheduler.TimerCB (System.Object o)
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem
  at System.Threading.ThreadPoolWorkQueue.Dispatch
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback

Two issues:

  • For some reason, when run on the Debian 12, the database is almost immediately read-only preventing writes. Might be that the threads are locking each other out, but works fine on Debian 11.
  • For some reason, when it self-restarts on Debian 12, we end up with two processes, causing it to grow and clone out of control until the server grinds to a halt. Could be all sorts of issues, but again, restarts work fine on Debian 11.

What I'll start with, is to try to build/run with dotnet instead of msbuild/mono. Perhaps this is a known issue that has since been fixed.

gerritbot added a comment.May 29 2025, 3:23 PM

Change #1150753 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Remove redundant ca-certificates-mono workaround

https://gerrit.wikimedia.org/r/1150753

Maintenance_bot removed a project: Patch-For-Review.May 29 2025, 3:30 PM
fnegri closed subtask T395274: Temporary quota increase for 'cvn' as Declined.Jun 11 2025, 9:38 AM
gerritbot added a comment.Jul 28 2025, 1:14 PM

Change #1173380 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] vendor: Fix return type deprecation warning on PHP 8.4

https://gerrit.wikimedia.org/r/1173380

gerritbot added a project: Patch-For-Review.Jul 28 2025, 1:14 PM
gerritbot added a comment.Jul 28 2025, 1:17 PM

Change #1173384 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] build: Update php-parallel-lint to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173384

gerritbot added a comment.Jul 28 2025, 1:20 PM

Change #1173380 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] vendor: Update ulrichsg/getopt-php to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173380

gerritbot added a comment.Jul 28 2025, 1:20 PM

Change #1173384 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] build: Update php-parallel-lint to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173384

gerritbot added a comment.Jul 28 2025, 1:26 PM

Change #1173391 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] build: Update phan to fix crash on PHP 8.4

https://gerrit.wikimedia.org/r/1173391

gerritbot added a comment.Jul 28 2025, 1:28 PM

Change #1173391 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] build: Update phan to fix crash on PHP 8.4

https://gerrit.wikimedia.org/r/1173391

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 1:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits