Page MenuHomePhabricator
Create Task
Maniphest T395164

Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie
Closed, ResolvedPublic
Actions

Description

T306066: Cloud VPS "cvn" project Stretch deprecation

Debian 11 is not being phased out yet, but WMF CI is removing PHP 7.4 as we speak, which means in order to maintain useful code coverage of tools like cvn-stillalive, we'll want to be running a newer PHP version, and I'm not about to maintain some local package override, so we might as well do the Debian upgrade a bit earlier than usual and get it over with.

Ref https://gerrit.wikimedia.org/r/c/integration/config/+/1127098/4/zuul/layout.yaml

    name: labs/countervandalism/stillalive
-   - name: composer-test-php74-to-php83
+   - name: composer-test-php81-to-php83

Instances

Useful links

Admin CC:
User:AntiCompositeNumber @AntiCompositeNumber
User:Azariv @Az1568
User:Barras @Barras
User:Cameron11598 @Cameron11598
User:Krinkle @Krinkle
User:Rxy @Rxy

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
docs: Write "Move all bots" sectionKrinklelabs/countervandalism/cvn-infrastructuremaster+39 -12
Move all cvn-app12 tasks to cvn-app16Krinklelabs/countervandalism/stillalivemaster+25 -26
Move all cvn-app12 tasks to cvn-app15Krinklelabs/countervandalism/stillalivemaster+19 -27
docs: Update runbook for CVNBot python, rewrite in markdownKrinklelabs/countervandalism/cvn-infrastructuremaster+404 -357
Switch CVNClerkBot to cvn-app15Krinklelabs/countervandalism/stillalivemaster+15 -17
apache-config: Include wildcard to ease testingKrinklelabs/countervandalism/cvn-infrastructuremaster+9 -25
build: Update phan to fix crash on PHP 8.4Krinklelabs/countervandalism/stillalivemaster+1 -1
Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"Krinklelabs/countervandalism/stillalivemaster+4 -12
build: Update php-parallel-lint to fix PHP 8.4 deprecation warningKrinklelabs/countervandalism/stillalivemaster+1 -1
vendor: Update ulrichsg/getopt-php to fix PHP 8.4 deprecation warningKrinklelabs/countervandalism/stillalivemaster+363 -379
setup: Remove redundant ca-certificates-mono workaroundKrinklelabs/countervandalism/cvn-infrastructuremaster+1 -11
Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13Krinklelabs/countervandalism/stillalivemaster+12 -4
Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly"Krinklelabs/countervandalism/stillalivemaster+9 K -4 K
mysql_cvnclerkbot.sql: Dump fresh backupKrinklelabs/countervandalism/cvn-infrastructuremaster+5 -5
setup: Install `cron` by defaultKrinklelabs/countervandalism/cvn-infrastructuremaster+1 -0
setup: Autodiscover the /etc/php/X.Y directoryKrinklelabs/countervandalism/cvn-infrastructuremaster+6 -5
setup: Adopt version-neutral php package aliases, remove unused php-apcu-bcKrinklelabs/countervandalism/cvn-infrastructuremaster+7 -8
setup: Restore `git clone` of cvn-infrastructure repoKrinklelabs/countervandalism/cvn-infrastructuremaster+1 -0
docroot: Fix link to Grafana dashboard, remove old Nagf linkKrinklelabs/countervandalism/cvn-infrastructuremaster+1 -4
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2025, 7:36 PM
Krinkle updated the task description. (Show Details)May 23 2025, 7:37 PM
Krinkle updated the task description. (Show Details)May 23 2025, 7:39 PM
gerritbot added a comment.May 23 2025, 7:41 PM

Change #1149737 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] docroot: Fix link to Grafana dashboard, remove old Nagf link

https://gerrit.wikimedia.org/r/1149737

gerritbot added a project: Patch-For-Review.May 23 2025, 7:41 PM
Stashbot added a comment.May 23 2025, 7:45 PM

Mentioned in SAL (#countervandalism) [2025-05-23T19:45:52Z] <Krinkle> Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), T395164)

gerritbot added a comment.May 23 2025, 8:02 PM

Change #1149739 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Adopt version-neutral php package aliases, remove unused php-apcu-bc

https://gerrit.wikimedia.org/r/1149739

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149738 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Restore `git clone` of cvn-infrastructure repo

https://gerrit.wikimedia.org/r/1149738

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149738 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Restore `git clone` of cvn-infrastructure repo

https://gerrit.wikimedia.org/r/1149738

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149737 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] docroot: Fix link to Grafana dashboard, remove old Nagf link

https://gerrit.wikimedia.org/r/1149737

gerritbot added a comment.May 23 2025, 8:04 PM

Change #1149739 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Adopt version-neutral php package aliases, remove unused php-apcu-bc

https://gerrit.wikimedia.org/r/1149739

gerritbot added a comment.May 23 2025, 8:20 PM

Change #1149744 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Autodiscover the /etc/php/X.Y directory

https://gerrit.wikimedia.org/r/1149744

Stashbot added a comment.May 23 2025, 8:22 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:22:24Z] <Krinkle> Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, T395164

Stashbot added a comment.May 23 2025, 8:29 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:29:04Z] <Krinkle> Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, T395164

Stashbot added a comment.May 23 2025, 8:30 PM

Mentioned in SAL (#countervandalism) [2025-05-23T20:30:20Z] <Krinkle> Shut off cvn-apache10, T395164

Krinkle claimed this task.May 23 2025, 8:30 PM

That's enough for today.

gerritbot added a comment.May 26 2025, 5:55 PM

Change #1149744 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Autodiscover the /etc/php/X.Y directory

https://gerrit.wikimedia.org/r/1149744

Stashbot added a comment.May 26 2025, 5:57 PM

Mentioned in SAL (#countervandalism) [2025-05-26T17:57:21Z] <Krinkle> Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref T395164

Krinkle mentioned this in T395274: Temporary quota increase for 'cvn'.May 26 2025, 6:04 PM
gerritbot added a comment.May 26 2025, 6:18 PM

Change #1150745 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Add `sudo mkdir -p /etc/cron.hourly`

https://gerrit.wikimedia.org/r/1150745

gerritbot added a comment.May 26 2025, 6:50 PM

Change #1150745 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Install `cron` by default

https://gerrit.wikimedia.org/r/1150745

Maintenance_bot removed a project: Patch-For-Review.May 26 2025, 7:30 PM
gerritbot added a comment.May 26 2025, 7:43 PM

Change #1150753 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] setup: Remove redundant ca-certificates-mono workaround

https://gerrit.wikimedia.org/r/1150753

gerritbot added a project: Patch-For-Review.May 26 2025, 7:43 PM
gerritbot added a comment.May 26 2025, 7:55 PM

Change #1150756 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] mysql_cvnclerkbot.sql: Dump fresh backup

https://gerrit.wikimedia.org/r/1150756

gerritbot added a comment.May 26 2025, 7:56 PM

Change #1150756 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] mysql_cvnclerkbot.sql: Dump fresh backup

https://gerrit.wikimedia.org/r/1150756

gerritbot added a comment.May 26 2025, 8:33 PM

Change #1150764 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly"

https://gerrit.wikimedia.org/r/1150764

gerritbot added a comment.May 26 2025, 8:43 PM

Change #1150764 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly"

https://gerrit.wikimedia.org/r/1150764

gerritbot added a comment.May 26 2025, 8:47 PM

Change #1150768 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13

https://gerrit.wikimedia.org/r/1150768

gerritbot added a comment.May 26 2025, 8:48 PM

Change #1150768 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13

https://gerrit.wikimedia.org/r/1150768

Krinkle updated the task description. (Show Details)May 28 2025, 3:48 PM
gerritbot added a comment.May 28 2025, 10:48 PM

Change #1151809 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

gerritbot added a comment.May 28 2025, 10:49 PM

Change #1151809 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

Krinkle added a comment.EditedMay 28 2025, 10:49 PM
In T395164#10866143, @gerritbot wrote:

Change #1151809 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13"

https://gerrit.wikimedia.org/r/1151809

Reason for revert: Bot keeps going into a boot loop.

cvn-app13 CVNBot.exe[5403]: ERROR [Thread Pool Worker] CVNBot.Program [CVNBot17] Caught unhandled exception in global catcher
Mono.Data.Sqlite.SqliteException (0x80004005): Attempt to write a read-only database
  at Mono.Data.Sqlite.SQLite3.Reset
  at Mono.Data.Sqlite.SQLite3.Step
  at Mono.Data.Sqlite.SqliteDataReader.NextResult
  at Mono.Data.Sqlite.SqliteDataReader..ctor
  at (wrapper remoting-invoke-with-check)
  at Mono.Data.Sqlite.SqliteCommand.ExecuteReader
  at Mono.Data.Sqlite.SqliteCommand.ExecuteNonQuery
  at CVNBo.ListManager.CollectGarbage (System.Object stateInfo)
  at System.Threading.Timer+Scheduler.TimerCB (System.Object o)
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem
  at System.Threading.ThreadPoolWorkQueue.Dispatch
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback

Two issues:

  • For some reason, when run on the Debian 12, the database is almost immediately read-only preventing writes. Might be that the threads are locking each other out, but works fine on Debian 11.
  • For some reason, when it self-restarts on Debian 12, we end up with two processes, causing it to grow and clone out of control until the server grinds to a halt. Could be all sorts of issues, but again, restarts work fine on Debian 11.

What I'll start with, is to try to build/run with dotnet instead of msbuild/mono. Perhaps this is a known issue that has since been fixed.

gerritbot added a comment.May 29 2025, 3:23 PM

Change #1150753 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] setup: Remove redundant ca-certificates-mono workaround

https://gerrit.wikimedia.org/r/1150753

Maintenance_bot removed a project: Patch-For-Review.May 29 2025, 3:30 PM
fnegri closed subtask T395274: Temporary quota increase for 'cvn' as Declined.Jun 11 2025, 9:38 AM
gerritbot added a comment.Jul 28 2025, 1:14 PM

Change #1173380 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] vendor: Fix return type deprecation warning on PHP 8.4

https://gerrit.wikimedia.org/r/1173380

gerritbot added a project: Patch-For-Review.Jul 28 2025, 1:14 PM
gerritbot added a comment.Jul 28 2025, 1:17 PM

Change #1173384 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] build: Update php-parallel-lint to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173384

gerritbot added a comment.Jul 28 2025, 1:20 PM

Change #1173380 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] vendor: Update ulrichsg/getopt-php to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173380

gerritbot added a comment.Jul 28 2025, 1:20 PM

Change #1173384 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] build: Update php-parallel-lint to fix PHP 8.4 deprecation warning

https://gerrit.wikimedia.org/r/1173384

gerritbot added a comment.Jul 28 2025, 1:26 PM

Change #1173391 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] build: Update phan to fix crash on PHP 8.4

https://gerrit.wikimedia.org/r/1173391

gerritbot added a comment.Jul 28 2025, 1:28 PM

Change #1173391 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] build: Update phan to fix crash on PHP 8.4

https://gerrit.wikimedia.org/r/1173391

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 1:31 PM
Krinkle added a subtask: T419075: labs/countervandalism/cvn-api fails CI in PHP 8.4 and 8.5.Mar 24 2026, 6:51 AM
Krinkle closed subtask T419075: labs/countervandalism/cvn-api fails CI in PHP 8.4 and 8.5 as Resolved.Mar 24 2026, 7:53 AM
Andrew mentioned this in T430589: CVN: Replace Bullseye nfs server with a new Trixie server.Jul 16 2026, 5:42 PM
AntiCompositeNumber added a project: Cloud-VPS (Debian Bullseye Deprecation).Jul 31 2026, 1:14 AM
AntiCompositeNumber moved this task from Backlog to VPS projects on the Cloud-VPS (Debian Bullseye Deprecation) board.
AntiCompositeNumber added a comment.Aug 2 2026, 11:40 PM

Seems like the WMCS deadline for this has snuck up on us a bit.

Do we have any known blockers still? Are you (@Krinkle) working on it or should I start working on it?

Looking at https://openstack-browser.toolforge.org/project/cvn, cvn-app13 and cvn-app14 have been created on Bookworm. Would it make more sense to go directly to Trixie instead?

Krinkle added a comment.EditedAug 3 2026, 1:28 AM

Sure, feel free to destroy those and re-create with Trixie.

The "new" cvn-app Bookworm hosts aren't in use yet because we haven't decided between msbuild (invest effort in getting the old thing to run on the new OS, leaves local dev unsolved or hard to setup) and dotnet (various blockers and unknowns in making CVNBot compile and run properly, see example problem above).

komla merged a task: T435764: Migrate cvn away from Debian Bullseye to Bookworm/Trixie.Aug 24 2026, 8:40 AM
komla subscribed.
Krinkle added a comment.Aug 24 2026, 8:43 AM

Rather than invest in backporting old Mono to new Debian, or invest in migrating CVNBot to the newer dotnet/msbuild ecosystem; I'm instead picking up T327136: Rewrite CVNBot in Python, which feels like a better use of time.

There's a few bugs to work through, but I'm hopeful this should get us running within a couple more weeks.

AntiCompositeNumber added a subtask: T327136: Rewrite CVNBot in Python.Aug 31 2026, 7:31 PM
AntiCompositeNumber changed the status of subtask T327136: Rewrite CVNBot in Python from Open to In Progress.
Krinkle closed subtask T327136: Rewrite CVNBot in Python as Resolved.Sep 1 2026, 5:59 PM
Stashbot added a comment.Sep 1 2026, 6:01 PM

Mentioned in SAL (#countervandalism) [2026-09-01T18:01:17Z] <Krinkle> Delete cvn-app13 and cvn-app14 (empty Debian 12 Bookworm instances), to be replaced with Debian 13 Trixie. ref T395164

Krinkle renamed this task from Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) to Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie (PHP 7.4 > PHP 8.4).Sep 1 2026, 6:02 PM
Krinkle triaged this task as High priority.
gerritbot added a comment.Sep 1 2026, 7:19 PM

Change #1333282 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] apache-config: Include wildcard to ease testing

https://gerrit.wikimedia.org/r/1333282

gerritbot added a project: Patch-For-Review.Sep 1 2026, 7:19 PM

Change #1333282 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] apache-config: Include wildcard to ease testing

https://gerrit.wikimedia.org/r/1333282

Maintenance_bot removed a project: Patch-For-Review.Sep 1 2026, 7:31 PM
Stashbot added a comment.Sep 2 2026, 2:14 AM

Mentioned in SAL (#countervandalism) [2026-09-02T02:14:03Z] <Krinkle> Switch cvn.wmflabs.org and cvn.wmcloud.org web proxy from cvn-apache11 to cvn-apache12, ref T395164

gerritbot added a comment.Sep 2 2026, 4:06 AM

Change #1333458 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] docs: Update runbook for CVNBot python and rewrite in markdown

https://gerrit.wikimedia.org/r/1333458

gerritbot added a project: Patch-For-Review.Sep 2 2026, 4:06 AM
gerritbot added a comment.Sep 2 2026, 6:30 AM

Change #1333553 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Switch CVNClerkBot to cvn-app15

https://gerrit.wikimedia.org/r/1333553

gerritbot added a comment.Sep 2 2026, 6:36 AM

Change #1333553 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Switch CVNClerkBot to cvn-app15

https://gerrit.wikimedia.org/r/1333553

gerritbot added a comment.Sep 2 2026, 6:49 AM

Change #1333458 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] docs: Update runbook for CVNBot python, rewrite in markdown

https://gerrit.wikimedia.org/r/1333458

Maintenance_bot removed a project: Patch-For-Review.Sep 2 2026, 7:31 AM
gerritbot added a comment.Sep 2 2026, 7:32 AM

Change #1333592 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Move all cvn-app12 tasks to cvn-app15

https://gerrit.wikimedia.org/r/1333592

gerritbot added a project: Patch-For-Review.Sep 2 2026, 7:32 AM

Change #1333592 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Move all cvn-app12 tasks to cvn-app15

https://gerrit.wikimedia.org/r/1333592

gerritbot added a comment.Sep 2 2026, 7:54 AM

Change #1333620 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/stillalive@master] Move all cvn-app12 tasks to cvn-app16

https://gerrit.wikimedia.org/r/1333620

gerritbot added a comment.Sep 2 2026, 7:55 AM

Change #1333620 merged by jenkins-bot:

[labs/countervandalism/stillalive@master] Move all cvn-app12 tasks to cvn-app16

https://gerrit.wikimedia.org/r/1333620

gerritbot added a comment.Sep 2 2026, 7:55 AM

Change #1333655 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/countervandalism/cvn-infrastructure@master] docs: Write "Move all bots" section

https://gerrit.wikimedia.org/r/1333655

Krinkle closed this task as Resolved.Sep 2 2026, 8:07 AM
Krinkle removed a project: Patch-For-Review.
Krinkle updated the task description. (Show Details)
Stashbot added a comment.Sep 2 2026, 8:08 AM

Mentioned in SAL (#countervandalism) [2026-09-02T08:08:27Z] <Krinkle> Delete cvn-app10 and cvn-app12, ref T395164

Krinkle renamed this task from Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie (PHP 7.4 > PHP 8.4) to Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie.Sep 2 2026, 8:11 AM
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
gerritbot added a comment.Sep 2 2026, 8:34 AM

Change #1333655 merged by Krinkle:

[labs/countervandalism/cvn-infrastructure@master] docs: Write "Move all bots" section

https://gerrit.wikimedia.org/r/1333655

Krinkle mentioned this in T327136: Rewrite CVNBot in Python.Sep 3 2026, 3:30 AM
Krinkle mentioned this in T434103: Requests to extend shutdown timeline for Bullseye deprecation.Fri, Sep 25, 8:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits