| Marostegui | |
| Oct 30 2025, 8:10 AM |
| F68507197: Captura de pantalla 2025-10-30 a las 9.08.50.png | |
| Oct 30 2025, 8:10 AM |
When doing a reimage of es1033 with reuse-parts.cfg (where we basically preserve /srv dir) I am running into issues in the installer - see below screenshot
If I hit "No" the system installs correctly (of course without swap, which is not what we want but still, seems to be the only issue).
I've not tested the installation with partman/custom/db.cfg yet (which formats /srv)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Request | Jhancock.wm | T408407 decommission es2028 | ||
| Open | Marostegui | T422365 Migration to Debian Trixie of production database-related hosts | |||
| Resolved | Marostegui | T406981 Compile and test MariaDB 10.11 on Debian Trixie | |||
| Resolved | Marostegui | T407472 Install a testing db with Debian Trixie | |||
| Resolved | MoritzMuehlenhoff | T408777 Swap partition issue when installing a DB with Debian Trixie | |||
| Resolved | Marostegui | T410369 Install Debian Trixie on one s1 host | |||
| Resolved | Jclark-ctr | T410388 PXE failing on db1169 |
This also seems to happen with - partman/custom/db.cfg as well so not only with reuse-parts.cfg
I discussed with @MoritzMuehlenhoff whether it is a good moment to migrate our installs to UEFI and stop using legacy BIOS. We don't really have any strong preference on any or other method, whatever is easier for everyone.
The only requirement is to keep being able to reimage hosts with the option to keep /srv intact and another recipe to fully format the OS.
Change #1202047 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Remove es2028
Change #1202047 merged by Marostegui:
[operations/puppet@production] installserver: Remove es2028
Change #1202918 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] preseed: Configure es2028 with db.cfg
Change #1202918 merged by Muehlenhoff:
[operations/puppet@production] preseed: Configure es2028 with db.cfg
I've debugged this and the issue is unrelated to the Partman recipe, but what makes it fail is the pre-existing swap partition. The swap partition is correctly parsed and detected by d-i, but ultimately when the d-i component responsible for setting up swap (partman-basicfilesystems) attempts to create the swap partition using "mkswap /dev/sda2" this fails with
mkswap: write error: Text file busy
And in fact the existing swap partition is activated:
~ # cat /proc/swaps Filename Type Size Used Priority /dev/sda2 partition 7812092 0 -2
That also explains that if one acknowledges the error message over the serial console, the installation simply proceeds fine (since the pre-existing swap partition simply continues to be used).
Maybe something changed in an earlier part of d-i or maybe the initrd, so that a pre-existing swap partition from an earlier installation is now activated, but wasn't with Bookworm? Needs to be narrowed down further.
Change #1203011 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Add temporary trixie variant of db.cfg for debugging
Change #1203011 merged by Muehlenhoff:
[operations/puppet@production] Add temporary trixie variant of db.cfg for debugging
Change #1203030 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Add temporary trixie variant of db.cfg for debugging
Change #1203030 merged by Muehlenhoff:
[operations/puppet@production] Add temporary trixie variant of db.cfg for debugging
Change #1203770 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] partman: Test Partman recipe for DB reuse on trixie
Change #1203770 merged by Muehlenhoff:
[operations/puppet@production] partman: Test Partman recipe for DB reuse on trixie
Change #1203778 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Test Partman workaround to also cover the reuse workflow
Change #1203778 merged by Muehlenhoff:
[operations/puppet@production] Test Partman workaround to also cover the reuse workflow
Change #1203788 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] es2028: Test standard reuse recipe
Change #1203788 merged by Muehlenhoff:
[operations/puppet@production] es2028: Test standard reuse recipe
Change #1203806 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] reuse-db-trixie.cfg: Try different option
Change #1203806 merged by Muehlenhoff:
[operations/puppet@production] reuse-db-trixie.cfg: Try different option
Change #1203847 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Cleanup unused setting
Change #1203847 merged by Muehlenhoff:
[operations/puppet@production] Cleanup unused setting
The reason we haven't seen this behaviour so far is most probably because all the other roles on trixie either use the flat layout of a VM or an LVM-based Partman recipe (while the DB layouts don't use LVM).
I'm going to report this to Debian next week (and plan to propose a patch to address this by scanning currently used swap in partman-basicfilesystems and then running swapoff on them).
But to unblock us, I've also created a workaround:
As a test I installed es2028 with trixie, placed a dummy file in /srv and reinstalled with reuse-db-trixie.cfg and the file in /srv was kept.
One question is whether we want to integrate the trixie changes into the existing db variants. It's probably fine (given the conditions don't trigger on Bookworm), but needs testing for confirmation.
Thanks @MoritzMuehlenhoff for debugging and finding a workaround for this.
I think ideally we should try to integrate the changes so we don't have to keep track of both files whilst we do the migration. We are very far from migrating so maybe we can try to integrate them in the mean time, but this is of course up to you and if you'd have time for it.
This would simplify the migration, as we'd not need to maintain to different regex and move hosts while migrating them.
Thanks again for the time.
As we've discussed on IRC, I will try to reimage es2028 back to bookworm using partman/custom/reuse-db-trixie.cfg and see how it goes.
Change #1206067 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Configure es2028 with db-trixie.cfg
Change #1206067 merged by Marostegui:
[operations/puppet@production] installserver: Configure es2028 with db-trixie.cfg
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie
Change #1206195 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Reformat es2028
Change #1206195 merged by Marostegui:
[operations/puppet@production] installserver: Reformat es2028
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie completed:
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm executed with errors:
I've tried to reimage es2028 with bookworm formatting its /srv with db-trixie.cfg using: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1206067/2/modules/profile/data/profile/installserver/preseed.yaml#215 but /srv wasn't formatted.
Doing the same with Trixie resulted in the same, not getting /srvformatted
After talking to @MoritzMuehlenhoff we decided to test whether db.cfg works at all, as we've not used it for anything else apart from fresh installs for a while and indeed, it doesn't seem to be working: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1206195 this and a reimage to bookworm didn't format /srv/ for es2028.
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
We have to re-test everything because puppet has been disabled in apt1002 since Friday.
Change #1206319 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Add db-trixie.cfg to es2028
Change #1206319 merged by Marostegui:
[operations/puppet@production] installserver: Add db-trixie.cfg to es2028
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
Reimaged es2028 into bookworm with db-trixie.cfg and it correctly formatted /srv
I am now going to place a file in /srv and reimage it with reuse-db-trixie.cfg with bookworm and see if it is not erased.
Change #1206359 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Do not format /srv on es2028
Change #1206359 merged by Marostegui:
[operations/puppet@production] installserver: Do not format /srv on es2028
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm completed:
Change #1206362 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] partman: Apply workarounds to swap handling affecting trixie installations
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS bookworm completed:
Change #1206362 merged by Marostegui:
[operations/puppet@production] partman: Apply workarounds to swap handling affecting trixie installations
The installation worked and es2028 got installed with bookworm and the file wasn't deleted.
I've merged: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1206362 which is what makes db.cfg and db-reuse.cfg usable by both Bookworm and Trixie and then I will clean up es2028 from the preseed list.
es2028 will remain up before it gets decommissioned (T408407) so @MoritzMuehlenhoff can fix+test this upstream
Thanks a lot @MoritzMuehlenhoff for all the help!
Change #1206381 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Clean up es2028
Change #1206381 merged by Marostegui:
[operations/puppet@production] installserver: Clean up es2028
Change #1206708 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] installserver: Readd es2028 with modified db-trixie partman config
Change #1206708 merged by Muehlenhoff:
[operations/puppet@production] installserver: Readd es2028 with modified db-trixie partman config