Page MenuHomePhabricator
Create Task
Maniphest T429595

Explore apt pinning / hold for gitlab-ce package
Closed, ResolvedPublic
Actions

Description

GitLab upgrades are managed by the sre.gitlab.upgrade cookbook which surrounds the package upgrade with proper downtimes, backups and depooling.

However it's still possible to manually upgrade the gitlab-ce apt package and trigger an upgrade without this additional safeguards (which has happened in T426164).

So we should explore how we can prevent accidental upgrades with measures like apt-pinning or apt hold.

A brief research returned apt-mark hold is probably all we need:

hold
hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed.

unhold
unhold is used to cancel a previously set hold on a package to allow all actions again.

showhold
showhold is used to print a list of packages on hold in the same way as for the other show commands.

https://manpages.ubuntu.com/manpages/bionic/man8/apt-mark.8.html#prevent-changes-for-a-package

The cookbook could unhold the package, do a upgrade and hold it after a successful upgrade.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
sre.gitlab.upgrade: hold and unhold gitlab-ce packageJeltooperations/cookbooksmaster+7 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jelto created this task.Jun 18 2026, 12:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2026, 12:44 PM
LSobanski renamed this task from Explore apt pining / hold for gitlab-ce package to Explore apt pinning / hold for gitlab-ce package.Jun 19 2026, 7:47 AM
MoritzMuehlenhoff added a comment.Jun 19 2026, 9:17 AM

apt-mark hold sounds like the right solution. We can add apt::hold and apt::unhold defines in the apt class for it.

LSobanski assigned this task to Jelto.Jun 22 2026, 3:27 PM
LSobanski triaged this task as Medium priority.
LSobanski moved this task from Incoming to Work in Progress on the Collaboration-Services board.
gerritbot added a comment.Jun 23 2026, 12:44 PM

Change #1305112 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/cookbooks@master] sre.gitlab.upgrade: hold and unhold gitlab-ce package

https://gerrit.wikimedia.org/r/1305112

gerritbot added a project: Patch-For-Review.Jun 23 2026, 12:44 PM
gerritbot added a comment.Jun 23 2026, 1:05 PM

Change #1305112 merged by jenkins-bot:

[operations/cookbooks@master] sre.gitlab.upgrade: hold and unhold gitlab-ce package

https://gerrit.wikimedia.org/r/1305112

ops-monitoring-bot added a comment.Jun 23 2026, 1:13 PM

Cookbook cookbooks.sre.gitlab.upgrade was started by jelto@cumin1003 on GitLab host gitlab1003.wikimedia.org with reason: Test apt-mark hold

ops-monitoring-bot added a comment.Jun 23 2026, 1:16 PM

Cookbook cookbooks.sre.gitlab.upgrade started by jelto@cumin1003 on GitLab host gitlab1003.wikimedia.org with reason: Test apt-mark hold completed successfully on GitLab host gitlab1003.wikimedia.org with reason: Test apt-mark hold

Jelto closed this task as Resolved.Jun 23 2026, 1:26 PM
In T429595#12036556, @MoritzMuehlenhoff wrote:

apt-mark hold sounds like the right solution. We can add apt::hold and apt::unhold defines in the apt class for it.

Thank you for the feedback! In the change above I added the hold and unhold commands to sre.gitlab.upgrade. I'd like to avoid manual puppet patches to hold and unhold the package so moving this to the cookbook reduces the number of manual steps. Let me know if we missed anything here.

I tested the cookbook on a replica and after the cookbook execution the gitlab-ce package is set to hold. I also set the hold on all other GitLab hosts:

sudo cumin 'P{O:gitlab}' 'apt-mark showhold'
3 hosts will be targeted:
gitlab[1003-1004,2002].wikimedia.org
OK to proceed on 3 hosts? Enter the number of affected hosts to confirm or "q" to quit: 3
===== NODE GROUP =====                                                                                                                                        
(3) gitlab[1003-1004,2002].wikimedia.org                                                                                                                      
----- OUTPUT for command #1: 'apt-mark showhold' -----                                                                                                        
gitlab-ce
Maintenance_bot removed a project: Patch-For-Review.Jun 23 2026, 1:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits