Page MenuHomePhabricator

debmonitor send status update before the package actually got upgraded
Closed, ResolvedPublic

Description

When doing an upgrade, debmonitor send the update status before the package being actually upgraded. I noticed that when upgrading Jenkins on contint1001.wikimedia.org. https://debmonitor.wikimedia.org/packages/jenkins already shown the package upgraded but the ugprade hasn't actually be done.

$ sudo apt upgrade
...
Get:1 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/thirdparty jenkins all 2.176.2 [77.2 MB]
...
Fetched 88.1 MB in 1s (57.9 MB/s)                                          
No directory, logging in with HOME=/
INFO:debmonitor:Got 14 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server

At this point I guess the update has been send. Then:

(Reading database ... 61447 files and directories currently installed.)
...
Unpacking jenkins (2.176.2) over (2.176.1) ...

There is thus a possibility for a package to fail to upgrade but be listed as having been upgraded.

Event Timeline

hashar created this task.Jul 17 2019, 3:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2019, 3:20 PM
Volans triaged this task as Normal priority.Jul 17 2019, 6:40 PM
Volans added subscribers: MoritzMuehlenhoff, Volans.

@hashar debmonitor uses Dpkg::Pre-Install-Pkgs for this feature because it's the only available hook from APT/DPKG that gives us the details of the operation that is made.
Any other option seemed very suboptimal. Things like save some temporary data with all the concurrency and stale risks involved + run something in Post-Invoke or send the full list each time in Post-Invoke or having a 2-way commit on the server side.

We normally install upgrades via debdeploy in production and that keeps track of any failure so that the operator can fix them and very rarely we run a full apt upgrade.

FWIW There is also a daily crontab that sends the whole list to Debmonitor to reconcile any possible discrepancy.
Thoughts?

There is thus a possibility for a package to fail to upgrade but be listed as having been upgraded.

Not really; as we don't leave packages in broken state (Icinga alerts on that), the corrective action will trigger the next apt hook which updates the state.

@Volans sounds good. I guess my concern was to potentially have a wrong state, but the daily crontab would indeed align the debmonitor database with the reality. So that addresses my concern :-]

I have filled this task merely to raise some awareness to a potential issue. But that seems well covered (single package install via debmonitor, Icinga alert etc). Feel free to mark this task resolved!

Volans closed this task as Resolved.Jul 18 2019, 8:01 AM
Volans claimed this task.

Ack, thanks.