Page MenuHomePhabricator
Create Task
Maniphest T368744

Allow debmonitor to store the Debian version-id in the OS field
Closed, ResolvedPublic
Actions

Description

Debmonitor should show the Debian version-id in the OS column, instead of simply "Debian".

The change involves two systems:

  • debmonitor-client - the script needs to be able to recognize the version-id and push it to the debmonitor backend.
  • debmonitor (backend) - the django app needs to be able to update the OS version according to what debmonitor-client reports.

The change for debmonitor-client was merged in https://gerrit.wikimedia.org/r/c/operations/software/debmonitor-client/+/1043780, and deployed manually on build2001 only (other nodes being reimaged may pick up the new version as well).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
src_packages: add migration for OS modelVolansoperations/software/debmonitormaster+27 -0
hosts/views.py: add logging when upgrading the host's OSElukeyoperations/software/debmonitormaster+5 -0
hosts/views.py: upgrade OS when existing upgradable pkgs are foundElukeyoperations/software/debmonitormaster+8 -0
Move debmonitor discovery record to debmonitor2003Elukeyoperations/dnsmaster+1 -1
Release version 0.5.0-1Elukeyoperations/software/debmonitordebian+46 -8
Hosts: automatically migrate to a new OSVolansoperations/software/debmonitormaster+61 -0
Allow to save new OS names without them being present on the DBElukeyoperations/software/debmonitormaster+84 -33
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Jun 28 2024, 3:31 PM
gerritbot added a comment.Jun 28 2024, 3:32 PM

Change #1049966 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/debmonitor@master] Allow to save new OS names without them being present on the DB

https://gerrit.wikimedia.org/r/1049966

gerritbot added a project: Patch-For-Review.Jun 28 2024, 3:32 PM
elukey added a comment.EditedJun 28 2024, 3:34 PM

docker-reporter-base-images.service on build2001 reports an issue with the dcl-puppet-client image:

[2024-06-28T04:15:48] Unable to update image 'docker-registry.wikimedia.org/dcl-puppet-client:bookworm'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/debmonitor/images/views.py", line 165, in update_image
    _update_v1(request, name, os, payload)
  File "/usr/lib/python3/dist-packages/debmonitor/images/views.py", line 211, in _update_v1
    _process_upgradable(im, os, images_packages, existing_upgradable_not_updated, item)
  File "/usr/lib/python3/dist-packages/debmonitor/images/views.py", line 280, in _process_upgradable
    existing.save()
  File "/usr/lib/python3/dist-packages/debmonitor/images/models.py", line 137, in save
    self.full_clean()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'package_version': ['OS mismatch between emacs-nox 1:28.2+1-15 (Debian) and docker-registry.wikimedia.org/dcl-puppet-client:bookworm (Debian 12)']}
elukey added a comment.Jun 28 2024, 4:17 PM

On db1195 I see for emacs-nox:

MariaDB [debmonitor]> select * from bin_packages_package where name = 'emacs-nox'
    -> ;
+-----+-----------+
| id  | name      |
+-----+-----------+
| 609 | emacs-nox |

MariaDB [debmonitor]> select * from bin_packages_packageversion where package_id = '609';
| id    | version              | created                    | modified                   | os_id | package_id | src_package_version_id |

| 60738 | 1:27.1+1-3.1+deb11u5 | 2024-06-25 20:00:12.779158 | 2024-06-25 20:00:12.779208 |     1 |        609 |                  31477 |
| 60749 | 1:27.1+1-3.1+deb11u5 | 2024-06-25 20:12:03.426410 | 2024-06-25 20:12:03.426456 |     4 |        609 |                  31480 |

| 51866 | 1:28.2+1-15          | 2023-05-21 02:51:24.809095 | 2023-05-21 02:51:24.809208 |     1 |        609 |

Version 1:28.2+1-15 has indeed the old Debian OS value, but other versions have both old and new one (see os_id 1 vs 4 for 1:27.1+1-3.1+deb11u5). My understanding is that build2001 is trying to update docker-registry.wikimedia.org/dcl-puppet-client:bookworm with a list of packages that includes emacs-nox version 1:28.2+1-15, but on save the foreign key on bin_packages_packageversion fails because there is not version 1:28.2+1-15 with the OS id related to Debian 12 (the one carried by the Docker image).

I checked other package versions and they report both os_ids (1 and 5), so I checked that it is not a bookworm specific issue (the os-id 4 above is related to Bullseye).

At this point the only explanation that I can give is that running debmonitor-client on docker-registry.wikimedia.org/dcl-puppet-client:bookworm leads, for some reason, to have packages tagged with "Debian" OS?

elukey added a comment.Jun 28 2024, 4:29 PM

Ok I see, I ran debmonitor inside the dcl image:

"os": "Debian 12",
"uninstalled": [],
"update_type": "full",
"upgradable": [
    {
        "name": "linux-perf",
        "source": "linux",
        "version_from": "6.1.85-1",
        "version_to": "6.1.90-1"
    },
    {
        "name": "emacs-nox",
        "source": "emacs",
        "version_from": "1:28.2+1-15",
        "version_to": "1:28.2+1-15+deb12u3"
    },

Probably the culprit is that in _process_upgradable (image/views.py) we should modify existing = image_packages.get(item['name'], None) to take into consideration also the OS version?

elukey triaged this task as Medium priority.Jul 1 2024, 2:28 PM
elukey added a comment.Jul 1 2024, 4:27 PM

After a chat with Riccardo some things came up:

  • It seems that the issue comes up when debmonitor-client is upgraded on a node without a clean up first (like on a reimage, since we delete the host's debmonitor state while doing it).
  • Forcing a clean via spicerack and a subsequent debmonitor run makes the db state ok.

Attempted fix for a single host:

>>> spicerack.debmonitor().host_delete('build2001.codfw.wmnet')
>>> spicerack.remote().query('build2001.codfw.wmnet').run_sync('debmonitor-client-unpriv')

Attempted fix for images:

for image in `sudo journalctl -u docker-reporter-base-images.service  | grep fail | grep Jul| cut -d " " -f 11`; do sudo curl -X DELETE "https://debmonitor.discovery.wmnet/images/${image}" --cert "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_')-key.pem"; done

And then a run of docker-reporter-base-images.

Info: https://wikitech.wikimedia.org/wiki/DebMonitor#Manually_remove_an_image_from_DebMonitor

gerritbot added a comment.Jul 2 2024, 9:58 AM

Change #1051299 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/debmonitor@master] Images: automatically migrate to a new OS

https://gerrit.wikimedia.org/r/1051299

gerritbot added a comment.Jul 2 2024, 1:05 PM

Change #1049966 merged by jenkins-bot:

[operations/software/debmonitor@master] Allow to save new OS names without them being present on the DB

https://gerrit.wikimedia.org/r/1049966

elukey moved this task from Backlog to In Progress on the User-Elukey board.Jul 8 2024, 1:30 PM
gerritbot added a comment.Jul 11 2024, 1:08 PM

Change #1051299 merged by jenkins-bot:

[operations/software/debmonitor@master] Hosts: automatically migrate to a new OS

https://gerrit.wikimedia.org/r/1051299

Maintenance_bot removed a project: Patch-For-Review.Jul 11 2024, 1:31 PM
elukey added a comment.Jul 11 2024, 5:00 PM

Next steps:

  • create a new debmonitor (server side) release and upgrade.
  • rollout a little subset of servers to the new debmonitor-client, and check that no errors are raised.
  • finish the rollout of debmonitor-client to the whole fleet.
elukey moved this task from In Progress to Waiting for others on the User-Elukey board.Jul 12 2024, 3:57 PM
gerritbot added a comment.Jul 17 2024, 1:12 PM

Change #1054879 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/debmonitor@debian] Release version 0.5.0-1

https://gerrit.wikimedia.org/r/1054879

gerritbot added a project: Patch-For-Review.Jul 17 2024, 1:12 PM
elukey moved this task from Waiting for others to In Progress on the User-Elukey board.Jul 17 2024, 2:07 PM
elukey moved this task from In Progress to Waiting for others on the User-Elukey board.Jul 24 2024, 3:41 PM
gerritbot added a comment.Jul 31 2024, 6:47 AM

Change #1054879 merged by Elukey:

[operations/software/debmonitor@debian] Release version 0.5.0-1

https://gerrit.wikimedia.org/r/1054879

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 7:30 AM
Stashbot added a comment.Jul 31 2024, 12:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T12:57:35Z] <elukey> update debmonitor-server and python3-debmonitor to bookworm-wikimedia - T368744

elukey added a comment.EditedAug 1 2024, 3:21 PM

Tried to test the new debmonitor-server on debmonitor2003:

  • changed sretest1001 /etc/hosts to point debmonitor.discovery.wmnet to debmonitor2003's ip
  • dropped via spicerack the data on debmonitor for the host
  • ran the debmonitor client on the host

This is what I got in the logs:

[2024-08-01T15:03:20] Unable to update host 'sretest1001'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
hosts.models.HostPackage.DoesNotExist: HostPackage matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 160, in update
    _update_v1(request, name, os, payload)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 194, in _update_v1
    _process_installed(host, os, host_packages, existing_not_updated, item)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 251, in _process_installed
    host_packages[package_version.package.name], _ = HostPackage.objects.get_or_create(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/models.py", line 136, in save
    self.full_clean()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'__all__': ['Host package with this Host and Binary package already exists.']}
[2024-08-01T15:03:20] Internal Server Error: /hosts/sretest1001.eqiad.wmnet/update

It happens also for debmonitor 0.4.0-3, so it must be something related to the standby node (maybe). Will need to check how it is configured to figure out a good rollout strategy.

Tested today and it doesn't seem to happen with debmonitor1003.

gerritbot added a comment.Aug 6 2024, 12:09 PM

Change #1060094 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Move debmonitor discovery record to debmonitor2003

https://gerrit.wikimedia.org/r/1060094

gerritbot added a project: Patch-For-Review.Aug 6 2024, 12:09 PM
gerritbot added a comment.Aug 6 2024, 2:27 PM

Change #1060094 abandoned by Elukey:

[operations/dns@master] Move debmonitor discovery record to debmonitor2003

https://gerrit.wikimedia.org/r/1060094

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2024, 2:31 PM
elukey mentioned this in T371899: Review how the debmonitor server processes hosts/images when starting fresh.Aug 6 2024, 3:05 PM

The issue is described in T371899. I proceeded anyway to upgrade both debmonitor server hosts, all good so far.

Next step: upgrade the debmonitor-client fleetwide.

elukey added a comment.Aug 7 2024, 8:41 AM

Rolled out the change to the hadoop cluster, this is the only error that I got:

[2024-08-07T08:38:59] Unable to update host 'an-worker1101.eqiad.wmnet'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
bin_packages.models.PackageVersion.DoesNotExist: PackageVersion matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 168, in update
    _update_v1(request, name, os, payload)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 203, in _update_v1
    _host_package_migrate_os(os, host_package)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 242, in _host_package_migrate_os
    package_version, _ = PackageVersion.objects.get_or_create(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/debmonitor/bin_packages/models.py", line 43, in get_or_create
    return super().get_or_create(**arguments)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/debmonitor/bin_packages/models.py", line 83, in save
    self.full_clean()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'__all__': ['Package version with this Package, Version and Operating system already exists.']}
[2024-08-07T08:38:59] Internal Server Error: /hosts/an-worker1101.eqiad.wmnet/update

It seems similar to the timeout issue highlighted above, I think that due to the high number of packages we may incur in this issue when updating a lot of packages.

elukey added a comment.Aug 7 2024, 1:20 PM

Buster and Bookworm rollouts done, no big issues registered. The only drawback is that due to the high volume of writes to the db (since we are changing the Debian version etc..) the UI gets not responsive for a bit, and we get some alarms. This is due to T371899.

elukey moved this task from Waiting for others to In Progress on the User-Elukey board.Aug 7 2024, 1:24 PM
elukey closed this task as Resolved.Aug 13 2024, 12:58 PM
elukey reopened this task as Open.Aug 14 2024, 2:37 PM

Found another issue:

[2024-08-14T13:33:13] Unable to update host 'kubernetes1051.eqiad.wmnet'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 168, in update
    _update_v1(request, name, os, payload)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 227, in _update_v1
    _process_upgradable(host, os, host_packages, existing_upgradable_not_updated, item)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 318, in _process_upgradable
    existing.save()
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/models.py", line 136, in save
    self.full_clean()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'package_version': ['OS mismatch between linux-perf 5.10.221-1 (Debian) and kubernetes1051.eqiad.wmnet (Debian 11)']}

I usually notice those as debmonitor failures to root@. A quick fix it it happens, from something like spicerack_shell.py (see my home dir on cumin1002):

elukey@cumin1002:~$ sudo ./spicerack_shell.py 
DEBUG:git.cmd:Popen(['git', 'version'], cwd=/home/elukey, universal_newlines=False, shell=None, istream=None)
DEBUG:git.cmd:Popen(['git', 'version'], cwd=/home/elukey, universal_newlines=False, shell=None, istream=None)
>>> spicerack.remote().query('kafka-main2003.codfw.wmnet').run_sync('debmonitor-client-unpriv')
>>> spicerack.remote().query('kafka-main2003.codfw.wmnet').run_sync('debmonitor-client-unpriv')
elukey added a comment.Aug 27 2024, 1:30 PM

Today I cleaned up some db nodes reported as debmonitor client failures while I was on holiday:

>>> spicerack.debmonitor().host_delete('db1246.eqiad.wmnet')
INFO:spicerack.debmonitor:Removed host db1246.eqiad.wmnet from Debmonitor
Removed host db1246.eqiad.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db1238.eqiad.wmnet')
INFO:spicerack.debmonitor:Removed host db1238.eqiad.wmnet from Debmonitor
Removed host db1238.eqiad.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2148.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2148.codfw.wmnet from Debmonitor
Removed host db2148.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2205.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2205.codfw.wmnet from Debmonitor
Removed host db2205.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2168.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2168.codfw.wmnet from Debmonitor
Removed host db2168.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2203.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2203.codfw.wmnet from Debmonitor
Removed host db2203.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2220.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2220.codfw.wmnet from Debmonitor
Removed host db2220.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db1179.eqiad.wmnet')
INFO:spicerack.debmonitor:Removed host db1179.eqiad.wmnet from Debmonitor
Removed host db1179.eqiad.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db1246.eqiad.wmnet')
INFO:spicerack.debmonitor:Removed host db1246.eqiad.wmnet from Debmonitor
Removed host db1246.eqiad.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db1238.eqiad.wmnet')
INFO:spicerack.debmonitor:Host db1238.eqiad.wmnet already missing on Debmonitor
Host db1238.eqiad.wmnet already missing on Debmonitor
>>> spicerack.debmonitor().host_delete('db2148.codfw.wmnet')
INFO:spicerack.debmonitor:Host db2148.codfw.wmnet already missing on Debmonitor
Host db2148.codfw.wmnet already missing on Debmonitor
>>> spicerack.debmonitor().host_delete('db2205.codfw.wmnet')
INFO:spicerack.debmonitor:Removed host db2205.codfw.wmnet from Debmonitor
Removed host db2205.codfw.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2168.codfw.wmnet')
INFO:spicerack.debmonitor:Host db2168.codfw.wmnet already missing on Debmonitor
Host db2168.codfw.wmnet already missing on Debmonitor
>>> spicerack.debmonitor().host_delete('db2203.codfw.wmnet')
INFO:spicerack.debmonitor:Host db2203.codfw.wmnet already missing on Debmonitor
Host db2203.codfw.wmnet already missing on Debmonitor
>>> spicerack.debmonitor().host_delete('db1179.eqiad.wmnet')
INFO:spicerack.debmonitor:Removed host db1179.eqiad.wmnet from Debmonitor
Removed host db1179.eqiad.wmnet from Debmonitor
>>> spicerack.debmonitor().host_delete('db2168.codfw.wmnet')
INFO:spicerack.debmonitor:Host db2168.codfw.wmnet already missing on Debmonitor
Host db2168.codfw.wmnet already missing on Debmonitor

Their data will be refilled during the next debmonitor client run (happening within 24 hours).

gerritbot added a comment.Aug 27 2024, 1:43 PM

Change #1067354 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/debmonitor@master] hosts/views.py: add logging when upgrading the host's OS

https://gerrit.wikimedia.org/r/1067354

gerritbot added a project: Patch-For-Review.Aug 27 2024, 1:43 PM
elukey moved this task from In Progress to Stalled on the User-Elukey board.Aug 29 2024, 1:31 PM
MoritzMuehlenhoff subscribed.Sep 3 2024, 2:48 PM
elukey added a comment.EditedSep 3 2024, 2:50 PM

Happened again for:

[2024-09-03T14:39:43] Unable to update host 'lvs3009.esams.wmnet'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 168, in update
    _update_v1(request, name, os, payload)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 227, in _update_v1
    _process_upgradable(host, os, host_packages, existing_upgradable_not_updated, item)
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/views.py", line 318, in _process_upgradable
    existing.save()
  File "/usr/lib/python3/dist-packages/debmonitor/hosts/models.py", line 136, in save
    self.full_clean()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'package_version': ['OS mismatch between udev 247.3-7+deb11u4 (Debian) and lvs3009.esams.wmnet (Debian 11)']}

The hosts' uptime is ~29 days, and /var/log/apt/history doesn't indicate anything changing recently about udev. The package do have some extra candidates for install though:

elukey@lvs3009:~$ apt-cache policy udev
udev:
  Installed: 247.3-7+deb11u4
  Candidate: 247.3-7+deb11u6
  Version table:
     252.29-1~deb12u1~bpo11+1 100
        100 http://mirrors.wikimedia.org/debian bullseye-backports/main amd64 Packages
     247.3-7+deb11u6 500
        500 http://security.debian.org/debian-security bullseye-security/main amd64 Packages
     247.3-7+deb11u5 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
 *** 247.3-7+deb11u4 500
        500 http://mirrors.wikimedia.org/debian bullseye-updates/main amd64 Packages
        100 /var/lib/dpkg/status

From IRC:

<volans> the last good call to debmonitor from lvs3009 was at 2024-09-02T12:01:15, the first failed one at 2024-09-02T12:31:11 and then it failed every 30m
<volans> because of the puppet timer, once the wrong data is there

From puppet's log:

Sep  2 12:01:15 lvs3009 puppet-agent-cronjob: INFO:debmonitor:Found 590 installed binary packages
Sep  2 12:01:15 lvs3009 puppet-agent-cronjob: INFO:debmonitor:Found 46 upgradable binary packages (including new dependencies)
Sep  2 12:01:15 lvs3009 puppet-agent-cronjob: INFO:debmonitor:Successfully sent the upgradable update to the DebMonitor server


Sep  2 12:31:10 lvs3009 puppet-agent-cronjob: INFO:debmonitor:Found 590 installed binary packages
Sep  2 12:31:10 lvs3009 puppet-agent-cronjob: INFO:debmonitor:Found 48 upgradable binary packages (including new dependencies)
Sep  2 12:35:12 lvs3009 puppet-agent-cronjob: ERROR:debmonitor:Failed to execute DebMonitor CLI: HTTPSConnectionPool(host='debmonitor.discovery.wmnet', port=443): Max retries exceeded with url: /hosts/lvs3009.esams.wmnet/update (Caused by ResponseError('too many 500 error responses'))

Indeed it seems that two new upgradable packages were registered (46 vs 48 in the above logs).

gerritbot added a comment.Sep 4 2024, 8:50 AM

Change #1070540 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/debmonitor@master] hosts/views.py: upgrade OS when existing upgradable pkgs are found

https://gerrit.wikimedia.org/r/1070540

gerritbot added a comment.Sep 4 2024, 9:46 AM

Change #1070540 abandoned by Elukey:

[operations/software/debmonitor@master] hosts/views.py: upgrade OS when existing upgradable pkgs are found

https://gerrit.wikimedia.org/r/1070540

gerritbot added a comment.Sep 4 2024, 9:46 AM

Change #1067354 abandoned by Elukey:

[operations/software/debmonitor@master] hosts/views.py: add logging when upgrading the host's OS

https://gerrit.wikimedia.org/r/1067354

elukey closed this task as Resolved.Sep 4 2024, 9:48 AM

@Volans checked in the debmonitor's DB and it seemed that only lvs3009 and cumin2002 were holding packages with the "Debian" label, so we cleaned them up and ran the debmonitor client manually (to get a fresh copy). Riccardo also ran GC manually for Host/Image packages via systemctl start debmonitor-maintenance-gc.service on debmonitor1003:

Deleted 417 PackageVersion objects not referenced by any HostPackage or ImagePackage
Deleted 274 SrcPackageVersion objects not referenced by any PackageVersion

The best explanation that we can give is that due to some race conditions, some hosts were not properly upgraded and ended up in errors as outlined above. In theory this should be enough to avoid future errors.

Optimistically closing, I'll reopen if new errors come to root@

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2024, 10:31 AM
gerritbot added a comment.Jun 24 2025, 1:54 PM

Change #1163368 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/debmonitor@master] src_packages: add migration for OS model

https://gerrit.wikimedia.org/r/1163368

gerritbot added a project: Patch-For-Review.Jun 24 2025, 1:54 PM
gerritbot added a comment.Jun 25 2025, 4:07 PM

Change #1163368 merged by jenkins-bot:

[operations/software/debmonitor@master] src_packages: add migration for OS model

https://gerrit.wikimedia.org/r/1163368

Maintenance_bot removed a project: Patch-For-Review.Jun 25 2025, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits