Page MenuHomePhabricator

apt error in clouddb1001
Closed, ResolvedPublic

Description

While working on T301949 and trying to install MariaDB 10.1 in the new VM tools-db-1, I mistakenly typed this command on clouddb1001 instead of tools-db-1:

# dpkg -i wmf-mariadb101_10.1.44-1_amd64.deb
...
Unpacking wmf-mariadb101 (10.1.44-1) over (10.1.39-1)

This would upgrade the MariaDB package from 10.1.39 to 10.1.44, but as I realized I was running this on the wrong server and it could cause some issues on the ToolsDB primary database, I pressed Ctrl+C which left the package in an inconsistent state. @taavi attempted a fix by reinstalling the deb file wmf-mariadb101_10.1.39-1_amd64.deb found in /home/jynus (as that version is no longer available in the wikimedia repo).

Puppet is still unhappy though:

E: The package wmf-mariadb101 needs to be reinstalled, but I can't find an archive for it.
ERR: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-mariadb101' returned 100

Event Timeline

fnegri changed the task status from Open to In Progress.Jan 30 2023, 10:05 AM
fnegri triaged this task as High priority.

Change 884833 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] P:wmcs::services: simplify toolsdb pinning

https://gerrit.wikimedia.org/r/884833

Mentioned in SAL (#wikimedia-cloud) [2023-01-30T10:53:33Z] <dhinus> restarting mariadb in clouddb1001 to apply minor version upgrade (T328273)

Change 884833 merged by FNegri:

[operations/puppet@production] P:wmcs::services: simplify toolsdb pinning

https://gerrit.wikimedia.org/r/884833

Mentioned in SAL (#wikimedia-cloud) [2023-01-30T11:02:03Z] <dhinus> 'SET GLOBAL read_only = 0;' after restarting mariadb in clouddb1001 (T328273)

Restarting caused 10 minutes of read-only time because I forgot you need to run SET GLOBAL read_only = 0; manually after a restart. :/

I created T328282 to consider creating a cookbook (cc @dcaro).