Page MenuHomePhabricator

openstack db: figure out new versions for galera & mariadb
Closed, ResolvedPublic

Description

In the parent task T302146: Galera on cloudcontrol1004 going out of sync it is described the suffering we're experiencing lately with the openstack database.

At this point, we suspect of some problem with the galera/mariadb version combo.
There are some comments online further supporting this evidence:

This task is to track all the changes required to make new versions available and tested as a potential solution.

Event Timeline

Mentioned in SAL (#wikimedia-cloud) [2022-02-24T10:37:04Z] <arturo> [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install (T302482)

aborrero changed the task status from Open to In Progress.Feb 24 2022, 10:37 AM
aborrero triaged this task as High priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 765493 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] aptrepo: introduce component bullseye-wikimedia/thirdparty/openstack-db

https://gerrit.wikimedia.org/r/765493

Change 765493 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] aptrepo: introduce component bullseye-wikimedia/thirdparty/openstack-db

https://gerrit.wikimedia.org/r/765493

Mentioned in SAL (#wikimedia-operations) [2022-02-24T12:04:41Z] <arturo> aborrero@apt1001:~$ sudo -i reprepro --component thirdparty/openstack-db update bullseye-wikimedia (T302482)

Change 765536 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] galera: install packages from our custom component

https://gerrit.wikimedia.org/r/765536

Change 765536 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] galera: install packages from our custom component

https://gerrit.wikimedia.org/r/765536

Change 765540 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] galera: fix typo in priority

https://gerrit.wikimedia.org/r/765540

Change 765540 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] galera: type fixes

https://gerrit.wikimedia.org/r/765540

This should be ready to go:

aborrero@cumin1001:~$ sudo cumin cloudcontrol*.wikimedia.org 'apt-cache policy mariadb-server galera-4'
6 hosts will be targeted:
cloudcontrol[2001,2003-2004]-dev.wikimedia.org,cloudcontrol[1003-1005].wikimedia.org
Ok to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit 6
===== NODE GROUP =====                                                                                                                                                                                             
(2) cloudcontrol[1004-1005].wikimedia.org                                                                                                                                                                          
----- OUTPUT of 'apt-cache policy...-server galera-4' -----                                                                                                                                                        
mariadb-server:                                                                                                                                                                                                    
  Installed: 1:10.5.12-0+deb11u1                                                                                                                                                                                   
  Candidate: 1:10.5.12-0+deb11u1
  Version table:
 *** 1:10.5.12-0+deb11u1 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
galera-4:
  Installed: 26.4.9-0+deb11u1
  Candidate: 26.4.9-0+deb11u1
  Version table:
 *** 26.4.9-0+deb11u1 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
===== NODE GROUP =====                                                                                                                                                                                             
(4) cloudcontrol[2001,2003-2004]-dev.wikimedia.org,cloudcontrol1003.wikimedia.org                                                                                                                                  
----- OUTPUT of 'apt-cache policy...-server galera-4' -----                                                                                                                                                        
mariadb-server:                                                                                                                                                                                                    
  Installed: 1:10.5.12-0+deb11u1                                                                                                                                                                                   
  Candidate: 1:10.5.15+maria~bullseye
  Version table:
     1:10.5.15+maria~bullseye 1002
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/thirdparty/openstack-db amd64 Packages
 *** 1:10.5.12-0+deb11u1 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
galera-4:
  Installed: 26.4.9-0+deb11u1
  Candidate: 26.4.11-bullseye
  Version table:
     26.4.11-bullseye 1002
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/thirdparty/openstack-db amd64 Packages
 *** 26.4.9-0+deb11u1 500
        500 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
================           

Please @Andrew reopen if you think we need other versions. They are available in the vendor repos (for example, mariadb 10.6)

Change 955841 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Galera: allow installing debian-hosted packages for Bookworm or later

https://gerrit.wikimedia.org/r/955841

Change 955841 merged by FNegri:

[operations/puppet@production] Galera: allow installing debian-hosted packages for Bookworm or later

https://gerrit.wikimedia.org/r/955841