Page MenuHomePhabricator

[ERROR] Can’t open and lock privilege tables: Table ‘mysql.servers’ doesn’t exist in engine
Closed, InvalidPublic

Description

After the reboot on October 22, sites of ceres-01 (see prototype) and mars (see survey) on wikisp project are not reachable.

Status from mars:

galahad@mars:~$ sudo systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-10-27 04:10:57 UTC; 26s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 144345 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 144346 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 144348 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl >
    Process: 144416 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 144416 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 313ms

Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] InnoDB: Loading buffer pool(s) from /srv/sqldata/ib_buffer_pool
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: '>
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] InnoDB: Buffer pool(s) load completed at 221027  4:10:57
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] Server socket created on IP: '::'.
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Aborting
Oct 27 04:10:57 mars systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 04:10:57 mars systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 27 04:10:57 mars systemd[1]: Failed to start MariaDB 10.5.15 database server.

Status from ceres-01

galahad@ceres-01:~$ sudo systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-10-24 15:54:36 UTC; 2 days ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 489 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 522 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 535 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set>
    Process: 610 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 610 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 221ms

Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [Note] Plugin 'FEEDBACK' is disabled.
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not >
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: >
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [Note] Server socket created on IP: '::'.
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Aborting
Oct 24 15:54:36 ceres-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 15:54:36 ceres-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 24 15:54:36 ceres-01 systemd[1]: Failed to start MariaDB 10.5.15 database server.

I'm trying to connect mariadb but..

galahad@ceres-01:~$ sudo mariadb
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)

Is there any way of recovery?

Thanks in advance.

Event Timeline

Upd: After I rebuild ceres-01 remains the same. It's probably something from the puppet.

And yes, it is something from puppet simplelamp.

No hubo solución aparente, tuve que hacer rebuild.