Page MenuHomePhabricator

[ERROR] Can’t open and lock privilege tables: Table ‘mysql.servers’ doesn’t exist in engine
Closed, InvalidPublic

Description

After the reboot on October 22, sites of ceres-01 (see prototype) and mars (see survey) on wikisp project are not reachable.

Status from mars:

galahad@mars:~$ sudo systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-10-27 04:10:57 UTC; 26s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 144345 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 144346 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 144348 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl >
    Process: 144416 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 144416 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 313ms

Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] InnoDB: Loading buffer pool(s) from /srv/sqldata/ib_buffer_pool
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: '>
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] InnoDB: Buffer pool(s) load completed at 221027  4:10:57
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [Note] Server socket created on IP: '::'.
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
Oct 27 04:10:57 mars mariadbd[144416]: 2022-10-27  4:10:57 0 [ERROR] Aborting
Oct 27 04:10:57 mars systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 04:10:57 mars systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 27 04:10:57 mars systemd[1]: Failed to start MariaDB 10.5.15 database server.

Status from ceres-01

galahad@ceres-01:~$ sudo systemctl status mariadb
● mariadb.service - MariaDB 10.5.15 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-10-24 15:54:36 UTC; 2 days ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 489 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 522 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 535 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set>
    Process: 610 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 610 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 221ms

Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [Note] Plugin 'FEEDBACK' is disabled.
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not >
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: >
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [Note] Server socket created on IP: '::'.
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
Oct 24 15:54:36 ceres-01 mariadbd[610]: 2022-10-24 15:54:36 0 [ERROR] Aborting
Oct 24 15:54:36 ceres-01 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 15:54:36 ceres-01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 24 15:54:36 ceres-01 systemd[1]: Failed to start MariaDB 10.5.15 database server.

I'm trying to connect mariadb but..

galahad@ceres-01:~$ sudo mariadb
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)

Is there any way of recovery?

Thanks in advance.

Event Timeline

Upd: After I rebuild ceres-01 remains the same. It's probably something from the puppet.

And yes, it is something from puppet simplelamp.

No hubo solución aparente, tuve que hacer rebuild.

If you want us to debug this issue then you should not close the ticket as invalid.

If there are no files in the data_directory (/srv/sqldata) then that has no relation to how mariadb package was installed.

Are you not using /srv/sqldata when you manually install things?