Page MenuHomePhabricator

deployment-eventlog05 puppet error about missing mysql heartbeat.heartbeat table
Closed, ResolvedPublic3 Estimated Story Points

Description

Puppet was disabled. I reenabled it and ran sudo puppet agent -tv to re-run and debug. Result is a lot of errors. I re-ran sudo puppet agent --disable to leave it as it was. I'd appreciate if someone could take a look. Thanks.

Event Timeline

maurelio@deployment-eventlog05:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-eventlog05.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1522404289'
Notice: /Stage[main]/Mariadb::Heartbeat/Exec[pt-heartbeat]/returns: DBD::mysql::db selectrow_arrayref failed: Table 'heartbeat.heartbeat' doesn't exist [for Statement "SHOW CREATE TABLE `heartbeat`.`heartbeat`"] at /usr/local/bin/pt-heartbeat-wikimedia line 4923.
Error: /usr/bin/perl             /usr/local/bin/pt-heartbeat-wikimedia             --defaults-file=/dev/null             --user=root --host=localhost -D heartbeat             --shard=m4 --datacenter=eqiad             --update --replace --interval=1             --set-vars="binlog_format=STATEMENT"             -S /run/mysqld/mysqld.sock --daemonize             --pid /var/run/pt-heartbeat.pid returned 2 instead of one of [0]
Error: /Stage[main]/Mariadb::Heartbeat/Exec[pt-heartbeat]/returns: change from notrun to 0 failed: /usr/bin/perl             /usr/local/bin/pt-heartbeat-wikimedia             --defaults-file=/dev/null             --user=root --host=localhost -D heartbeat             --shard=m4 --datacenter=eqiad             --update --replace --interval=1             --set-vars="binlog_format=STATEMENT"             -S /run/mysqld/mysqld.sock --daemonize             --pid /var/run/pt-heartbeat.pid returned 2 instead of one of [0]
Notice: Applied catalog in 8.87 seconds
EddieGP triaged this task as Medium priority.Apr 13 2018, 11:17 PM
Krenair renamed this task from deployment-eventlog05 puppet errors to deployment-eventlog05 puppet error about missing mysql heartbeat.heartbeat table.May 25 2018, 2:17 AM
Nuria moved this task from Incoming to Operational Excellence on the Analytics board.
Nuria added a project: Analytics-Kanban.
elukey set the point value for this task to 3.Jun 1 2018, 9:01 AM

Thanks for the report! I added the following to the heartbeat database, and puppet now runs fine:

CREATE TABLE heartbeat (
  ts                    varchar(26) NOT NULL,
  server_id             int unsigned NOT NULL PRIMARY KEY,
  file                  varchar(255) DEFAULT NULL,    -- SHOW MASTER STATUS
  position              bigint unsigned DEFAULT NULL, -- SHOW MASTER STATUS
  relay_master_log_file varchar(255) DEFAULT NULL,    -- SHOW SLAVE STATUS 
  exec_master_log_pos   bigint unsigned DEFAULT NULL, -- SHOW SLAVE STATUS
  shard                 varchar(10) DEFAULT 'unknown' -- mediawiki extension
  datacenter            char(5) DEFAULT NULL          -- mediawiki extension
);

(Grabbed from pt-heartbeat-wikimedia's comments)

Change 436748 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging:db: remove unused heartbeat

https://gerrit.wikimedia.org/r/436748

Change 436748 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging:db: remove unused heartbeat

https://gerrit.wikimedia.org/r/436748