While looking at T122449, I rebooted integration-slave-trusty-1015 and it seems jobs running on it are failing with:
/bin/mw-install-mysql.sh 00:00:43.846 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T122449 Jenkins job mediawiki-extensions-qunit Karma timeout on odd-numbered build slaves | |||
| Resolved | hashar | T122459 integration-slave-trusty-1015 : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock |
The patch contint: Put mysql db on tmpfs for role::ci::slave::labs https://gerrit.wikimedia.org/r/#/c/204528/ went missing from the CI puppetmaster. I cherry picked it again. And puppet did:
Notice: /Stage[main]/Role::Ci::Slave::Labs/File[/var/lib/mysql]/owner: owner changed 'root' to 'mysql' Notice: /Stage[main]/Role::Ci::Slave::Labs/File[/var/lib/mysql]/group: group changed 'root' to 'mysql' Notice: /Stage[main]/Role::Ci::Slave::Labs/File[/var/lib/mysql]/mode: mode changed '1777' to '0775' ... Notice: /Stage[main]/Role::Ci::Slave::Labs/Exec[create-mysql-datadir]/returns: executed successfully Notice: /Stage[main]/Role::Ci::Slave::Labs/Service[mysql]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Role::Ci::Slave::Labs/Service[mysql]: Unscheduling refresh on Service[mysql]
And it is back:
integration-slave-trusty-1015:~# ps -u mysql f PID TTY STAT TIME COMMAND 11738 ? Ssl 0:00 /usr/sbin/mysqld