dbstore2002 needs to be reimaged - it is a trusty server.
Once it is reimaged, clone data from dbstore2001 or from its snapshot (dbstore2001:/srv/tmp/dbstore2002.tar.gz)
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| mariadb: Install jessie on dbstore2002 | operations/puppet | production | +0 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Cmjohnson | T136860 rack/setup/install/deploy labsdb1009-labsdb1011 | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | jcrespo | T50930 Database replication problems - production and labs (tracking) | |||
| Resolved | jcrespo | T134203 enwiki_p replica on s1 is corrupted | |||
| Resolved | jcrespo | T140788 Labs databases rearchitecture (tracking) | |||
| Resolved | • chasemp | T147052 Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production | |||
| Resolved | Marostegui | T130128 Fix dbstore2001 and dbstore2002 | |||
| Resolved | Marostegui | T146261 Reimage dbstore2001 as jessie | |||
| Resolved | Marostegui | T149457 install new disks into dbstore2001 | |||
| Resolved | Marostegui | T150017 Reimage dbstore2002 |
Event Timeline
Comment Actions
Change 319859 had a related patch set uploaded (by Marostegui):
mariadb: Install jessie on dbstore2002
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2016-11-04T15:35:47Z] <marostegui> reimage dbstore2002 - T150017
Comment Actions
Script wmf_auto_reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:
['dbstore2002.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201611041538_marostegui_738.log.
Comment Actions
Completed auto-reimage of hosts:
['dbstore2002.codfw.wmnet']
and were ALL successful.
Comment Actions
root@dbstore2002:/srv/sqldata# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.6 (jessie) Release: 8.6 Codename: jessie
I am now transferring back the snapshot
Comment Actions
dbstore2002 is now running 10.0.28, mysql_upgrade went fine and tokuDB engine is loaded. The slaves are catching up too.
root@dbstore2002:/opt/wmf-mariadb10/bin# mysql --skip-ssl -e "select version();"
+-----------------+
| version() |
+-----------------+
| 10.0.28-MariaDB |
+-----------------+
root@dbstore2002:/opt/wmf-mariadb10/bin# mysql --skip-ssl -e "show engines" | grep -i toku
TokuDB YES Percona TokuDB Storage Engine with Fractal Tree(tm) Technology YES YES YES
root@dbstore2002:/opt/wmf-mariadb10/bin# mysql --skip-ssl -e "show all slaves status\G" | egrep -i "Connection|Seconds"
Connection_name: s1
Seconds_Behind_Master: 31721
Connection_name: s3
Seconds_Behind_Master: 31537
Connection_name: s4
Seconds_Behind_Master: 31470