es2045 had to be reimaged and had its /srv formatted.
It needs to be recloned from es2046.
This means es5 in codfw will be serving with just the master for some hours.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| es2046: Disable notifications | Marostegui | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Marostegui | T422365 Migration to Debian Trixie of production database-related hosts | |||
| Resolved | Marostegui | T428572 Migrate es5 section to Debian Trixie | |||
| Resolved | Marostegui | T428993 Reclone es2045 | |||
| Resolved | Jhancock.wm | T429113 es2045 down |
Event Timeline
Comment Actions
Change #1301880 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] es2046: Disable notifications
Comment Actions
Change #1301880 merged by Marostegui:
[operations/puppet@production] es2046: Disable notifications
Comment Actions
This host has been recovered from backups - thanks @jcrespo for the help.
We are right now doing a db-compare to make sure data is consistent with the "master".
Comment Actions
Transfer the data from a live db to another host (from a screen on a cumin host):
cumin2002:~# transfer.py --type=xtrabackup es2044.codfw.wmnet:/run/mysqld/mysqld.sock es2045.codfw.wmnet:/srv/sqldata 2026-06-16 09:07:02 INFO: About to transfer /run/mysqld/mysqld.sock from es2044.codfw.wmnet to ['es2045.codfw.wmnet']:['/srv/sqldata'] (8857795044988 bytes). 2026-06-16 19:09:15 WARNING: Original size on es2044.codfw.wmnet is 8857795044988 bytes but transferred size is 8834269869919 bytes after copy to es2045.codfw.wmnet. 2026-06-16 19:09:17 INFO: Parallel checksum of source on es2044.codfw.wmnet and the transmitted ones on es2045.codfw.wmnet match. 2026-06-16 19:09:17 INFO: 8834269869919 bytes correctly transferred from es2044.codfw.wmnet to es2045.codfw.wmnet 2026-06-16 19:09:19 INFO: Cleaning up.... ✅
Preparing the data after transfer (from a screen in the target host):
es2045:~# xtrabackup --prepare --target-dir /srv/sqldata --use-memory 40G --open-files-limit 200000 xtrabackup based on MariaDB server 10.11.16-MariaDB Linux (x86_64) [00] 2026-06-17 06:39:35 cd to /srv/sqldata/ [00] 2026-06-17 06:39:35 open files limit requested 200000, set to 200000 [00] 2026-06-17 06:39:35 This target seems to be not prepared yet. [00] 2026-06-17 06:39:35 mariabackup: using the following InnoDB configuration for recovery: [00] 2026-06-17 06:39:35 innodb_data_home_dir = . [00] 2026-06-17 06:39:35 innodb_data_file_path = ibdata1:12M:autoextend [00] 2026-06-17 06:39:35 innodb_log_group_home_dir = . [00] 2026-06-17 06:39:35 Starting InnoDB instance for recovery. [00] 2026-06-17 06:39:35 mariabackup: Using 42949672960 bytes for buffer pool (set by --use-memory parameter) 2026-06-17 6:39:35 0 [Note] InnoDB: Compressed tables use zlib 1.3.1 2026-06-17 6:39:35 0 [Note] InnoDB: Number of transaction pools: 1 2026-06-17 6:39:35 0 [Note] InnoDB: Using AVX512 instructions 2026-06-17 6:39:35 0 [Note] InnoDB: Using Linux native AIO 2026-06-17 6:39:35 0 [Note] InnoDB: innodb_buffer_pool_size_max=40960m, innodb_buffer_pool_size=40960m 2026-06-17 6:39:38 0 [Note] InnoDB: Completed initialization of buffer pool 2026-06-17 6:39:38 0 [Note] InnoDB: Buffered log writes (block size=512 bytes) 2026-06-17 6:39:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=20137211362279 2026-06-17 6:39:38 0 [Note] InnoDB: End of log at LSN=20137211366507 2026-06-17 6:39:38 0 [Note] InnoDB: To recover: 10 pages 2026-06-17 6:39:38 0 [Note] InnoDB: Last binlog file './es2023-bin.011816', position 142323651 [00] 2026-06-17 06:39:38 Last binlog file ./es2023-bin.011816, position 142323651 recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 1 0 (0.0 seconds); [00] 2026-06-17 06:39:38 completed OK! ✅
If the snapshot had existed on dbprov, this would be simplified & faster with a single command, as it is pre-prepared and compressed (from a screen on cumin):
# transfer.py --type=decompress dbprov2004.codfw.wmnet:/srv/backups/snapshots/latest/snapshot.es5.2026-10-19--00-01-34.tar.gz es2045.codfw.wmnet:/srv/sqldata
Comment Actions
Data looks good, i've compared the tables on enwiki, wikidata and commons and all looks fine.