Follow up to T275268: Address "image" table capacity problems by storing pdf/djvu text outside file metadata
Status:
- eqiad:
- db1121
- db1141
- db1142
- db1143
- db1144
- db1145
- db1146
- db1147
- db1148
- db1149
- db1150
- db1160
- dbstore1007
- codfw
Follow up to T275268: Address "image" table capacity problems by storing pdf/djvu text outside file metadata
Status:
Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:23:46Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1145.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:39:56Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:40:00Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:41:35Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:41:38Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:42:21Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:42:24Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:44:13Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:44:15Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:46:02Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:46:05Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:29Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:31Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:35Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17837 and previous config saved to /var/cache/conftool/dbconfig/20211125-084834-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:19:21Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17840 and previous config saved to /var/cache/conftool/dbconfig/20211125-101921-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:34:26Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17841 and previous config saved to /var/cache/conftool/dbconfig/20211125-103425-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:49:31Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17842 and previous config saved to /var/cache/conftool/dbconfig/20211125-104930-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:36Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17843 and previous config saved to /var/cache/conftool/dbconfig/20211125-110435-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:40Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:43Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:46Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17844 and previous config saved to /var/cache/conftool/dbconfig/20211125-110443-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:43:45Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17849 and previous config saved to /var/cache/conftool/dbconfig/20211125-144344-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:58:50Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17850 and previous config saved to /var/cache/conftool/dbconfig/20211125-145849-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:13:54Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17851 and previous config saved to /var/cache/conftool/dbconfig/20211125-151354-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:28:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17852 and previous config saved to /var/cache/conftool/dbconfig/20211125-152858-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:03Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1148.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:06Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1148.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:09Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17853 and previous config saved to /var/cache/conftool/dbconfig/20211125-152906-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T16:41:53Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17861 and previous config saved to /var/cache/conftool/dbconfig/20211125-164153-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T16:56:58Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17863 and previous config saved to /var/cache/conftool/dbconfig/20211125-165657-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:12:03Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17864 and previous config saved to /var/cache/conftool/dbconfig/20211125-171202-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:07Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17866 and previous config saved to /var/cache/conftool/dbconfig/20211125-172707-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:13Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:16Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:19Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17867 and previous config saved to /var/cache/conftool/dbconfig/20211125-172714-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T18:43:36Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17868 and previous config saved to /var/cache/conftool/dbconfig/20211125-184336-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T18:58:41Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17869 and previous config saved to /var/cache/conftool/dbconfig/20211125-185841-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:13:46Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17870 and previous config saved to /var/cache/conftool/dbconfig/20211125-191345-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:51Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17871 and previous config saved to /var/cache/conftool/dbconfig/20211125-192850-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:55Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:58Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:51Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:56Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-26T07:43:20Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17873 and previous config saved to /var/cache/conftool/dbconfig/20211126-074320-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-26T07:58:25Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17874 and previous config saved to /var/cache/conftool/dbconfig/20211126-075824-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-26T08:13:29Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17875 and previous config saved to /var/cache/conftool/dbconfig/20211126-081329-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-26T08:28:34Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17876 and previous config saved to /var/cache/conftool/dbconfig/20211126-082834-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2021-11-26T10:04:51Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2021-11-26T10:04:56Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143
Mentioned in SAL (#wikimedia-operations) [2022-01-11T06:51:18Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18510 and previous config saved to /var/cache/conftool/dbconfig/20220111-065118-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T06:56:40Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18511 and previous config saved to /var/cache/conftool/dbconfig/20220111-065640-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:41:54Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18520 and previous config saved to /var/cache/conftool/dbconfig/20220111-074154-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:42:02Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18521 and previous config saved to /var/cache/conftool/dbconfig/20220111-074202-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:48:00Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18523 and previous config saved to /var/cache/conftool/dbconfig/20220111-074800-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:33:14Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18531 and previous config saved to /var/cache/conftool/dbconfig/20220111-083314-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:33:22Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18532 and previous config saved to /var/cache/conftool/dbconfig/20220111-083322-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:41:52Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18533 and previous config saved to /var/cache/conftool/dbconfig/20220111-084151-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-11T09:27:06Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18541 and previous config saved to /var/cache/conftool/dbconfig/20220111-092706-ladsgroup.json