Page MenuHomePhabricator

Optimize commonswiki image table
Closed, ResolvedPublic

Description

Follow up to T275268: Address "image" table capacity problems by storing pdf/djvu text outside file metadata

Status:

  • eqiad:
    • db1121
    • db1141
    • db1142
    • db1143
    • db1144
    • db1145
    • db1146
    • db1147
    • db1148
    • db1149
    • db1150
    • db1160
    • dbstore1007
  • codfw

Related Objects

Mentioned Here
P18541 dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)'
P18533 dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)'
P18532 dbctl commit (dc=all): 'Depooling db2124 (T296143)'
P18531 dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)'
P18523 dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)'
P18521 dbctl commit (dc=all): 'Depooling db2117 (T296143)'
P18520 dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)'
P18511 dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)'
P18510 dbctl commit (dc=all): 'Depooling db2114 (T296143)'
P17876 dbctl commit (dc=all): 'After maintenance db1160 (T296143)'
P17875 dbctl commit (dc=all): 'After maintenance db1160 (T296143)'
P17874 dbctl commit (dc=all): 'After maintenance db1160 (T296143)'
P17873 dbctl commit (dc=all): 'After maintenance db1160 (T296143)'
P17872 dbctl commit (dc=all): 'Depooling db1160 (T296143)'
P17871 dbctl commit (dc=all): 'After maintenance db1149 (T296143)'
P17870 dbctl commit (dc=all): 'After maintenance db1149 (T296143)'
P17869 dbctl commit (dc=all): 'After maintenance db1149 (T296143)'
P17868 dbctl commit (dc=all): 'After maintenance db1149 (T296143)'
P17867 dbctl commit (dc=all): 'Depooling db1149 (T296143)'
P17866 dbctl commit (dc=all): 'After maintenance db1148 (T296143)'
P17864 dbctl commit (dc=all): 'After maintenance db1148 (T296143)'
P17863 dbctl commit (dc=all): 'After maintenance db1148 (T296143)'
P17861 dbctl commit (dc=all): 'After maintenance db1148 (T296143)'
P17853 dbctl commit (dc=all): 'Depooling db1148 (T296143)'
P17852 dbctl commit (dc=all): 'After maintenance db1147 (T296143)'
P17851 dbctl commit (dc=all): 'After maintenance db1147 (T296143)'
P17850 dbctl commit (dc=all): 'After maintenance db1147 (T296143)'
P17849 dbctl commit (dc=all): 'After maintenance db1147 (T296143)'
P17844 dbctl commit (dc=all): 'Depooling db1147 (T296143)'
P17843 dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)'
P17842 dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)'
P17841 dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)'
P17840 dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)'
P17837 dbctl commit (dc=all): 'Depooling db1146:3314 (T296143)'
P17834 dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)'
P17833 dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)'
P17832 dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)'
P17831 dbctl commit (dc=all): 'After maintenance db1144:3314 (T296143)'
P17830 dbctl commit (dc=all): 'Depooling db1144:3314 (T296143)'
P17829 dbctl commit (dc=all): 'After maintenance db1143 (T296143)'
P17828 dbctl commit (dc=all): 'After maintenance db1143 (T296143)'
P17827 dbctl commit (dc=all): 'After maintenance db1143 (T296143)'
P17826 dbctl commit (dc=all): 'After maintenance db1143 (T296143)'
P17821 dbctl commit (dc=all): 'Depooling db1143 (T296143)'
P17820 dbctl commit (dc=all): 'After maintenance db1142 (T296143)'
P17819 dbctl commit (dc=all): 'After maintenance db1142 (T296143)'
P17817 dbctl commit (dc=all): 'After maintenance db1142 (T296143)'
P17815 dbctl commit (dc=all): 'After maintenance db1142 (T296143)'
P17813 dbctl commit (dc=all): 'Depooling db1142 (T296143)'
P17812 dbctl commit (dc=all): 'After maintenance db1141 (T296143)'
P17811 dbctl commit (dc=all): 'After maintenance db1141 (T296143)'
P17810 dbctl commit (dc=all): 'After maintenance db1141 (T296143)'
P17809 dbctl commit (dc=all): 'After maintenance db1141 (T296143)'
P17808 dbctl commit (dc=all): 'Depooling db1141 (T296143)'
P17807 dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After optimize table (T296143)'
P17806 dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After optimize table (T296143)'
P17805 dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: After optimize table (T296143)'
P17804 dbctl commit (dc=all): 'db1121 (re)pooling @ 10%: After optimize table (T296143)'
P17800 dbctl commit (dc=all): 'Depooling db1121 (T296143)'
T275268: Address "image" table capacity problems by storing pdf/djvu text outside file metadata

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:23:46Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1145.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:39:56Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:40:00Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:41:35Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:41:38Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:42:21Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:42:24Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:44:13Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:44:15Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:46:02Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:46:05Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:29Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:31Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1146.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T08:48:35Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17837 and previous config saved to /var/cache/conftool/dbconfig/20211125-084834-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:19:21Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17840 and previous config saved to /var/cache/conftool/dbconfig/20211125-101921-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:34:26Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17841 and previous config saved to /var/cache/conftool/dbconfig/20211125-103425-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:49:31Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17842 and previous config saved to /var/cache/conftool/dbconfig/20211125-104930-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:36Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1146:3314 (T296143)', diff saved to https://phabricator.wikimedia.org/P17843 and previous config saved to /var/cache/conftool/dbconfig/20211125-110435-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:40Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:43Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1147.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T11:04:46Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17844 and previous config saved to /var/cache/conftool/dbconfig/20211125-110443-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:43:45Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17849 and previous config saved to /var/cache/conftool/dbconfig/20211125-144344-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:58:50Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17850 and previous config saved to /var/cache/conftool/dbconfig/20211125-145849-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:13:54Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17851 and previous config saved to /var/cache/conftool/dbconfig/20211125-151354-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:28:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1147 (T296143)', diff saved to https://phabricator.wikimedia.org/P17852 and previous config saved to /var/cache/conftool/dbconfig/20211125-152858-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:03Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1148.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:06Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1148.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T15:29:09Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17853 and previous config saved to /var/cache/conftool/dbconfig/20211125-152906-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T16:41:53Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17861 and previous config saved to /var/cache/conftool/dbconfig/20211125-164153-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T16:56:58Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17863 and previous config saved to /var/cache/conftool/dbconfig/20211125-165657-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:12:03Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17864 and previous config saved to /var/cache/conftool/dbconfig/20211125-171202-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:07Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1148 (T296143)', diff saved to https://phabricator.wikimedia.org/P17866 and previous config saved to /var/cache/conftool/dbconfig/20211125-172707-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:13Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:16Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1149.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T17:27:19Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17867 and previous config saved to /var/cache/conftool/dbconfig/20211125-172714-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T18:43:36Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17868 and previous config saved to /var/cache/conftool/dbconfig/20211125-184336-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T18:58:41Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17869 and previous config saved to /var/cache/conftool/dbconfig/20211125-185841-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:13:46Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17870 and previous config saved to /var/cache/conftool/dbconfig/20211125-191345-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:51Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1149 (T296143)', diff saved to https://phabricator.wikimedia.org/P17871 and previous config saved to /var/cache/conftool/dbconfig/20211125-192850-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:55Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T19:28:58Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:51Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:56Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1160.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-25T20:43:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-26T07:43:20Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17873 and previous config saved to /var/cache/conftool/dbconfig/20211126-074320-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-26T07:58:25Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17874 and previous config saved to /var/cache/conftool/dbconfig/20211126-075824-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-26T08:13:29Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17875 and previous config saved to /var/cache/conftool/dbconfig/20211126-081329-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-26T08:28:34Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'After maintenance db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17876 and previous config saved to /var/cache/conftool/dbconfig/20211126-082834-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2021-11-26T10:04:51Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143

Mentioned in SAL (#wikimedia-operations) [2021-11-26T10:04:56Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance T296143

Ladsgroup updated the task description. (Show Details)
Ladsgroup moved this task from In progress to Done on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2022-01-11T06:51:18Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18510 and previous config saved to /var/cache/conftool/dbconfig/20220111-065118-ladsgroup.json

wrong ticket :facepalm: The script is gonna make some noise. Sorry.

Mentioned in SAL (#wikimedia-operations) [2022-01-11T06:56:40Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18511 and previous config saved to /var/cache/conftool/dbconfig/20220111-065640-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:41:54Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T296143)', diff saved to https://phabricator.wikimedia.org/P18520 and previous config saved to /var/cache/conftool/dbconfig/20220111-074154-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:42:02Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18521 and previous config saved to /var/cache/conftool/dbconfig/20220111-074202-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T07:48:00Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18523 and previous config saved to /var/cache/conftool/dbconfig/20220111-074800-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:33:14Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T296143)', diff saved to https://phabricator.wikimedia.org/P18531 and previous config saved to /var/cache/conftool/dbconfig/20220111-083314-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:33:22Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18532 and previous config saved to /var/cache/conftool/dbconfig/20220111-083322-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T08:41:52Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18533 and previous config saved to /var/cache/conftool/dbconfig/20220111-084151-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2022-01-11T09:27:06Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T296143)', diff saved to https://phabricator.wikimedia.org/P18541 and previous config saved to /var/cache/conftool/dbconfig/20220111-092706-ladsgroup.json