Page MenuHomePhabricator
Create Task
Maniphest T318206

SQL/XML dumps multiple errors "Couldn't update index json file, Continuing anyways" from 09 21 2022 near midnight
Closed, ResolvedPublic
Actions

Description

Sample output, for one of the 5 wikis affected:

2022-09-20 23:43:30: fawiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/fawiki/latest ...
2022-09-20 23:43:30: fawiki Checkdir dir /mnt/dumpsdata/xmldatadumps/public/fawiki/latest ...
2022-09-20 23:43:30: fawiki adding rss feed file /mnt/dumpsdata/xmldatadumps/public/fawiki/latest/fawiki-latest-pages-articles.xml.bz2-rss.xml
2022-09-20 23:43:30: fawiki Reading fawiki-20220920-pages-articles.xml.bz2 checksum for md5 from file /mnt/dumpsdata/xmldatadumps/public/fawiki/20220920/md5sums-fawiki-20220920-pages-articles.xml.bz2.txt
2022-09-20 23:43:30: fawiki Reading fawiki-20220920-pages-articles.xml.bz2 checksum for sha1 from file /mnt/dumpsdata/xmldatadumps/public/fawiki/20220920/sha1sums-fawiki-20220920-pages-articles.xml.bz2.txt
Couldn't update index.html file. Continuing anyways
Couldn't update index json file. Continuing anyways
Couldn't update index.html file. Continuing anyways

Details

SubjectRepoBranchLines +/-
start daily cleanup job of sql/xmldumps later in the dayoperations/puppetproduction+1 -1
keep minimum older sql/xml dump files on generation hostsoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

ArielGlenn triaged this task as High priority.Sep 21 2022, 5:32 AM
ArielGlenn created this task.
Restricted Application added a subscriber: Huji. · View Herald TranscriptSep 21 2022, 5:32 AM
gerritbot added a comment.Sep 21 2022, 5:32 AM

Change 833625 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn):

[operations/puppet@production] keep minimum older sql/xml dump files on generation hosts

https://gerrit.wikimedia.org/r/833625

gerritbot added a project: Patch-For-Review.Sep 21 2022, 5:32 AM
gerritbot added a comment.Sep 21 2022, 5:37 AM

Change 833625 merged by ArielGlenn:

[operations/puppet@production] keep minimum older sql/xml dump files on generation hosts

https://gerrit.wikimedia.org/r/833625

ArielGlenn added a comment.Sep 21 2022, 5:46 AM

After some poking around, it turned out that the problem was:

Στιγμιότυπο από 2022-09-21 08-26-27.png (269×831 px, 23 KB)

(dumpsdata1003 graph from https://grafana.wikimedia.org/d/000000377/host-overview?var-server=dumpsdata1003&var-datasource=eqiad%20prometheus%2Fops&orgId=1&from=1663652279988&to=1663735437092

We'll keep one less dupm run of the smaller wikis until the new dumpsdata hosts can be rolled out. Those are waiting on completion of the raid monitoring changes for the new controller, see T315608

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 6:30 AM
gerritbot added a comment.Sep 21 2022, 9:39 AM

Change 833736 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn):

[operations/puppet@production] start daily cleanup job of sql/xmldumps later in the day

https://gerrit.wikimedia.org/r/833736

gerritbot added a project: Patch-For-Review.Sep 21 2022, 9:39 AM
gerritbot added a comment.Sep 21 2022, 9:41 AM

Change 833736 merged by ArielGlenn:

[operations/puppet@production] start daily cleanup job of sql/xmldumps later in the day

https://gerrit.wikimedia.org/r/833736

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 10:30 AM
ArielGlenn closed this task as Resolved.Nov 8 2022, 9:35 AM

No pending actions left, so I'm closing this.

ArielGlenn moved this task from Backlog to Done on the Dumps-Generation board.Jun 22 2023, 4:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL