Page MenuHomePhabricator
Create Task
Maniphest T363927

100's of dump failures due to etcd issue
Open, HighPublic
Actions

Description

Original alert to ops-dumps: https://groups.google.com/a/wikimedia.org/g/ops-dumps/c/efjbIJHS--Q

Example stack:

*** Wiki: aawikibooks
=====================
[20240501085807]: Notice: Undefined index: es7 in /srv/mediawiki/wmf-config/etcd.php on line 116
Warning: Invalid argument supplied for foreach() in /srv/mediawiki/wmf-config/etcd.php on line 116
Notice: Undefined index: es7 in /srv/mediawiki/wmf-config/etcd.php on line 116
Warning: Invalid argument supplied for foreach() in /srv/mediawiki/wmf-config/etcd.php on line 116
Notice: Undefined index: es7 in /srv/mediawiki/wmf-config/etcd.php on line 116
Warning: Invalid argument supplied for foreach() in /srv/mediawiki/wmf-config/etcd.php on line 116

Related Objects

Event Timeline

xcollazo created this task.Wed, May 1, 5:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, May 1, 5:48 PM
xcollazo triaged this task as High priority.Wed, May 1, 5:49 PM
xcollazo added projects: Dumps-Generation, Data Products (Data Products Sprint 13).
xcollazo added a comment.Wed, May 1, 5:53 PM

This may just be a cascading effect of the etcd work done on T358636.

Will investigate if the jobs are recovering.

xcollazo moved this task from Backlog to Active on the Dumps-Generation board.Wed, May 1, 6:04 PM
xcollazo added a comment.Wed, May 1, 6:08 PM

A cursory look on snapshot1012, which runs enwiki, and on snapshot1008, which run all the small wikis shows no issues, so looks like dumps recovered.

State at https://dumps.wikimedia.org/backup-index.html shows progress for all dumps.

Will continue monitoring.

VirginiaPoundstone moved this task from Sprint Backlog to In Process on the Data Products (Data Products Sprint 13) board.Wed, May 1, 10:19 PM
VirginiaPoundstone moved this task from In Process to Sprint Backlog on the Data Products (Data Products Sprint 13) board.
xcollazo added a comment.Thu, May 2, 4:46 PM

Dumps continue running and continue making progress. So definitely a transient issue. Closing.

xcollazo moved this task from Sprint Backlog to Done on the Data Products (Data Products Sprint 13) board.Thu, May 2, 4:46 PM
xcollazo moved this task from Active to Done on the Dumps-Generation board.Fri, May 10, 7:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL