Page MenuHomePhabricator

Re-enable `set -e` in Wikibase entity dump scripts
Closed, ResolvedPublic

Description

We have repeatedly been slow to notice that the Wikidata entity dumps weren't generated correctly (see e.g. T403882). This is partially due to the fact that the dump generation scripts don't set -e, i.e. fail when one of the commands they execute fails. This in turn causes the airflow DAGs to report successful runs even though there were errors and the dump files aren't available.

This change was made during the migration to airflow (see T400383#11038714), but various problems have been fixed since then and they have been working fine in recent weeks. They should be robust enough for us to re-enable set -e so that we'll know if the dump generation fails.

Event Timeline

Change #1207109 had a related patch set uploaded (by Itamar Givon; author: Itamar Givon):

[operations/dumps@master] Replace 'let' with arithmetic expansion

https://gerrit.wikimedia.org/r/1207109

Change #1207110 had a related patch set uploaded (by Itamar Givon; author: Itamar Givon):

[operations/dumps@master] Clean up existing symlink before creating a new one

https://gerrit.wikimedia.org/r/1207110

Change #1207111 had a related patch set uploaded (by Itamar Givon; author: Itamar Givon):

[operations/dumps@master] Restore strict error handling

https://gerrit.wikimedia.org/r/1207111

Change #1207109 merged by jenkins-bot:

[operations/dumps@master] Replace 'let' with arithmetic expansion

https://gerrit.wikimedia.org/r/1207109

Change #1207110 merged by jenkins-bot:

[operations/dumps@master] Clean up existing symlink before creating a new one

https://gerrit.wikimedia.org/r/1207110

Change #1207111 merged by jenkins-bot:

[operations/dumps@master] Restore strict error handling

https://gerrit.wikimedia.org/r/1207111

Change #1223213 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dumps@master] Remove the legacy cleanup code from dumpwikibasejson.sh

https://gerrit.wikimedia.org/r/1223213

Change #1223213 merged by Btullis:

[operations/dumps@master] Remove the legacy cleanup code from dumpwikibasejson.sh

https://gerrit.wikimedia.org/r/1223213