Page MenuHomePhabricator
Create Task
Maniphest T251858

hdfs-rsync of mediawiki history dumps fails due to source not present (yet)
Closed, ResolvedPublic
Actions

Description

Due to the delayed new mw history snapshot, the related dumps on labstore nodes are failing since May 1st due to:

May 05 05:00:01 labstore1006 kerberos-run-command[19045]: User dumpsgen executes as user dumpsgen the command ['/bin/bash', '-c', '/usr/local/bin/hdfs-rsync -r -t --delete --exclude "readme.html" --chmod=go-w hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date="$
May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Error: Argument parsing error:
May 05 05:00:07 labstore1006 kerberos-run-command[19045]:         Error validating src list:
May 05 05:00:07 labstore1006 kerberos-run-command[19045]:                 hdfs:///wmf/data/archive/mediawiki/history/2020-04 does not exist
May 05 05:00:07 labstore1006 kerberos-run-command[19045]: Try --help for more information.

Desired solution: Check for folder existence in HDFS before launching hdfs-rsync (see comments below).

Details

SubjectRepoBranchLines +/-
dumps::web::fetches::analytics::job: fix rsync bash scriptoperations/puppetproduction+5 -4
web::fetches::analytics::job: do not rsync mediawiki if missing sourceoperations/puppetproduction+31 -8
Customize query in gerrit

Related Objects

Event Timeline

elukey triaged this task as Medium priority.May 5 2020, 6:09 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2020, 6:09 AM
JAllemandou renamed this task from Mediawiki history dumps fails due to source not present (yet) to hdfs-rsync of mediawiki history dumps fails due to source not present (yet).May 5 2020, 12:54 PM

Should we update the job to look at new month only after a few days? Or shall we add a hdfs call to only launch hdfs-rsync if the source is present?

elukey added a comment.May 5 2020, 1:39 PM
In T251858#6108422, @JAllemandou wrote:

Should we update the job to look at new month only after a few days? Or shall we add a hdfs call to only launch hdfs-rsync if the source is present?

I would prefer to launch the rsync only if there is data, so if we are delayed like this month no alarm with fire.

JAllemandou added a comment.May 5 2020, 1:46 PM

Makes sense @elukey - this also means that we'll never get an error if no data shows up. I assume it's ok, as we have SLA emails on the production side.

elukey added a comment.May 5 2020, 1:47 PM
In T251858#6108699, @JAllemandou wrote:

Makes sense @elukey - this also means that we'll never get an error if no data shows up. I assume it's ok, as we have SLA emails on the production side.

Yes I think that if we want to alarm about data not showing up we need a different alarm (like the SLA etc..)

JAllemandou added a comment.May 5 2020, 1:52 PM

Well here we are, decision is made :)Task description will be updated with the desired solution.

JAllemandou updated the task description. (Show Details)May 5 2020, 1:53 PM
mforns claimed this task.May 6 2020, 3:47 PM
mforns added a project: Analytics-Kanban.
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot added a comment.May 6 2020, 6:16 PM

Change 594773 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] web::fetches::analytics::job: do not rsync mediawiki if missing source

https://gerrit.wikimedia.org/r/594773

gerritbot added a project: Patch-For-Review.May 6 2020, 6:16 PM
mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.May 6 2020, 6:29 PM
Ottomata moved this task from Incoming to Ops Week on the Analytics board.May 11 2020, 3:55 PM
gerritbot added a comment.Jun 1 2020, 9:28 AM

Change 594773 merged by Elukey:
[operations/puppet@production] web::fetches::analytics::job: do not rsync mediawiki if missing source

https://gerrit.wikimedia.org/r/594773

gerritbot added a comment.Jun 1 2020, 9:46 AM

Change 601316 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] dumos::web::fetches::analytics::job: fix rsync bash script

https://gerrit.wikimedia.org/r/601316

gerritbot added a comment.Jun 1 2020, 9:49 AM

Change 601316 merged by Elukey:
[operations/puppet@production] dumps::web::fetches::analytics::job: fix rsync bash script

https://gerrit.wikimedia.org/r/601316

elukey added a comment.Jun 1 2020, 9:52 AM

Seems to work!

Jun 01 09:51:22 labstore1006 kerberos-run-command[12033]: Ignoring missing hdfs source hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date="$(/bin/date +%Y-%m-15) -1 month" +"%Y-%m"),$(/bin/date --date="$(/bin/date +%Y-%m-15) -2 month" +"%Y-%m")}
Jun 01 09:51:22 labstore1006 systemd[1]: analytics-dumps-fetch-mediawiki_history_dumps.service: Succeeded.

Let's keep this open for a few days to see if when the time comes the rsync does its job correctly.

elukey merged a task: T254124: labstore1006/1007: issue copying mediawiki_history_dumps files from Hadoop HDFS.Jun 1 2020, 9:52 AM
elukey added subscribers: aborrero, bd808, Andrew and 2 others.
Maintenance_bot removed a project: Patch-For-Review.Jun 1 2020, 10:11 AM
mforns moved this task from In Code Review to Done on the Analytics-Kanban board.Jun 1 2020, 6:50 PM
• fdans closed this task as Resolved.Jun 15 2020, 4:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL