Page MenuHomePhabricator
Create Task
Maniphest T125079

Problem with hadoop data ingestion impacting data delivery [8 pts]
Closed, ResolvedPublic
Actions

Description

Currently, we consume the pagecounts raw files that you generate every hour as an input of our Wikipedia infrastructure.

http://dumps.wikimedia.org/other/pagecounts-raw/2016/2016-01/

We have noticed that there are no new files from today 28th January, and because of that we are having issues in our side.

I would like to know if this issue is already noticed in your side and if there is a current action to fix the generation of this files.

Event Timeline

DianaArq created this task.Jan 28 2016, 2:06 PM
DianaArq raised the priority of this task from to Needs Triage.
DianaArq updated the task description. (Show Details)
DianaArq subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 28 2016, 2:06 PM
ArielGlenn added a project: Analytics.Jan 28 2016, 2:16 PM
ArielGlenn set Security to None.
ArielGlenn subscribed.

I checked on stat1002:/mnt/hdfs/wmf/data/archive/pagecounts-raw/2016/2016-01 and the files aren't there.

Aklapper triaged this task as High priority.Jan 28 2016, 2:34 PM
JAllemandou subscribed.Jan 28 2016, 2:41 PM

Hello,
Issue is know, an email has been sent to the analytics list about the problem.
Hadoop data ingestion has been failing yesterday, and was restored around 13:00UTC today , but all the jobs down the pipeline are late.
Data will show up, but it'll take some time.

Milimetric renamed this task from pagecounts raw from 28/01 are not present to pagecounts raw from 28/01 are not present [3 pts].Jan 28 2016, 6:09 PM
Milimetric assigned this task to JAllemandou.
Milimetric edited projects, added: Analytics-Kanban; removed: Analytics.
Milimetric moved this task from Next Up to In Code Review on the Analytics-Kanban board.
Milimetric moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.
JAllemandou renamed this task from pagecounts raw from 28/01 are not present [3 pts] to Problem with hadoop data ingestion impacting data delivery [8 pts].Jan 30 2016, 11:14 AM
JAllemandou moved this task from Ready to Deploy to In Progress on the Analytics-Kanban board.
JAllemandou moved this task from In Progress to Done on the Analytics-Kanban board.Jan 31 2016, 9:51 AM
• Nuria closed this task as Resolved.Feb 2 2016, 4:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits