Page MenuHomePhabricator
Create Task
Maniphest T337088

Druid Webrequest sampled 128 has missing data data for 1 hour
Closed, ResolvedPublic
Actions

Description

I've noticed that tonight the data in druid in the webrequest_sampled_128 dataset has some data problem for the slot between 2023-05-20 00:00:00 and 2023-05-20 01:00:00.

Screenshot 2023-05-20 at 10.43.59.png (1×2 px, 250 KB)

When grouping by webrequest_source It looks like almost all of text data is missing while the upload one seems ok:

image.png (1×2 px, 307 KB)

As a comparison the data in the the live dataset populated by benthos looks totally fine.
Did something break in the pipeline for that hour?
I'm also curious to know if there is any monitoring in place to detect this kind of situations.

Related Objects
Search...

Event Timeline

Volans created this task.May 20 2023, 8:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 20 2023, 8:13 AM
ssingh subscribed.May 24 2023, 1:56 PM
Krinkle subscribed.May 24 2023, 2:09 PM
elukey edited projects, added Data-Engineering; removed Analytics-Radar.May 24 2023, 3:09 PM
Stashbot added a comment.May 25 2023, 8:37 AM

Mentioned in SAL (#wikimedia-analytics) [2023-05-25T08:37:43Z] <joal> rerun druid_load_webrequest_sampled_128_daily 2023-05-20 to reload missing hour (T337088)

JAllemandou subscribed.May 25 2023, 1:33 PM

I have tried rerunning the loading job but this has not solved the issue. More investigation is needed.

JAllemandou mentioned this in T337482: Problem with webrequest data for one hour: 2023-05-20 hour 0 to 1.May 25 2023, 1:41 PM
JAllemandou added a parent task: T337482: Problem with webrequest data for one hour: 2023-05-20 hour 0 to 1.May 25 2023, 1:46 PM
BTullis assigned this task to JAllemandou.May 25 2023, 4:12 PM
BTullis added a project: Data Pipelines (Sprint 14).
BTullis moved this task from Next Up to Incident/Unexpected work on the Data Pipelines (Sprint 14) board.
BTullis moved this task from Incoming (new tickets) to Ops Week on the Data-Engineering board.May 25 2023, 4:14 PM
JAllemandou moved this task from Incident/Unexpected work to Done on the Data Pipelines (Sprint 14) board.May 26 2023, 6:55 AM

Thanks a lot for having raised the issue, this is now solved (see parent ticket).

Volans closed this task as Resolved.May 29 2023, 2:52 PM

@JAllemandou thanks a lot for fixing it, I'm resolving this as you have the other one open for potential follow ups on the monitoring side.

Volans mentioned this in T337460: Increase webrequest_sampled_live Druid datasource's retention.Jun 19 2023, 9:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL