Page MenuHomePhabricator

Raw webrequest partitions for 2014-10-07T1[789]:xx:xx not marked successful
Closed, DeclinedPublic

Description

The three bits partitions [1] on 2014-10-07T1[789]:xx:xx, were not marked
successful.

What happened?

[1]


qchris@stat1002 jobs: 0 time: 13:05:11 // exit code: 0
cwd: ~/cluster-scripts
./dump_webrequest_status.sh

+---------------------+--------+--------+--------+--------+
| Date                |  bits  |  text  | mobile | upload |
+---------------------+--------+--------+--------+--------+

[...]

| 2014-10-07T15:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-07T16:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-07T17:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-07T18:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-07T19:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-07T20:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-07T21:xx:xx |    .   |    .   |    .   |    .   |

[...]

+---------------------+--------+--------+--------+--------+

Statuses:

. --> Partition is ok
X --> Partition is not ok (duplicates, missing, or nulls)

Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=71435

Details

Reference
bz71882

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:43 AM
bzimport set Reference to bz71882.
bzimport added a subscriber: Unknown Object (MLST).

Since checking missings across hour boundaries does not make the
missing go away [1], it does not seem to be the neighbored race
condition described in bug 69615.

Since it seems only esams bits are affected, it might be another
instance of bug 71435.

More investigation needed.

[1]


qchris@stat1002 jobs: 0 time: 13:17:03 // exit code: 0
cwd: ~/refinery
hive -f three_hour_stats.hql -d table=wmf_raw.webrequest -d webrequest_source=bits -d year=2014 -d month=10 -d day=7 -d hourA=17 -d hourB=18 -d hourC=19
[...]
Total MapReduce CPU Time Spent: 0 days 4 hours 9 minutes 5 seconds 150 msec
OK
hostname sequence_min sequence_max count_actual count_expected count_different count_duplicate count_null_sequence percent_different
cp3022.esams.wikimedia.org 2082145621 2135356589 52052666 53210969 1158303 0 0 -2.1768124538382305
cp3021.esams.wikimedia.org 2078380896 2131585483 52714377 53204588 490211 0 0 -0.9213697886355214
cp3020.esams.wikimedia.org 1828915695 1882208121 50611125 53292427 2681302 0 0 -5.031300225827583
cp3019.esams.wikimedia.org 1830855032 1884131067 51714434 53276036 1561602 0 0 -2.931152760689628
Time taken: 473.312 seconds, Fetched: 4 row(s)

kevinator set Security to None.