Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	QChris
	Oct 14 2014, 11:12 AM

Project Tags

Referenced Files

None

Subscribers

Description

Between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx several
partitions, were not marked successful [1]. It seems bits was most
affected, followed by upload and to a lesser extent text and mobile.

What happened?

[1]

qchris@stat1002 jobs: 0 time: 11:07:47 // exit code: 0
cwd: ~
cluster-scripts/dump_webrequest_status.sh

+---------------------+--------+--------+--------+--------+
| Date                |  bits  |  text  | mobile | upload |
+---------------------+--------+--------+--------+--------+

[...]

| 2014-10-13T11:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-13T12:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-13T13:xx:xx |    X   |    X   |    X   |    X   |    
| 2014-10-13T14:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-13T15:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-13T16:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-13T17:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-13T18:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-10-13T19:xx:xx |    X   |    .   |    .   |    X   |    
| 2014-10-13T20:xx:xx |    X   |    .   |    .   |    X   |    
| 2014-10-13T21:xx:xx |    X   |    .   |    .   |    X   |    
| 2014-10-13T22:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-10-13T23:xx:xx |    .   |    .   |    .   |    .   |

[...]

+---------------------+--------+--------+--------+--------+

Statuses:

. --> Partition is ok
X --> Partition is not ok (duplicates, missing, or nulls)

pass cluster-scripts/dump_webrequest_status.sh

Version: unspecified
Severity: normal

Details

Reference: bz72028

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T72085 Raw webrequest partitions that were not marked successful
Resolved	Ottomata	T72087 Kafka partition leader elections causing a drop of a few log lines
Resolved	Ottomata	T71667 Kafka broker analytics1021 not receiving messages every now and then
Declined	None	T78081 Raw upload webrequest partitions for 2014-12-08T17/1H not marked successful
Resolved	Ottomata	T74809 Raw webrequest partitions that were not marked successful due to only esams caches causing unknown problems
Declined	None	T74028 Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:52 AM

• bzimport added a project: Analytics-Clusters.

• bzimport set Reference to bz72028.

• bzimport added a subscriber: Unknown Object (MLST).

QChris created this task.Oct 14 2014, 11:12 AM

For 2014-10-13T13:xx:xx it affected all caches with the only exception
of

cp1056.eqiad.wmnet (bits)
cp1057.eqiad.wmnet (bits)
cp3019.esams.wikimedia.org (bits)
cp3020.esams.wikimedia.org (bits)

(which are exactly the machines that saw the ACK experiments [1],
and we did not see missing log lines for any of them.)

For that hour, we saw no duplicates, but intermittent loss between
2014-10-13T13:37:15 and 2014-10-13T13:38:16 which is worth

bits <1 second
text <2 seconds
mobile <2 seconds
upload <1 second

This nicely matches the dropout of analytics1021 from its partition leader role [2].

I marked the 2014-10-13T13:xx:xx partitions as ok.

[1] https://git.wikimedia.org/blob/operations%2Fpuppet.git/ccc17ce0780f6c56ddcac4f4dcd9f90b2dc0d346/manifests%2Frole%2Fcache.pp#L510
[2] https://bugzilla.wikimedia.org/show_bug.cgi?id=69667#c14

The failed partitions between 2014-10-13T15:xx:xx--2014-10-13T21:xx:xx
have all exclusively been esams caches.
Hence, filing under the esams bug.

(Since it also is about analytics1021 dropping out of it's leader role,
also blocking on bug 69667)

Ironholds unsubscribed.Dec 5 2014, 7:39 PM

• kevinator triaged this task as Medium priority.Dec 5 2014, 11:10 PM

• kevinator set Security to None.

Ottomata closed this task as Declined.Mar 31 2015, 2:12 PM

Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xxClosed, DeclinedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx
Closed, DeclinedPublic
Actions

Related Objects
Search...