eventgate-main latencies very high since the failover to codfw
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Sep 2 2020, 10:07 AM

Description

When we transitioned all services but mediawiki to codfw, we saw the eventgate latencies spike up for MediaWiki, which was kinda-expected given we were going cross-dc:

These are the eventgate latencies seen from servers in eqiad.

Now we expected latencies to go down once we transitioned mediawiki to codfw. Instead not just it didn't go down, but we also saw a very high latency in restbase (which was transitioned together with eventgate to codfw)

This seems like a possible cause of T261763 and is a very serious performance regression.

Related Objects

Mentioned In: T261763: Unexplained increase in save times, possibly associated with DC switchover
Mentioned Here: T261763: Unexplained increase in save times, possibly associated with DC switchover

Event Timeline

Joe created this task.Sep 2 2020, 10:07 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2020, 10:07 AM

Setting priority to UBN! given the seriousness of the perf regression.

Daimona added a subscriber: Daimona.Sep 2 2020, 10:10 AM

It looks like kafka2003 is the culprit - its broker latencies are in the order of 1 seconds.

So this is probably due to all the purges going through the codfw kafka2003 server, and that we still haven't partitioned the purge topic.

In normal conditions, the purges are almost evenly distributed across datanceters, as restbase-async is codfw-only, and sends its purges to the local eventgate, while mediawiki is in eqiad and sends its purges to eqiad's eventgate.

So as a temporary mitigation while we find a good strategy, I propose to do the following:

move restbase-async to eqiad
repool eventgate-main in eqiad

JMeybohm added a subscriber: JMeybohm.Sep 2 2020, 10:40 AM

We added two additional partitions to resource_purge, and this seems to have solved the issue, mostly.

Joe mentioned this in T261763: Unexplained increase in save times, possibly associated with DC switchover.Sep 2 2020, 3:37 PM

FYI I also increased partitions to 3 for resource_change as well.

Darwinius added a subscriber: Darwinius.Sep 2 2020, 9:58 PM

Reporting the SAL entries that we mistakenly logged to another task:

Mentioned in SAL (#wikimedia-operations) [2020-09-02T14:28:58Z] <elukey> execute kafka topics --alter --topic codfw.resource-purge --partitions 3 and kafka topics --alter --topic eqiad.resource-purge --partitions 3 on kafka-main codfw

Mentioned in SAL (#wikimedia-operations) [2020-09-02T14:31:44Z] <elukey> execute kafka topics --alter --topic codfw.resource-purge --partitions 3 and kafka topics --alter --topic eqiad.resource-purge --partitions 3 on kafka-main eqiad

Mentioned in SAL (#wikimedia-operations) [2020-09-02T18:32:49Z] <ottomata> execute kafka topics --alter --topic codfw.resource-purge --partitions 3 and kafka topics --alter --topic eqiad.resource-purge --partitions 3 on kafka jumbo-eqiad (for consistency with main)

Mentioned in SAL (#wikimedia-operations) [2020-09-02T18:34:30Z] <ottomata> execute kafka topics --alter --topic codfw.resource_change --partitions 3 and kafka topics --alter --topic eqiad.resource_change --partitions 3 on kafka main-eqiad

Mentioned in SAL (#wikimedia-operations) [2020-09-02T18:37:21Z] <ottomata> execute kafka topics --alter --topic codfw.resource_change --partitions 3 and kafka topics --alter --topic eqiad.resource_change --partitions 3 on kafka main-codfw

Mentioned in SAL (#wikimedia-operations) [2020-09-02T18:38:11Z] <ottomata> execute kafka topics --alter --topic codfw.resource_change --partitions 3 and kafka topics --alter --topic eqiad.resource_change --partitions 3 on kafka jumbo-eqiad (for consistency with main)

eventgate-main latencies very high since the failover to codfwClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

eventgate-main latencies very high since the failover to codfw
Closed, ResolvedPublic
Actions