The kafka topic mediawiki.job.cdnPurge is currently receiving many (most?) purge messages. Talking with @Joe I found out that not all the HTCP purges currently sent are also written to kafka .
To help with T133821, we would need to ensure that every time a multicast purge is sent, the purge is written to kafka as well.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Pchelolo | T250781 Move all purge traffic to kafka | |||
| Resolved | Smalyshev | T125138 Create Kafka event relayer subclass able to handle WAN cache and CDN purges | |||
| Resolved | Smalyshev | T134535 Create a class to make EventRelayer to send events to EventBus endpoint | |||
| Resolved | • Pchelolo | T254828 Move wikitech purges to kafka |
AFAIK cdnPurgeJob is only involved if the delayed purge is required if reboundDelay option is set. For every rebound purge job there's an instant purge multicast.
Is the request here to move all purges onto the jobqueue, regardless of whether they're delayed vs immediate?
The kafka topic mediawiki.job.cdnPurge is currently receiving many (most?) purge messages.
Maybe most by volume, but it's semantically very diferrent and a rather internal detail of when it is and isn't involving a job. The job spec is also considered internal to MW and no other stuff should rely on that.
In 2016, as part of T97562 and T125138, an event relay channel was set up for this in MediaWiki
This still exists today and is ready to be subscribed to e.g. by EventBus to to send things to Kafka, or anything else we want to do with it.
There is also a EventRelayerKafka class in MediaWiki that woudl talk to Kafka directly. It was originally created for broadcasting WANObjectCache purge events, before we decided to go with mcrouter.
Current event produder is is in CdnCacheUpdate.php
Change 596463 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Add EventBus backed EventRelayer for CDN purges.
Change 596721 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/core@master] WIP demo: DI for cdn purges
Change 599150 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Enable kafka purges production on group0 wikis
Change 596463 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Add EventBus backed EventRelayer for CDN purges.
Change 599150 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable kafka purges production on group0 wikis
Change 602125 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/deployment-charts@master] EventGate-main: allow resource-purge to be produced
Change 602125 merged by Ottomata:
[operations/deployment-charts@master] EventGate-main: allow resource-purge to be produced
Change 603514 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Disable HTCP purges for test.wikipedia.org
Change 603530 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Beta: Switch from HTCP purging to kafka purging
Change 603514 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable HTCP purges for test.wikipedia.org
Mentioned in SAL (#wikimedia-operations) [2020-06-08T18:20:29Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Disable HTCP purges for testwiki (T250781) (part 1) (duration: 00m 59s)
Mentioned in SAL (#wikimedia-operations) [2020-06-08T18:23:13Z] <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: Disable HTCP purges for testwiki (T250781) (part 2) (duration: 00m 56s)
Change 603649 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] [No-op]: Add precautions for kafka-purges before transition
Change 603654 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Enable kafka purges everywhere.
Change 603655 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Disbalse HTCP purges where kafka purges are enabled
Change 603675 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Allow accepting multiple event types in EnableEventBus variable.
Change 603675 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Allow accepting multiple event types in EnableEventBus variable.
Change 603649 merged by jenkins-bot:
[operations/mediawiki-config@master] [No-op]: Add precautions for kafka-purges before transition
Change 603654 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable kafka purges everywhere.
Change 604430 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: make upload consume purges from kafka
Change 604430 merged by Ema:
[operations/puppet@production] cache: make upload consume purges from kafka
Mentioned in SAL (#wikimedia-operations) [2020-06-10T16:06:06Z] <ema> cp3051: restart purged to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
Mentioned in SAL (#wikimedia-operations) [2020-06-10T16:12:07Z] <ema> restart purged on all cache hosts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
Mentioned in SAL (#wikimedia-operations) [2020-06-10T16:13:00Z] <ema> correction: restart purged on all *cache_upload* hosts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604430/ T250781 T133821
Change 603655 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable HTCP purges where kafka purges are enabled
Big wikis are now done, but there's still a bit of long tail work left
Change 604469 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Use EventRelayerNull for wikitech kafka purges
Change 604469 merged by jenkins-bot:
[operations/mediawiki-config@master] Use EventRelayerNull for wikitech kafka purges
Change 604743 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: make Kafka cluster name configurable
Change 603530 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta: Switch from HTCP purging to kafka purging
Change 604743 merged by Ema:
[operations/puppet@production] purged: make Kafka cluster name configurable
Change 607298 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] EventBus: Emit kafka purges for everything
Change 607300 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Mark kafka purge events as EVENT_PURGE type
Change 607298 merged by jenkins-bot:
[operations/mediawiki-config@master] EventBus: Emit kafka purges for everything
Change 607300 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Mark kafka purge events as EVENT_PURGE type
Change 607590 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Enable kafka purges on wikitech
Change 607593 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Disable HTCP purging everywhere
Change 607596 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Cleanup: remove temporary wmgDisableHTCP variable
Change 607590 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable kafka purges on wikitech
https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /607590
Change 607593 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable HTCP purging everywhere
Mentioned in SAL (#wikimedia-operations) [2020-07-08T18:17:19Z] <ppchelko@deploy1001> Synchronized wmf-config/reverse-proxy.php: Disable HTCP purging everywhere gerrit:607593 T250781 reverse-proxy.php (duration: 01m 04s)
Mentioned in SAL (#wikimedia-operations) [2020-07-08T18:18:41Z] <ppchelko@deploy1001> Synchronized wmf-config/wikitech.php: Disable HTCP purging everywhere gerrit:607593 T250781 wikitech.php (duration: 01m 04s)
Mentioned in SAL (#wikimedia-operations) [2020-07-08T18:20:20Z] <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: Disable HTCP purging everywhere gerrit:607593 T250781 CS.php (duration: 01m 03s)
Change 607596 merged by jenkins-bot:
[operations/mediawiki-config@master] Cleanup: remove temporary wmgDisableHTCP variable
Mentioned in SAL (#wikimedia-operations) [2020-07-08T18:27:12Z] <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Cleanup: remove temporary wmgDisableHTCP variable gerrit:607596 T250781 IS.php (duration: 01m 01s)
After deploying the latest config changes and some cleanups all purges are now going via Kafka and there's zero HTCP packets received by purged: https://grafana.wikimedia.org/d/RvscY1CZk/purged?orgId=1&from=1594230979757&to=1594232779757&var-datasource=esams%20prometheus%2Fops&var-cluster=cache_text&var-instance=cp3050
Change 612836 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: stop passing -mcast_addrs
Change 612836 merged by Ema:
[operations/puppet@production] purged: stop passing -mcast_addrs