Page MenuHomePhabricator

wdqs-updater kafka poller should use an explicit consumer group
Closed, ResolvedPublic

Description

The kafka consumer group is used in reporting. For example, see consumer group lag grafana dashboard. The current random group that we use makes it difficult to know what the consumer really is. We should replace it by something more explicit.

Details

Related Gerrit Patches:

Event Timeline

Gehel created this task.Mar 2 2018, 11:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2018, 11:20 AM

We still have to ensure in some way that it is distinct from others. We could however add hostname prefix or maybe make it somehow more clear to improve logging. Maybe use hostname + timestamp + random would make it better?

Change 416861 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Change Kafka consumer ID to more recognizable string

https://gerrit.wikimedia.org/r/416861

Smalyshev triaged this task as Medium priority.Mar 6 2018, 10:41 PM

Change 417115 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/417115

Change 417115 merged by Gehel:
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/417115

Mentioned in SAL (#wikimedia-operations) [2018-03-08T13:56:04Z] <gehel> restart wdqs-updater on wdqs1005 to validate new config option - T188716

Change 416861 merged by jenkins-bot:
[wikidata/query/rdf@master] Make Kafka consumer ID be command-line parameter

https://gerrit.wikimedia.org/r/416861

Change 418873 had a related patch set uploaded (by Gehel; owner: Smalyshev):
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/418873

Change 418873 merged by Gehel:
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/418873

Smalyshev closed this task as Resolved.Apr 26 2018, 4:24 AM
Smalyshev claimed this task.