Page MenuHomePhabricator
Create Task
Maniphest T188716

wdqs-updater kafka poller should use an explicit consumer group
Closed, ResolvedPublic
Actions

Description

The kafka consumer group is used in reporting. For example, see consumer group lag grafana dashboard. The current random group that we use makes it difficult to know what the consumer really is. We should replace it by something more explicit.

Details

SubjectRepoBranchLines +/-
Add consumer ID to Updater launch stringoperations/puppetproduction+2 -2
Make Kafka consumer ID be command-line parameterwikidata/query/rdfmaster+25 -11
Revert "Add consumer ID to Updater launch string"operations/puppetproduction+2 -2
Add consumer ID to Updater launch stringoperations/puppetproduction+2 -2
Customize query in gerrit

Event Timeline

Gehel created this task.Mar 2 2018, 11:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2018, 11:20 AM
Smalyshev added a comment.Mar 6 2018, 8:51 PM

We still have to ensure in some way that it is distinct from others. We could however add hostname prefix or maybe make it somehow more clear to improve logging. Maybe use hostname + timestamp + random would make it better?

gerritbot subscribed.Mar 6 2018, 10:41 PM

Change 416861 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Change Kafka consumer ID to more recognizable string

https://gerrit.wikimedia.org/r/416861

gerritbot added a project: Patch-For-Review.Mar 6 2018, 10:41 PM
Smalyshev triaged this task as Medium priority.Mar 6 2018, 10:41 PM
Smalyshev moved this task from Backlog to Done on the Discovery-Wikidata-Query-Service-Sprint board.Mar 6 2018, 10:43 PM
Smalyshev moved this task from Done to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.
gerritbot added a comment.Mar 7 2018, 9:47 PM

Change 417115 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/417115

gerritbot added a comment.Mar 8 2018, 1:54 PM

Change 417115 merged by Gehel:
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/417115

Stashbot subscribed.Mar 8 2018, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-08T13:56:04Z] <gehel> restart wdqs-updater on wdqs1005 to validate new config option - T188716

gerritbot added a comment.Mar 8 2018, 5:57 PM

Change 416861 merged by jenkins-bot:
[wikidata/query/rdf@master] Make Kafka consumer ID be command-line parameter

https://gerrit.wikimedia.org/r/416861

gerritbot added a comment.Mar 12 2018, 9:24 AM

Change 418873 had a related patch set uploaded (by Gehel; owner: Smalyshev):
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/418873

gerritbot added a comment.Mar 12 2018, 5:23 PM

Change 418873 merged by Gehel:
[operations/puppet@production] Add consumer ID to Updater launch string

https://gerrit.wikimedia.org/r/418873

Smalyshev moved this task from Needs review to Done on the Discovery-Wikidata-Query-Service-Sprint board.Mar 13 2018, 12:48 AM
Smalyshev closed this task as Resolved.Apr 26 2018, 4:24 AM
Smalyshev claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL