Page MenuHomePhabricator
Create Task
Maniphest T288231

Deploy the wdqs streaming updater to production
Closed, ResolvedPublic
Actions

Description

Values tracked during the process (please fill them when making progress)

  • Week of sept. 20:
    • Send com about the rollout
    • increase retention to 1month on codfw.rdf-streaming-updater.mutation (topic name is about to change) in kafka-main@codfw
    • make sure retention is 1month on eqiad.rdf-streaming-updater.mutation (topic name is about to change) in kafka-main@eqiad
  • Week of sept. 27:
    • (friday oct. 1): bootstrap flink, note DUMP_START_DATE and FLINK_(EQIAD|CODFW)_JOB_START
    • (friday oct. 1): pre-fetch the dumps to wdqs1009 and wdqs2008 and note LEXEME_DUMP and ENTITY_DUMP
    • (friday oct. 1): start the data-reload cookbook with --reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_EQIAD_JOB_START] on wdqs1009
    • (friday oct. 1): start the data-reload cookbook with --reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_CODFW_JOB_START] on wdqs2008
    • (friday oct. 1): merge the activation of the streaming updater profile on wdqs2008 while the reload is happening there
  • Week of oct. 4:
    • Monitor that the reload is progressing properly
  • Week of oct. 11:
    • Send a quick reminder com to users
    • Start data transfer (use the new option to activate kafka offsets propagation and always activate the streaming_updater profile via puppet on the target machine)
    • (internal cluster)
      • wdqs1009 -> wdqs1003
      • wdqs2008 -> wdqs2005
      • wdqs1003 -> wdqs1008
      • wdqs2008 -> wdqs2006
      • wdqs1003 -> wdqs1011
    • (external cluster)
      • wdqs1003 -> wdqs1004
      • wdqs2008 -> wdqs2001
      • wdqs1003 -> wdqs1005
      • wdqs2008 -> wdqs2002
      • wdqs1003 -> wdqs1006
      • wdqs2008 -> wdqs2003
      • wdqs1003 -> wdqs1007
      • wdqs2008 -> wdqs2004
      • wdqs1003 -> wdqs1012
      • wdqs2008 -> wdqs2007
      • wdqs1003 -> wdqs1013

Note: Migrations ended at 6am Oct 19.

Note: wdqs1010 is kept with the old updater and the old journal.

Details

SubjectRepoBranchLines +/-
wdqs: enable the streaming updater on wdqs1013operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1012operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1007operations/puppetproduction+2 -0
profile::query_service::monitor::wikidata: update streaming lag monitoroperations/puppetproduction+3 -3
wdqs: enable the streaming updater on wdqs1006operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1005operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1004operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1011operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1008operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2007operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs1003operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2004operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2003operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2002operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2001operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2006operations/puppetproduction+2 -0
wdqs: enable the streaming updater on wdqs2005operations/puppetproduction+2 -0
wdqs: activate the streaming_updater role on wdqs2008operations/puppetproduction+2 -0
wdqs: Prepare streaming updater settingsoperations/puppetproduction+28 -6
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

dcausse created this task.Aug 5 2021, 12:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2021, 12:24 PM
dcausse added a parent task: T244590: [Epic] Rework the WDQS updater as an event driven application.Aug 5 2021, 12:24 PM
dcausse added a parent task: T285710: WDQS lag detection required manual adjustment during DC switchover.
Maintenance_bot added a project: Wikidata.Aug 5 2021, 12:45 PM
• MPhamWMF triaged this task as High priority.Aug 9 2021, 6:09 PM
• MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board.Sep 13 2021, 3:17 PM
• MPhamWMF added a project: Discovery-Search (Current work).
• MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.Sep 13 2021, 3:44 PM
gerritbot added a comment.Sep 15 2021, 11:21 AM

Change 721280 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: Add a streaming updater profile

https://gerrit.wikimedia.org/r/721280

gerritbot added a project: Patch-For-Review.Sep 15 2021, 11:21 AM

Change 721281 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: activate the streaming_updater role on wdqs2008

https://gerrit.wikimedia.org/r/721281

dcausse updated the task description. (Show Details)Sep 15 2021, 11:57 AM
dcausse updated the task description. (Show Details)Sep 15 2021, 1:25 PM
gerritbot added a comment.Sep 17 2021, 12:52 AM

Change 721280 merged by Ryan Kemper:

[operations/puppet@production] wdqs: Prepare streaming updater settings

https://gerrit.wikimedia.org/r/721280

dcausse updated the task description. (Show Details)Sep 17 2021, 1:40 PM
dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Sep 20 2021, 3:30 PM
dcausse updated the task description. (Show Details)
dcausse updated the task description. (Show Details)Oct 1 2021, 8:00 AM
dcausse updated the task description. (Show Details)Oct 1 2021, 9:29 AM
dcausse updated the task description. (Show Details)Oct 1 2021, 10:25 AM
gerritbot added a comment.Oct 1 2021, 1:16 PM

Change 721281 merged by Gehel:

[operations/puppet@production] wdqs: activate the streaming_updater role on wdqs2008

https://gerrit.wikimedia.org/r/721281

dcausse updated the task description. (Show Details)Oct 1 2021, 1:17 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 1 2021, 2:10 PM
gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730794 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2005

https://gerrit.wikimedia.org/r/730794

gerritbot added a project: Patch-For-Review.Oct 14 2021, 1:23 PM

Change 730795 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2006

https://gerrit.wikimedia.org/r/730795

gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730796 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2001

https://gerrit.wikimedia.org/r/730796

gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730797 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2002

https://gerrit.wikimedia.org/r/730797

gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730798 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2003

https://gerrit.wikimedia.org/r/730798

gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730799 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2004

https://gerrit.wikimedia.org/r/730799

gerritbot added a comment.Oct 14 2021, 1:23 PM

Change 730800 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2007

https://gerrit.wikimedia.org/r/730800

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730814 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1003

https://gerrit.wikimedia.org/r/730814

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730815 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1008

https://gerrit.wikimedia.org/r/730815

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730816 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1011

https://gerrit.wikimedia.org/r/730816

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730817 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1004

https://gerrit.wikimedia.org/r/730817

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730818 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1005

https://gerrit.wikimedia.org/r/730818

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730819 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1006

https://gerrit.wikimedia.org/r/730819

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730820 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1007

https://gerrit.wikimedia.org/r/730820

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730821 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1012

https://gerrit.wikimedia.org/r/730821

gerritbot added a comment.Oct 14 2021, 2:35 PM

Change 730822 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1013

https://gerrit.wikimedia.org/r/730822

Stashbot added a comment.Oct 14 2021, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T15:52:51Z] <ryankemper> T288231 ryankemper@wdqs2005:~$ sudo depool

Stashbot added a comment.Oct 14 2021, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T15:54:18Z] <ryankemper> T288231 ryankemper@wdqs2008:~$ sudo depool

gerritbot added a comment.Oct 14 2021, 4:00 PM

Change 730794 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2005

https://gerrit.wikimedia.org/r/730794

Stashbot added a comment.Oct 14 2021, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T16:04:53Z] <ryankemper> T288231 ryankemper@wdqs2005:~$ sudo run-puppet-agent --force

Stashbot added a comment.Oct 14 2021, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T16:07:07Z] <ryankemper> T288231 About to ctrl+c out of ongoing data transfer because puppet run following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/730794 restarted blazegraph; we'll manually disable updater and kick off the transfer again

Stashbot added a comment.Oct 14 2021, 4:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T16:44:13Z] <ryankemper> T288231 Manually killed dangling pigz / nc processes on wdqs2008 (and wdqs2005 implicitly). Should be in the right state to re-start the data-transfer cookbook from again

RKemper subscribed.EditedOct 14 2021, 10:27 PM
[data-transfer cookbook -> kafka offset output]

(including this output since it's the first time we've relied on the new kafka offset part of the data-transfer process; won't be posting the output for any subsequent cookbooks runs)

Also note that the output is a little tricky to parse. I can't tell if it did succeed and the Error sending OffsetCommitRequest_v2... log lines were errors that were resolved upon retry, or if the cookbook falsely reported the offset. But judging by the fact that wdqs2005 is reporting the correct number of triples, I have to assume that it worked properly. (This also lines up with the fact that we do have log lines that seem to report success: Offsets set for target cluster "main", site "codfw" and consumer group "wdqs2005")

Transferring Kafka offsets
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
Probing node bootstrap-2 broker version
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
Probing node bootstrap-1 broker version
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
Same cluster, setting offsets...
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to kafka-main2004.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: connecting to kafka-main2001.codfw.wmnet:9093 [('2620:0:860:101:10:192:0:17', 9093, 0, 0) IPv6]
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection complete.
<BrokerConnection node_id=bootstrap-2 host=kafka-main2004.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to kafka-main2005.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:46', 9093, 0, 0) IPv6]
Group coordinator for wdqs2008 is BrokerMetadata(nodeId='coordinator-2003', host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2008
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Connection complete.
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0) IPv6]
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection complete.
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
Extracted offsets from source cluster "main", site "codfw" and consumer group "wdqs2008".
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to kafka-main2005.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:46', 9093, 0, 0) IPv6]
Group coordinator for wdqs2005 is BrokerMetadata(nodeId='coordinator-2003', host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2005
Error sending OffsetCommitRequest_v2 to node coordinator-2003 [NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <connecting> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0) IPv6]
Error sending OffsetCommitRequest_v2 to node coordinator-2003 [NodeNotReadyError: coordinator-2003]
Error sending OffsetCommitRequest_v2 to node coordinator-2003 [NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection complete.
Error sending OffsetCommitRequest_v2 to node coordinator-2003 [NodeNotReadyError: coordinator-2003]
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <handshake> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
<BrokerConnection node_id=bootstrap-1 host=kafka-main2004.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=bootstrap-3 host=kafka-main2005.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing connection.
Offsets set for target cluster "main", site "codfw" and consumer group "wdqs2005".
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=2004 host=kafka-main2004.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=2001 host=kafka-main2001.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=2005 host=kafka-main2005.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing connection.
<BrokerConnection node_id=coordinator-2003 host=kafka-main2003.codfw.wmnet:9093 <connected> [IPv6 ('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
Done.
Starting services [systemctl start wdqs-blazegraph && sleep 10 && systemctl start wdqs-updater]
Stashbot added a comment.Oct 14 2021, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:28:52Z] <ryankemper> T288231 ryankemper@wdqs2005:~$ sudo pool: transfer completed successfully; tests passing on host (used ssh -L 9999:localhost:80 wdqs2005.codfw.wmnet to establish tunnel)

gerritbot added a comment.Oct 14 2021, 10:30 PM

Change 730795 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2006

https://gerrit.wikimedia.org/r/730795

Stashbot added a comment.Oct 14 2021, 10:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:32:47Z] <ryankemper> T288231 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/730795; proceeding to data-transfer on wdqs2006: sudo rm -fv /srv/wdqs/data_loaded on wdqs2006 followed by ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "streaming updater cutover for wdqs2005" --blazegraph_instance blazegraph --task-id T288231

Stashbot added a comment.Oct 14 2021, 10:33 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:33:48Z] <ryankemper> T288231 Forgot about running puppet-agent on wdqs2006; aborted cookbook run

Stashbot added a comment.Oct 14 2021, 10:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-14T22:35:00Z] <ryankemper> T288231 Ran puppet on wdqs2006, now back to the cookbook run

Stashbot added a comment.Oct 15 2021, 2:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-15T02:14:38Z] <ryankemper> T288231 wdqs2006 data transfer complete and all tests passing on the host. All of codfw wdqs-internal is on the new streaming updater

• Zbyszko updated the task description. (Show Details)Oct 15 2021, 7:14 AM
Gehel updated the task description. (Show Details)Oct 15 2021, 7:15 AM
gerritbot added a comment.Oct 15 2021, 7:40 AM

Change 730796 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2001

https://gerrit.wikimedia.org/r/730796

• Zbyszko subscribed.Oct 15 2021, 7:43 AM
Gehel updated the task description. (Show Details)Oct 15 2021, 9:12 AM
gerritbot added a comment.Oct 15 2021, 9:14 AM

Change 730797 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2002

https://gerrit.wikimedia.org/r/730797

Gehel updated the task description. (Show Details)Oct 15 2021, 11:44 AM
gerritbot added a comment.Oct 15 2021, 11:48 AM

Change 730798 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2003

https://gerrit.wikimedia.org/r/730798

Gehel updated the task description. (Show Details)Oct 15 2021, 1:21 PM
gerritbot added a comment.Oct 15 2021, 1:23 PM

Change 730799 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2004

https://gerrit.wikimedia.org/r/730799

gerritbot added a comment.Oct 15 2021, 1:30 PM

Change 730814 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1003

https://gerrit.wikimedia.org/r/730814

Gehel updated the task description. (Show Details)Oct 15 2021, 3:09 PM
Gehel updated the task description. (Show Details)Oct 15 2021, 3:19 PM
gerritbot added a comment.Oct 15 2021, 3:21 PM

Change 730800 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs2007

https://gerrit.wikimedia.org/r/730800

gerritbot added a comment.Oct 15 2021, 3:22 PM

Change 730815 merged by Gehel:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1008

https://gerrit.wikimedia.org/r/730815

RKemper updated the task description. (Show Details)Oct 15 2021, 5:19 PM
gerritbot added a comment.Oct 15 2021, 8:07 PM

Change 730816 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1011

https://gerrit.wikimedia.org/r/730816

RKemper updated the task description. (Show Details)Oct 15 2021, 9:48 PM
gerritbot added a comment.Oct 15 2021, 9:49 PM

Change 730817 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1004

https://gerrit.wikimedia.org/r/730817

RKemper updated the task description. (Show Details)Oct 15 2021, 11:28 PM
gerritbot added a comment.Oct 15 2021, 11:46 PM

Change 730818 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1005

https://gerrit.wikimedia.org/r/730818

gerritbot added a comment.Oct 16 2021, 2:17 AM

Change 730819 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1006

https://gerrit.wikimedia.org/r/730819

elukey subscribed.Oct 16 2021, 3:38 PM

Hi folks, I was trying to figure out why we had so many UNKNOWNs related to wdqs nodes in icinga and I noticed this task. The new streaming updater metrics seem to have a typo:

wdqs_streaming_updater_kafka_stream_consumer_lag_Value

Meanwhile this is the puppet config:

query           => "scalar(wdqs_streaming_updater_kafka_stream_consumer_lag{instance=\"${::hostname}:9101\"})",
gerritbot added a comment.Oct 17 2021, 4:18 PM

Change 731282 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::query_service::monitor::wikidata: update streaming lag monitor

https://gerrit.wikimedia.org/r/731282

gerritbot added a comment.Oct 17 2021, 8:12 PM

Change 731282 merged by Ryan Kemper:

[operations/puppet@production] profile::query_service::monitor::wikidata: update streaming lag monitor

https://gerrit.wikimedia.org/r/731282

• Zbyszko updated the task description. (Show Details)Oct 18 2021, 6:45 AM
gerritbot added a comment.Oct 19 2021, 12:37 AM

Change 730820 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1007

https://gerrit.wikimedia.org/r/730820

gerritbot added a comment.Oct 19 2021, 2:20 AM

Change 730821 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1012

https://gerrit.wikimedia.org/r/730821

gerritbot added a comment.Oct 19 2021, 4:30 AM

Change 730822 merged by Ryan Kemper:

[operations/puppet@production] wdqs: enable the streaming updater on wdqs1013

https://gerrit.wikimedia.org/r/730822

RKemper updated the task description. (Show Details)Oct 19 2021, 4:32 AM
dcausse updated the task description. (Show Details)Oct 19 2021, 7:12 AM
dcausse moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.
Legoktm awarded a token.Oct 19 2021, 11:22 PM
Gehel closed this task as Resolved.Oct 28 2021, 8:09 AM
Gehel claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits