Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	Sep 2 2024, 9:48 AM

Description

wdqs2021 appears to have its updater consuming from codfw.rdf-streaming-updater.mutation and thus causing a high number of divergences since its journal was probably transferred from a journal loaded with the main graph. The data in this journal is probably not salvageable and a transfer from a sane journal should be done.
The service definition /lib/systemd/system/wdqs-updater.service does properly mention codfw.rdf-streaming-updater.mutation-main so I'm not sure what happened.

AC:

transfer the journal from wdqs2022 to wdqs2021
wdqs2021 consumers from codfw.rdf-streaming-updater.mutation-main (select wdqs2021 codfw.rdf-streaming-updater.mutation-main rate): https://grafana-rw.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1&var-site=codfw&var-k8sds=codfw%20prometheus%2Fk8s&var-opsds=codfw%20prometheus%2Fops&var-cluster_name=wdqs&viewPanel=2
the number divergences is close to 0 for wdqs2021: https://grafana-rw.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1&var-site=codfw&var-k8sds=codfw%20prometheus%2Fk8s&var-opsds=codfw%20prometheus%2Fops&var-cluster_name=wdqs&viewPanel=9

Related Objects
Search...

Status	Assigned	Task
Open	None	T335067 Epic: Wikidata Query Service stabilization
Resolved	BTracy-WMF	T337013 [Epic] Splitting the graph in WDQS
Resolved	RKemper	T364363 [Epic] Productionize federated wdqs graph-split endpoints
Resolved	bking	T373791 Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022

Event Timeline

dcausse created this task.Sep 2 2024, 9:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2024, 9:48 AM

dcausse added a parent task: T364363: [Epic] Productionize federated wdqs graph-split endpoints.Sep 2 2024, 9:49 AM

Maintenance_bot added a project: Wikidata.Sep 2 2024, 10:29 AM

Gehel triaged this task as High priority.Sep 2 2024, 3:08 PM

Gehel moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.

Gehel edited projects, added Data-Platform-SRE (2024.08.17 - 2024.09.06); removed Data-Platform-SRE.

Ottomata subscribed.Sep 3 2024, 12:04 PM

dr0ptp4kt subscribed.Sep 3 2024, 3:46 PM

bking claimed this task.Sep 3 2024, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-03T16:09:29Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Mentioned in SAL (#wikimedia-operations) [2024-09-03T16:57:55Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Unfortunately wdqs2021 is still consumer from the wrong topic after the transfer.
Looking closer it appears that the service definition for the wdqs-updater is duplicated in two locations:

/etc/systemd/system/wdqs-updater.service containing the wrong topic codfw.rdf-streaming-updater.mutation
/lib/systemd/system/wdqs-updater.service with the right topic codfw.rdf-streaming-updater.mutation-main

The version in /etc is probably taking precedence, this explains why previous transfer did not work as expect. The cause as to why /etc/systemd/system/wdqs-updater.service is there is unknown to me.

So we need to cleanup this host (possibly re-image it) to make sure no other stale files are present and re-run the transfer.

Gehel edited projects, added Data-Platform-SRE (2024.09.06 - 2024.09.27); removed Data-Platform-SRE (2024.08.17 - 2024.09.06).Sep 6 2024, 9:49 AM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.Sep 6 2024, 9:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2021.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2021.codfw.wmnet with OS bullseye executed with errors:

wdqs2021 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wdqs2021.codfw.wmnet" to get a root shellbut depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2021.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2021.codfw.wmnet with OS bullseye executed with errors:

wdqs2021 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409111344_bking_876306_wdqs2021.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wdqs2021.codfw.wmnet" to get a root shellbut depending on the failure this may not work.

Mentioned in SAL (#wikimedia-operations) [2024-09-11T15:50:21Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2021.codfw.wmnet with reason: T373791

Mentioned in SAL (#wikimedia-operations) [2024-09-11T15:50:36Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2021.codfw.wmnet with reason: T373791

Mentioned in SAL (#wikimedia-operations) [2024-09-11T16:25:40Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:17:42Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards

bking updated the task description. (Show Details)Sep 11 2024, 5:28 PM

The reimage and data-transfer from wdqs2022 to wdqs2021 appear to have worked.

I've repooled the wdqs-main pool in codfw: sudo confctl --object-type discovery select 'dnsdisc=wdqs-main,name=codfw' set/pooled=true
I've looked at the dashboards linked above and it appears the triples divergences are zero.
I'm not 100% sure about what the Consumer Kafka dashboard is supposed to look like.

Leaving in "Needs Review" so @dcausse can confirm it's working or send back to SRE if not.

@bking everything looks good, thanks!

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:55:41Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:55:57Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791

Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
Closed, ResolvedPublic
Actions

Related Objects
Search...