Page MenuHomePhabricator

wdqs indexer pipeline: failure due to pod name clash
Closed, ResolvedPublic

Description

The wdqs indexer pipeline failed due to an existing pod.

The WMF maintains WMFKubernetesPodOperator as a sub-class of the KubernetesPodOperation. This sub-class will add a randomized suffix to the pod name, avoiding issues like this one.

While using the WMFKubernetesPodOperator will not solve the problem of the existing (garbage?) pod by the same name, it will at least avoid subsequent runs of the pipeline running into the same problem.

AC:

  • The dump tasks in the indexer pipeline use KubernetesPodOperator with enabled random suffix support
  • The cause of the duplicate pod has been determined and resolved. (we will keep an eye on the pods, but at this point there is no way to investigate, seeing as the previous offender has been removed in order to fix the pipeline)

Event Timeline

trueg changed the task status from Open to In Progress.Sep 7 2026, 10:48 AM

The cause of the dangling pod is unclear. In the meantime it has been cleared up. If it should happen at any point again, we can investigate. For now, the only thing we can do is to keep an eye on it:

$ kube_env mediawiki-dumps-legacy dse-k8s-eqiad
$ kubectl get pod
NAME                                                   READY   STATUS    RESTARTS   AGE
mediawiki-dump-commons-mediainfo-rdf-dump              4/4     Running   0          23h
mediawiki-dump-wikidata-all-json-dump                  4/4     Running   0          18h
mediawiki-dump-wikidata-all-rdf-dump                   4/4     Running   0          19h
mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-szj6k   2/2     Running   0          18h
mediawiki-dumps-legacy-sync-toolbox-7cdb965bcd-72558   2/2     Running   0          18h