Page MenuHomePhabricator

Rewrite beta-update-databases to not use unstable Configuration Matrix
Closed, ResolvedPublic


The Configuration Matrix has two main problems:

  1. It has lots of infrastructure overhead by spawning new jobs for each database update.
  1. The "Configuration Matrix" plugin for Jenkins is not stable and causes deadlocks on a daily bases (where it waits for executors on the deployment-bastion.eqiad node which clearly has sufficient available executors). See T72597. This particular deadlock has only ever been observed on the deployment-bastion.eqiad node from a beta-update-databases-eqiad job.

Please rewrite this Jenkins job to use instead spawn parallel child processes within a single job. E.g. using bash, python, or otherwise. The current job clearly doesn't work properly and it doesn't look like we're getting any closer with the various upstreams (Gearman, Jenkins) to figure out what is causing this deadlock.

The only tricky bit is orchestrating the console output. A few ideas:

  • Write it to separate files collected as build artefact.
  • Or; Buffer it and send to stdout one at a time.
  • Or; Send all to stdout with a unique prefix.

Event Timeline

Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Krinkle, greg.

Change 210618 had a related patch set uploaded (by Thcipriani):
Add script from jenkins beta-update-databases

Change 210619 had a related patch set uploaded (by Thcipriani):
Change beta-update-databases to python script

Change 210619 merged by jenkins-bot:
Change beta-update-databases to python script

hashar added a subscriber: hashar.

The Jenkins job now uses a freestyle project with

There is a puppet patch pending but it is T98342

Change 210618 merged by Yuvipanda:
beta: Add script from Jenkins beta-update-databases