Page MenuHomePhabricator

Rewrite beta-update-databases to not use unstable Configuration Matrix
Closed, ResolvedPublic

Description

The Configuration Matrix has two main problems:

  1. It has lots of infrastructure overhead by spawning new jobs for each database update.
  1. The "Configuration Matrix" plugin for Jenkins is not stable and causes deadlocks on a daily bases (where it waits for executors on the deployment-bastion.eqiad node which clearly has sufficient available executors). See T72597. This particular deadlock has only ever been observed on the deployment-bastion.eqiad node from a beta-update-databases-eqiad job.

Please rewrite this Jenkins job to use instead spawn parallel child processes within a single job. E.g. using bash, python, or otherwise. The current job clearly doesn't work properly and it doesn't look like we're getting any closer with the various upstreams (Gearman, Jenkins) to figure out what is causing this deadlock.

The only tricky bit is orchestrating the console output. A few ideas:

  • Write it to separate files collected as build artefact.
  • Or; Buffer it and send to stdout one at a time.
  • Or; Send all to stdout with a unique prefix.

Event Timeline

Krinkle created this task.Apr 15 2015, 11:30 PM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Krinkle, greg.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 15 2015, 11:30 PM
hashar triaged this task as Low priority.Apr 29 2015, 10:01 AM

Change 210618 had a related patch set uploaded (by Thcipriani):
Add script from jenkins beta-update-databases

https://gerrit.wikimedia.org/r/210618

Change 210619 had a related patch set uploaded (by Thcipriani):
Change beta-update-databases to python script

https://gerrit.wikimedia.org/r/210619

Change 210619 merged by jenkins-bot:
Change beta-update-databases to python script

https://gerrit.wikimedia.org/r/210619

hashar added a subscriber: hashar.

The Jenkins job now uses a freestyle project with https://gerrit.wikimedia.org/r/#/c/210619/

There is a puppet patch pending but it is T98342

hashar closed this task as Resolved.May 13 2015, 10:14 AM
greg moved this task from INBOX to Done on the Release-Engineering-Team board.May 23 2015, 2:14 PM

Change 210618 merged by Yuvipanda:
beta: Add script from Jenkins beta-update-databases

https://gerrit.wikimedia.org/r/210618

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptAug 7 2015, 11:14 PM