Page MenuHomePhabricator

Maintenance script causing failed DB updates on Beta
Closed, ResolvedPublic

Description

The database update job for beta has been failing for the last 12+ hours, aborting after 45 minutes each.

03:20:22 ...Update 'FlowFixLog:version2' already loBuild timed out (after 45 minutes). Marking the build as aborted.

Is the final entry. It's not this actual update that's killing the wiki, it's the one after it. Possibly this one?

$updater->addPostDatabaseUpdateMaintenance( 'FlowUpdateWorkflowPageId' );

It might not even be Flow's fault, if this is a script that comes after it (and tbh, I'm not seeing any recent merges around when we started failing that would cause it...). Needs investigation asap.

Event Timeline

demon renamed this task from Maintenance script for causing failed DB updates on Beta to Maintenance script causing failed DB updates on Beta.Dec 6 2016, 4:27 AM

This is a semi-duplicate of T152379: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki., except that bug describes an earlier issue where Flow was definitely breaking update.php. The cause of that was fixed, after which the current behavior (T152379#2846697) appeared.

It seems to now be aborting on

07:20:19 ...rev_user_ip in table flow_revision already modified by patch /srv/mediawiki-staging/php-master/extensions/Flow/db_patches/patch-revision_user_ip.sql.
07:20:19 ...have rev_type_id field in flow_revision table.
07:20:19 ...Build timed out (after 45 minutes). Marking the build as aborted.
08:05:00 Build was aborted
08:05:00 [PostBuildScript] - Execution post build scripts.

https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13317/console

It last passed on https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13275/

It started failing an hour later then began aborting yesterday

The problem seems to be many unapplied schema patches, and jobs that take a long time

I'm running update.php via shell, and it seems to be a lot of unapplied scheme changes... And a lot of long running Flow jobs

I guess you've found the problem. Probably need to add all those sql files to the updated hook so in the future they will update the sql.

Catrope claimed this task.
Catrope reassigned this task from Catrope to SBisson.