Page MenuHomePhabricator

Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2';
Closed, ResolvedPublic

Description

When beta runs it's hourly update, it is failing with this error

Error: 1091 Can't DROP 'flow_ext_ref_idx_v2'; check that column/key exists (10.68.23.30)\n\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(967): Wikimedia\\Rdbms\\Database->reportQueryError(string, integer, string, string, boolean)\n#1 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(3218): Wikimedia\\Rdbms\\Database->query(string, string)\n#2 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(3167): Wikimedia\\Rdbms\\Database->sourceStream(unknown type, NULL, NULL, string, NULL)\n#3 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(673): Wikimedia\\Rdbms\\Database->sourceFile(string)\n#4 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(929): DatabaseUpdater->applyPatch(string, boolean, string)\n#5 [internal function]: DatabaseUpdater->modifyField(string, string, string, boolean)\n#6 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(472): call_user_func_array(array, array)\n#7 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(440): DatabaseUpdater->runUpdates(array, boolean)\n#8 /srv/mediawiki-staging/php-master/maintenance/update.php(174): DatabaseUpdater->doUpdates(array)\n#9 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(111): UpdateMediaWiki->execute()\n#10 /srv/mediawiki-staging/php-master/maintenance/update.php(219): require_once(string)\n#11 /srv/mediawiki-staging/multiversion/MWScript.php(99): require_once(string)\n#12 {main}\n")

See
https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17320/console

The problem started 24 hours ago.

Event Timeline

Paladox created this task.May 24 2017, 9:35 PM
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptMay 24 2017, 9:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Paladox triaged this task as Unbreak Now! priority.May 24 2017, 9:36 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptMay 24 2017, 9:36 PM

@Paladox: Could you elaborate how this task is related to T110446#3289439 (if it is)? Thanks!

@Aklapper @demon thought my change could have broke beta. So we reverted it. turns out my change did not break beta. So not related now.

This flow_ext_ref table is woefully inconsistent in beta. A migration went poorly.

Change 355448 had a related patch set uploaded (by Catrope; owner: Paladox):
[mediawiki/extensions/Flow@master] Only drop flow_ext_ref_idx_v2 index if it exists. This should fix problems in beta labs arising from truncating updatelog. (modifyExtensionField() relies on updatelog to avoid repeating schema changes, and generally expects changes to be idempotent.

https://gerrit.wikimedia.org/r/355448

Paladox added a comment.EditedMay 25 2017, 12:36 AM

Fails at

Warning: fopen(/tmp/mw-UIDGenerator-UID-88): failed to open stream: Permission denied in /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php on line 437\n[205a11057f046f7f7f484fe8] [no req] RuntimeException from line 442 of /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php: Could not open '/tmp/mw-UIDGenerator-UID-88'.\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php(120): UIDGenerator->getTimeAndDelay(string, integer, integer, integer)\n#1 /srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/UUID.php(157):

So that's unrelated, and I'm fixing it ^

Fails at
Warning: fopen(/tmp/mw-UIDGenerator-UID-88): failed to open stream: Permission denied in /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php on line 437\n[205a11057f046f7f7f484fe8] [no req] RuntimeException from line 442 of /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php: Could not open '/tmp/mw-UIDGenerator-UID-88'.\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php(120): UIDGenerator->getTimeAndDelay(string, integer, integer, integer)\n#1 /srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/UUID.php(157):

This is probably caused by someone running update.php manually with the wrong user, while troubleshooting this. Can someone change the owner and group back to the normal user and group Jenkins runs it as?

demon claimed this task.May 26 2017, 5:24 PM
greg added a subscriber: greg.May 26 2017, 5:24 PM
Paladox lowered the priority of this task from Unbreak Now! to High.May 26 2017, 8:54 PM

@demon fixed it so lowering priority. But the change to flow should still be merged to prevent it breaking again.

demon removed demon as the assignee of this task.May 30 2017, 10:53 PM

Change 355448 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/355448

Change 357525 had a related patch set uploaded (by Paladox; owner: Paladox):
[mediawiki/extensions/Flow@REL1_29] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/357525

Change 357525 merged by jenkins-bot:
[mediawiki/extensions/Flow@REL1_29] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/357525

demon closed this task as Resolved.Jun 14 2017, 12:21 AM
demon claimed this task.