Page MenuHomePhabricator
Create Task
Maniphest T166266

Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2';
Closed, ResolvedPublic
Actions

Description

When beta runs it's hourly update, it is failing with this error

Error: 1091 Can't DROP 'flow_ext_ref_idx_v2'; check that column/key exists (10.68.23.30)\n\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(967): Wikimedia\\Rdbms\\Database->reportQueryError(string, integer, string, string, boolean)\n#1 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(3218): Wikimedia\\Rdbms\\Database->query(string, string)\n#2 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(3167): Wikimedia\\Rdbms\\Database->sourceStream(unknown type, NULL, NULL, string, NULL)\n#3 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(673): Wikimedia\\Rdbms\\Database->sourceFile(string)\n#4 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(929): DatabaseUpdater->applyPatch(string, boolean, string)\n#5 [internal function]: DatabaseUpdater->modifyField(string, string, string, boolean)\n#6 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(472): call_user_func_array(array, array)\n#7 /srv/mediawiki-staging/php-master/includes/installer/DatabaseUpdater.php(440): DatabaseUpdater->runUpdates(array, boolean)\n#8 /srv/mediawiki-staging/php-master/maintenance/update.php(174): DatabaseUpdater->doUpdates(array)\n#9 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(111): UpdateMediaWiki->execute()\n#10 /srv/mediawiki-staging/php-master/maintenance/update.php(219): require_once(string)\n#11 /srv/mediawiki-staging/multiversion/MWScript.php(99): require_once(string)\n#12 {main}\n")

See
https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17320/console

The problem started 24 hours ago.

Details

SubjectRepoBranchLines +/-
Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.mediawiki/extensions/FlowREL1_29+6 -4
Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.mediawiki/extensions/Flowmaster+6 -4
Customize query in gerrit

Related Objects

Event Timeline

Paladox created this task.May 24 2017, 9:35 PM
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptMay 24 2017, 9:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Paladox triaged this task as Unbreak Now! priority.May 24 2017, 9:36 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptMay 24 2017, 9:36 PM
Paladox mentioned this in T165805: MediaWiki updater is broken with it failing with Query: SHOW TABLE STATUS LIKE 'revision' ESCAPE '`' .May 24 2017, 9:53 PM
Aklapper added a comment.May 24 2017, 10:22 PM

@Paladox: Could you elaborate how this task is related to T110446#3289439 (if it is)? Thanks!

Paladox added a comment.May 24 2017, 10:23 PM

@Aklapper @demon thought my change could have broke beta. So we reverted it. turns out my change did not break beta. So not related now.

• demon added a comment.May 24 2017, 10:24 PM

This flow_ext_ref table is woefully inconsistent in beta. A migration went poorly.

Paladox added a project: MediaWiki-libs-Rdbms.May 24 2017, 10:29 PM
gerritbot subscribed.May 25 2017, 12:27 AM

Change 355448 had a related patch set uploaded (by Catrope; owner: Paladox):
[mediawiki/extensions/Flow@master] Only drop flow_ext_ref_idx_v2 index if it exists. This should fix problems in beta labs arising from truncating updatelog. (modifyExtensionField() relies on updatelog to avoid repeating schema changes, and generally expects changes to be idempotent.

https://gerrit.wikimedia.org/r/355448

gerritbot added a project: Patch-For-Review.May 25 2017, 12:27 AM
Paladox added a comment.EditedMay 25 2017, 12:36 AM

Fails at

Warning: fopen(/tmp/mw-UIDGenerator-UID-88): failed to open stream: Permission denied in /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php on line 437\n[205a11057f046f7f7f484fe8] [no req] RuntimeException from line 442 of /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php: Could not open '/tmp/mw-UIDGenerator-UID-88'.\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php(120): UIDGenerator->getTimeAndDelay(string, integer, integer, integer)\n#1 /srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/UUID.php(157):

• demon added a comment.May 25 2017, 12:41 AM

So that's unrelated, and I'm fixing it ^

• Mattflaschen-WMF added a comment.May 25 2017, 12:41 AM
In T166266#3290883, @Paladox wrote:

Fails at

Warning: fopen(/tmp/mw-UIDGenerator-UID-88): failed to open stream: Permission denied in /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php on line 437\n[205a11057f046f7f7f484fe8] [no req] RuntimeException from line 442 of /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php: Could not open '/tmp/mw-UIDGenerator-UID-88'.\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/utils/UIDGenerator.php(120): UIDGenerator->getTimeAndDelay(string, integer, integer, integer)\n#1 /srv/mediawiki-staging/php-master/extensions/Flow/includes/Model/UUID.php(157):

This is probably caused by someone running update.php manually with the wrong user, while troubleshooting this. Can someone change the owner and group back to the normal user and group Jenkins runs it as?

• demon claimed this task.May 26 2017, 5:24 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptMay 26 2017, 5:24 PM
• demon moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.May 26 2017, 5:24 PM
greg subscribed.May 26 2017, 5:24 PM
Paladox lowered the priority of this task from Unbreak Now! to High.May 26 2017, 8:54 PM

@demon fixed it so lowering priority. But the change to flow should still be merged to prevent it breaking again.

• demon removed • demon as the assignee of this task.May 30 2017, 10:53 PM
• demon moved this task from In-progress to Done (within RelEng) on the Release-Engineering-Team (Kanban) board.
gerritbot added a comment.Jun 6 2017, 11:23 PM

Change 355448 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/355448

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)).Jun 7 2017, 12:00 AM
gerritbot added a comment.Jun 7 2017, 12:42 AM

Change 357525 had a related patch set uploaded (by Paladox; owner: Paladox):
[mediawiki/extensions/Flow@REL1_29] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/357525

• jmatazzoni edited projects, added Collaboration-Team-Triage (Collab-Team-Q4-Apr-Jun-2017); removed Collaboration-Team-Triage.Jun 13 2017, 7:54 PM
• jmatazzoni moved this task from Untriaged to Needs Review on the Collaboration-Team-Triage (Collab-Team-Q4-Apr-Jun-2017) board.
gerritbot added a comment.Jun 13 2017, 11:41 PM

Change 357525 merged by jenkins-bot:
[mediawiki/extensions/Flow@REL1_29] Replace index flow_ext_ref_idx_v2 with flow_ext_ref_idx_3 index if it exists.

https://gerrit.wikimedia.org/r/357525

• demon closed this task as Resolved.Jun 14 2017, 12:21 AM
• demon claimed this task.
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits