Page MenuHomePhabricator

addWiki.php broken creating ES tables
Open, Needs TriagePublic

Description

Initialising external storage cluster24...
[6f30182d25a2d432f75d027c] [no req]   Wikimedia\Rdbms\DBReadOnlyError from line 1163 of /srv/mediawiki/php-1.33.0-wmf.9/includes/libs/rdbms/database/Database.php: Database is read-only: The database has been automatically locked until the replica database servers become available
Backtrace:
#0 /srv/mediawiki/php-1.33.0-wmf.9/includes/libs/rdbms/database/Database.php(4354): Wikimedia\Rdbms\Database->query(string, string)
#1 /srv/mediawiki/php-1.33.0-wmf.9/extensions/WikimediaMaintenance/addWiki.php(229): Wikimedia\Rdbms\Database->sourceStream(resource)
#2 /srv/mediawiki/php-1.33.0-wmf.9/maintenance/doMaintenance.php(94): AddWiki->execute()
#3 /srv/mediawiki/php-1.33.0-wmf.9/extensions/WikimediaMaintenance/addWiki.php(399): include(string)
#4 /srv/mediawiki/multiversion/MWScript.php(100): include(string)
#5 {main}

While doing T197616: Create a production test wiki in group0 to parallel Wikimedia Commons

Event Timeline

Reedy created this task.Jan 3 2019, 7:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2019, 7:39 PM
Reedy added a subscriber: aaron.EditedJan 3 2019, 8:05 PM

Guessing this is maybe related to https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/commit/c0569fb89a449d86c5ad5aad5a178ac109de96df

Running addWiki.php caused some major replag and some index creations to seemingly hang on x1 on dbstore1002 and s4 on dbstore1002

Reedy updated the task description. (Show Details)Jan 3 2019, 8:35 PM
Reedy added a comment.Jan 3 2019, 9:51 PM

This might just be dbstore1002 sucking

jcrespo added a subscriber: jcrespo.Jan 7 2019, 6:06 PM

Did it finish?

Did it finish?

It wasn't related to dbstore1002 in the end, just something that got upset at the same time.

But the tables weren't created on extension1 (not sure why), and @jcrespo had to create them manually. Not had to create any more wikis yet, but there's some more that are up soon for doing it

kchapman moved this task from Inbox to Doing on the Performance-Team board.Jan 22 2019, 9:32 PM
kchapman assigned this task to aaron.
aaron added a comment.Jan 23 2019, 7:45 AM

I see

Error: 1050 Table 'blobs_cluster24' already exists (10.64.32.184)

from attempts to retry (presumably). I'll add a tableExist() to handle that better, since ES involves multiple masters (given that the $wgDefaultExternalStores array has 2+ things).

aaron added a comment.Jan 23 2019, 7:49 AM

To be useful, the tables.sql and other bits need to also handle idempotence or have some script parameter to skip them though...

Change 486406 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/WikimediaMaintenance@master] Make addWiki better handle second runs in case of errors

https://gerrit.wikimedia.org/r/486406

aaron moved this task from Doing to Blocked on the Performance-Team board.Feb 19 2019, 9:43 PM

Given that happens intermittently, is this task a blocker for making new wikis?

Reedy added a comment.Feb 21 2019, 4:57 PM

How do you know it happens intermittently?

How do you know it happens intermittently?

Oh I thought it's dbstore1002 being grumpy.

Any progress here?

@Reedy would you have time to review the related patch?

ARR8 added a subscriber: ARR8.Mar 16 2019, 11:34 PM

@Reedy There are several wikis pending, is it possible to try to create them now?

Change 486406 had a related patch set uploaded (by Ladsgroup; owner: Aaron Schulz):
[mediawiki/extensions/WikimediaMaintenance@master] Make addWiki better handle second runs in case of errors

https://gerrit.wikimedia.org/r/486406

Change 486406 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@master] Make addWiki better handle second runs in case of errors

https://gerrit.wikimedia.org/r/486406

okay, I reviewed the patch and merged it. I also made a follow up to clean the big method there if anyone wants to review it: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaMaintenance/+/499014
Later this week I can make all of the needed wikis.

Krinkle closed this task as Resolved.Mar 26 2019, 12:38 AM
aaron removed aaron as the assignee of this task.Fri, May 10, 2:47 AM
Krinkle moved this task from Blocked to Inbox on the Performance-Team board.Fri, May 10, 2:18 PM
kchapman claimed this task.Mon, May 13, 7:46 PM

Assigned to myself to track down ownership of the script

So, the followup for this...

I just did T223770: Please create beta zhwikivoyage and this was the error that came up, which is why we're not creating any new wikis atm. I think this bug should be closed, and a new one filed

[XOEw1KwQBHcAAH1T3fQAAABO] /wiki/%E9%A6%96%E9%A1%B5 MediaWiki\Revision\RevisionAccessException from line 1466 of /srv/mediawiki/php-master/includes/Revision/RevisionStore.php: Failed to load data blob from tt:1: Failed to load blob from address tt:1

Backtrace:

#0 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(1637): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#1 [internal function]: Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords(MediaWiki\Revision\SlotRecord)
#2 /srv/mediawiki/php-master/includes/Revision/SlotRecord.php(307): call_user_func(Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords;2290, MediaWiki\Revision\SlotRecord)
#3 /srv/mediawiki/php-master/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#4 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(226): MediaWiki\Revision\RevisionRecord->getContent(string, integer, NULL)
#5 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(193): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#6 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(142): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#7 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#8 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(197): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;958, MediaWiki\Revision\RenderedRevision, array)
#9 /srv/mediawiki/php-master/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#10 /srv/mediawiki/php-master/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#11 /srv/mediawiki/php-master/includes/page/Article.php(773): PoolCounterWork->execute()
#12 /srv/mediawiki/php-master/includes/actions/ViewAction.php(68): Article->view()
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(499): ViewAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(865): MediaWiki->performRequest()
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(515): MediaWiki->main()
#17 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#18 /srv/mediawiki/w/index.php(3): include(string)
#19 {main}
Reedy added a comment.EditedSun, May 19, 10:37 AM
MariaDB [zhwikivoyage]> select * from revision;
+--------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+---------------------------------+-------------------+--------------------+
| rev_id | rev_page | rev_text_id | rev_comment | rev_user | rev_user_text | rev_timestamp  | rev_minor_edit | rev_deleted | rev_len | rev_parent_id | rev_sha1                        | rev_content_model | rev_content_format |
+--------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+---------------------------------+-------------------+--------------------+
|      1 |        1 |           1 |             |        0 |               | 20190519102809 |              0 |           0 |    1337 |             0 | k6h2u286l68zxe442zja3k1ajjg0m2m | NULL              | NULL               |
|      2 |        2 |           2 |             |        0 |               | 20190519102810 |              0 |           0 |     108 |             0 | snzf2pvqgfabhypsdu7rumjosbo3wwo | NULL              | NULL               |
+--------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+---------------------------------+-------------------+--------------------+
2 rows in set (0.00 sec)

MariaDB [zhwikivoyage]> select * from text;
+--------+-----------------+---------------------+
| old_id | old_text        | old_flags           |
+--------+-----------------+---------------------+
|      1 | DB://cluster1/2 | utf-8,gzip,external |
|      2 | DB://cluster1/3 | utf-8,gzip,external |
+--------+-----------------+---------------------+
2 rows in set (0.00 sec)

MariaDB [zhwikivoyage]> select * from blobs1;
Empty set (0.00 sec)
Gilles moved this task from Inbox to Radar on the Performance-Team board.Mon, May 20, 8:31 PM
Gilles edited projects, added Performance-Team (Radar); removed Performance-Team.