5-10 times more rows are currently read from the s5 master than from all other masters together
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Sep 24 2016, 9:23 AM

Project Tags

Referenced Files

	F4519503: Screenshot from 2016-09-24 11-12-03.png
	Sep 24 2016, 9:23 AM

Subscribers

Description

https://grafana-admin.wikimedia.org/dashboard/db/mysql-aggregated?from=1474704583828&to=1474708183828&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=s1&var-shard=s2&var-shard=s3&var-shard=s4&var-shard=s5&var-shard=s6&var-shard=s7&var-role=master

This is not only a SPOF and a problem for latency for both wikidata and dewiki, it is also a huge blocker for cross-dc reads, as application servers from the remote datacenter won't be able to read from the master except on very rare and specific occasions (I am adding Performance-Team because of this).

This is not the first time it happens, but there is a large increase with constant high reads since 2016-09-24 2:37:42 UTC.

I do not know the cause for this yet- adding Wikidata because it is either that or dewiki having a really bad pattern, needs research.

Related Objects

Mentioned In: T146576: 502 Bad Gateway errors while trying to run simple queries with the Wikidata Query Service
Mentioned Here: rEWBA37a2c4327357: Don't fallback to master in WikiPageEntityMetaDataLookup etc.

Event Timeline

jcrespo created this task.Sep 24 2016, 9:23 AM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptSep 24 2016, 9:23 AM

Also seeing:

Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met:
[connect to 10.64.16.144 (wikidatawiki)]
#0 /srv/mediawiki/php-1.28.0-wmf.20/includes/libs/rdbms/TransactionProfiler.php(156): TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.28.0-wmf.20/includes/libs/rdbms/loadbalancer/LoadBalancer.php(573): TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.28.0-wmf.20/includes/dao/DBAccessBase.php(61): LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(175): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(76): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(199): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(164): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch()
#7 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(84): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#8 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(156): Wikibase\Lib\Store\WikiPageEntityRevisionLookup->getEntityRevision()
#9 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(138): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision()
#10 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(156): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision()
#11 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(138): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision()
#12 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(255): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision()
#13 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(237): Wikibase\Repo\Api\GetEntities->getEntityRevision()
#14 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(129): Wikibase\Repo\Api\GetEntities->getEntityRevisionsFromEntityIds()
#15 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(1427): Wikibase\Repo\Api\GetEntities->execute()
#16 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(511): ApiMain->executeAction()
#17 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(482): ApiMain->executeActionWithErrorHandling()
#18 /srv/mediawiki/php-1.28.0-wmf.20/api.php(83): ApiMain->execute()
#19 /srv/mediawiki/w/api.php(3): include()
#20 {main}

It only stopped from 19:24:45 to 20:00:45 on the 24th.

https://tendril.wikimedia.org/report/slow_queries_checksum?checksum=31bdf25cc42f95f4f4a9b493cafe0299&host=db1049&user=&schema=&hours=24

I see an unusual amount of GET_LOCK('LinksUpdate:job:pageid:28776419', 15), on s5-master mostly for linkupdates and category memberships. Which is strange because I would only expect that if there was lag (or job contention).

Esc3300 mentioned this in T146576: 502 Bad Gateway errors while trying to run simple queries with the Wikidata Query Service.Sep 25 2016, 11:52 AM

This stopped suddenly at 08:20:42 on the 25th and restarted at 19:44:29. This is most likely the result of a user script.

However, whatever is done, it should not read from the master.

In T146529#2664889, @aaron wrote:

Also seeing:
…

That has been fixed with 37a2c43273571af41ea38ad39c84f7cdde869de0 which has not been deployed, yet (the wmf/1.28.0-wmf.18 branch of Wikibase is still used).

Addshore moved this task from incoming to monitoring on the Wikidata board.Sep 26 2016, 11:07 AM

This stopped happening 2016-09-27 ~07:00.

5-10 times more rows are currently read from the s5 master than from all other masters togetherClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

5-10 times more rows are currently read from the s5 master than from all other masters together
Closed, ResolvedPublic
Actions