Just prior to paging event:
22:31:35 <+jinxer-wm> (FrontendUnavailable) firing: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DFrontendUnavailable 22:31:35 <+jinxer-wm> (FrontendUnavailable) firing: HAProxy (cache_text) has reduced HTTP availability #page - TODO - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DFrontendUnavailable 22:32:03 <+jinxer-wm> (ProbeDown) firing: (14) Service centrallog1001:6514 has failed probes (tcp_rsyslog_receiver_ip6) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
Error
- mwversion: 1.40.0-wmf.12
- reqId: 2d74f42d-0c72-4742-be98-a7627edddc94
- Find reqId in Logstash
normalized_message
[{reqId}] {exception_url} RuntimeException: Could not acquire lock for page ID '44623777'.exception.trace
from /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/LinksUpdate/LinksUpdate.php(244)
#0 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(474): MediaWiki\Deferred\LinksUpdate\LinksUpdate->doUpdate()
#1 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MediaWiki\Deferred\LinksUpdate\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#2 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(474): RefreshSecondaryDataUpdate->doUpdate()
#3 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(399): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#4 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, MediaWiki\JobQueue\JobQueueGroupFactory, string)
#5 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdatesScope.php(264): DeferredUpdates::{closure}(RefreshSecondaryDataUpdate, integer)
#6 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#7 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(235): DeferredUpdatesScope->processUpdates(integer, Closure)
#8 /srv/mediawiki/php-1.40.0-wmf.12/includes/MediaWiki.php(1102): DeferredUpdates::doUpdates()
#9 /srv/mediawiki/php-1.40.0-wmf.12/includes/MediaWiki.php(836): MediaWiki->restInPeace()
#10 /srv/mediawiki/php-1.40.0-wmf.12/api.php(125): MediaWiki->doPostOutputShutdown()
#11 /srv/mediawiki/php-1.40.0-wmf.12/api.php(45): wfApiMain()
#12 /srv/mediawiki/w/api.php(3): require(string)
#13 {main}Impact
Notes
Error
- mwversion: 1.40.0-wmf.12
- reqId: a89c7917-24a0-43cb-af9c-f6d4bff93ea7
- Find reqId in Logstash
normalized_message
[{reqId}] {exception_url} Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction
Function: Wikibase\Lib\Store\Sql\Terms\DatabaseTermStoreWriterBase::acquireAndInsertTerms
Query: SELECT wbit_term_in_lang_id AS `exception.trace
from /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(1555)
#0 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(1539): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(1513): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(893): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(1647): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/Database.php(1623): Wikimedia\Rdbms\Database->select(string, array, array, string, array, array)
#5 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->selectFieldValues(string, string, array, string, array)
#6 /srv/mediawiki/php-1.40.0-wmf.12/includes/libs/rdbms/database/DBConnRef.php(331): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.40.0-wmf.12/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermStoreWriterBase.php(114): Wikimedia\Rdbms\DBConnRef->selectFieldValues(string, string, array, string, array)
#8 /srv/mediawiki/php-1.40.0-wmf.12/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermStoreWriterBase.php(62): Wikibase\Lib\Store\Sql\Terms\DatabaseTermStoreWriterBase->acquireAndInsertTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#9 /srv/mediawiki/php-1.40.0-wmf.12/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php(23): Wikibase\Lib\Store\Sql\Terms\DatabaseTermStoreWriterBase->store(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#10 /srv/mediawiki/php-1.40.0-wmf.12/extensions/Wikibase/lib/includes/Store/ItemTermStoreWriterAdapter.php(32): Wikibase\Lib\Store\Sql\Terms\DatabaseItemTermStoreWriter->storeTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#11 /srv/mediawiki/php-1.40.0-wmf.12/extensions/Wikibase/repo/includes/Content/DataUpdateAdapter.php(60): Wikibase\Lib\Store\ItemTermStoreWriterAdapter->saveTermsOfEntity(Wikibase\DataModel\Entity\Item)
#12 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(474): Wikibase\Repo\Content\DataUpdateAdapter->doUpdate()
#13 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(Wikibase\Repo\Content\DataUpdateAdapter, Wikimedia\Rdbms\LBFactoryMulti)
#14 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(474): RefreshSecondaryDataUpdate->doUpdate()
#15 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(399): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#16 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::run(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, MediaWiki\JobQueue\JobQueueGroupFactory, string)
#17 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdatesScope.php(264): DeferredUpdates::{closure}(RefreshSecondaryDataUpdate, integer)
#18 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#19 /srv/mediawiki/php-1.40.0-wmf.12/includes/deferred/DeferredUpdates.php(235): DeferredUpdatesScope->processUpdates(integer, Closure)
#20 /srv/mediawiki/php-1.40.0-wmf.12/includes/MediaWiki.php(1102): DeferredUpdates::doUpdates()
#21 /srv/mediawiki/php-1.40.0-wmf.12/includes/MediaWiki.php(836): MediaWiki->restInPeace()
#22 /srv/mediawiki/php-1.40.0-wmf.12/api.php(125): MediaWiki->doPostOutputShutdown()
#23 /srv/mediawiki/php-1.40.0-wmf.12/api.php(45): wfApiMain()
#24 /srv/mediawiki/w/api.php(3): require(string)
#25 {main}