Page MenuHomePhabricator

Slow load times for Special:Homepage on cswiki
Open, MediumPublic

Description

As @Gilles noted in T240201#6602256, there were some particularly slow load times recently for Special:Homepage.

Looking at the last two days there are some spikes:

Looking at Logstash for those spikes, we see a lot of entries like this:

 Expectation (readQueryTime <= 5) by MediaWiki::main not met (actual: 59.928746938705):
query-m: SELECT tl_from,tl_title FROM `templatelinks` WHERE (tl_from = N AND tl_title IN ('X') AND tl_namespace = N)  [TRX#09f194]

Where the trace looks like this:

#0 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/TransactionProfiler.php(254): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, double)
#1 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1412): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1294): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1223): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(137): Wikimedia\Rdbms\Database->select(string, array, string, string)
#6 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(57): GrowthExperiments\NewcomerTasks\TemplateFilter->buildResultsMap(GrowthExperiments\NewcomerTasks\Task\TaskSet)
#7 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(78): GrowthExperiments\NewcomerTasks\TemplateFilter->filter(GrowthExperiments\NewcomerTasks\Task\TaskSet)
#8 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/objectcache/wancache/WANObjectCache.php(1548): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->GrowthExperiments\NewcomerTasks\TaskSuggester\{closure}(GrowthExperiments\NewcomerTasks\Task\TaskSet, integer, array, double, array)
#9 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#10 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(102): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#11 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(323): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest(User, array, array)
#12 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(248): GrowthExperiments\HomepageModules\SuggestedEdits->getTaskSet()
#13 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(381): GrowthExperiments\HomepageModules\SuggestedEdits->getJsData(string)
#14 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(162): GrowthExperiments\Specials\SpecialHomepage->outputJsData(string, array)
#15 /srv/mediawiki/php-1.36.0-wmf.14/includes/specialpage/SpecialPage.php(600): GrowthExperiments\Specials\SpecialHomepage->execute(NULL)
#16 /srv/mediawiki/php-1.36.0-wmf.14/includes/specialpage/SpecialPageFactory.php(1018): SpecialPage->run(NULL)
#17 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#18 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(940): MediaWiki->performRequest()
#19 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(543): MediaWiki->main()
#20 /srv/mediawiki/php-1.36.0-wmf.14/index.php(53): MediaWiki->run()
#21 /srv/mediawiki/php-1.36.0-wmf.14/index.php(46): wfIndexMain()
#22 /srv/mediawiki/w/index.php(3): require(string)
#23 {main}

It looks like almost all of these are happening on cswiki.

The configuration at MediaWiki:NewcomerTasks.json has not changed recently.

Event Timeline

kostajh created this task.Wed, Nov 4, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Nov 4, 2:11 PM

We had problems with this query in the past (see T264029) but this seems like a different issue.

It looks like this happened with wmf.13 as well (logs), again only on cswiki.

Aklapper renamed this task from Slow load times for Special:Hompeage on cswiki to Slow load times for Special:Homepage on cswiki.Wed, Nov 4, 2:35 PM

I grepped DBPerformance logs for issues with templatelinks table:

[urbanecm@mwlog1001 /srv/mw-log]$ grep readQueryTime DBPerformance.log | grep -A 1 cswiki | json-from-logs.sh | jq -r .query | sed 's/\s*\[TRX#.*//g' | sort | uniq -c
      2 query-m: SELECT MASTER_GTID_WAIT('X', N)
    307 query-m: SELECT tl_from,tl_title FROM `templatelinks` WHERE (tl_from = N AND tl_title IN ('X') AND tl_namespace = N)
      1 query-m: SELECT user_id FROM `user` WHERE user_name = 'X' LIMIT N LOCK IN SHARE MODE
      5 query-m: SELECT user_id,user_name,user_real_name,user_email,user_touched,user_token,user_email_authenticated,user_email_token,user_email_token_expires,user_registration,user_editcount,user_actor.actor_id FROM `user` JOIN `actor` `user_actor` ON ((user_actor.actor_
[urbanecm@mwlog1001 /srv/mw-log]$ grep readQueryTime DBPerformance.log | grep -A 1 cswiki | json-from-logs.sh | jq -r .query | sed 's/\s*\[TRX#.*//g' | sort | uniq -c | sort -rn
    307 query-m: SELECT tl_from,tl_title FROM `templatelinks` WHERE (tl_from = N AND tl_title IN ('X') AND tl_namespace = N)
      5 query-m: SELECT user_id,user_name,user_real_name,user_email,user_touched,user_token,user_email_authenticated,user_email_token,user_email_token_expires,user_registration,user_editcount,user_actor.actor_id FROM `user` JOIN `actor` `user_actor` ON ((user_actor.actor_
      2 query-m: SELECT MASTER_GTID_WAIT('X', N)
      1 query-m: SELECT user_id FROM `user` WHERE user_name = 'X' LIMIT N LOCK IN SHARE MODE
[urbanecm@mwlog1001 /srv/mw-log]$

(json-from-logs is my wrapper for sed 's/\s*{/\t{/g' | cut -f 2 | grep '^{' I use to easily get the json metadata that are in the log data)

It seems like this is the most frequent cswiki issue with DBPerformance, and apparently it doesn't happen at any other wiki:

[urbanecm@mwlog1001 /srv/mw-log]$ grep readQueryTime DBPerformance.log | grep -B 1 templatelinks | grep mw | cut -d ' ' -f 5 | sort | uniq -c
    307 cswiki
[urbanecm@mwlog1001 /srv/mw-log]$

I originally suspected that s2 is having some issues, because the query looks simple enough (tl_from is a primary key that restricts the search to a single row, and a where at a single row should be pretty fast). I don't really see any inefficiency from the MediaWiki side

@Marostegui Do you have any idea why the query takes more than 5 seconds (sometimes)?

SELECT tl_from,tl_title FROM templatelinks` WHERE (tl_from = N AND tl_title IN ('X') AND tl_namespace = N)`

I cannot currently reproduce any issues with the Special:Homepage at cswiki, and this seems to not happen always.

Tgr added a subscriber: Tgr.Thu, Nov 5, 4:21 AM

These warnings are all followed by a MySQL server has gone away query error like this:

Error 2006 from GrowthExperiments\NewcomerTasks\TemplateFilter::buildResultsMap, MySQL server has gone away (10.64.0.99) SELECT  tl_from,tl_title  FROM `templatelinks`    WHERE (tl_from = 557178 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1566125 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 710826 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 371571 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 627294 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 507361 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 157039 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 9031 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 755400 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1198541 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 11697 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 169723 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 63752 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 814054 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 331903 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 21067 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 23300 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 235790 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 791841 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1030239 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 8272 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 16318 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1372514 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1375660 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 813470 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1369585 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 1109394 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 896834 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 607367 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 650068 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 755147 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 747961 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 4314 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Který?')  AND tl_namespace = 10) OR (tl_from = 632362 AND tl_title IN ('Kdy?','Kdo?','Pravopis','Sloh','Transkripce','Reklama','NPOV','Kým?','Jaký?','Kter 10.64.0.99

#0 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(137): Wikimedia\Rdbms\Database->select(string, array, string, string)
#4 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(57): GrowthExperiments\NewcomerTasks\TemplateFilter->buildResultsMap(GrowthExperiments\NewcomerTasks\Task\TaskSet)
#5 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(78): GrowthExperiments\NewcomerTasks\TemplateFilter->filter(GrowthExperiments\NewcomerTasks\Task\TaskSet)
#6 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/objectcache/wancache/WANObjectCache.php(1548): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->GrowthExperiments\NewcomerTasks\TaskSuggester\{closure}(GrowthExperiments\NewcomerTasks\Task\TaskSet, integer, array, double, array)
#7 /srv/mediawiki/php-1.36.0-wmf.14/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#8 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(102): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#9 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(323): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest(User, array, array)
#10 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(248): GrowthExperiments\HomepageModules\SuggestedEdits->getTaskSet()
#11 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(381): GrowthExperiments\HomepageModules\SuggestedEdits->getJsData(string)
#12 /srv/mediawiki/php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(162): GrowthExperiments\Specials\SpecialHomepage->outputJsData(string, array)
#13 /srv/mediawiki/php-1.36.0-wmf.14/includes/specialpage/SpecialPage.php(600): GrowthExperiments\Specials\SpecialHomepage->execute(NULL)
#14 /srv/mediawiki/php-1.36.0-wmf.14/includes/specialpage/SpecialPageFactory.php(1018): SpecialPage->run(NULL)
#15 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#16 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(940): MediaWiki->performRequest()
#17 /srv/mediawiki/php-1.36.0-wmf.14/includes/MediaWiki.php(543): MediaWiki->main()
#18 /srv/mediawiki/php-1.36.0-wmf.14/index.php(53): MediaWiki->run()
#19 /srv/mediawiki/php-1.36.0-wmf.14/index.php(46): wfIndexMain()
#20 /srv/mediawiki/w/index.php(3): require(string)
#21 {main}

(The query probably gets truncated by the log formatter.)
So these are huge queries, even if they align with the index very well. The expectation logger somehow messes up the query string.

tl_from and tl_namespace form a prefix of the primary key, and tl_title is the third and last element of the primary key, so basically the DB engine would walk down a B-tree for every parenthesized segment in that query and then search the bottom node for every quoted string. The query covers 250 cached queue items, so there are that many parentheses, times a dozen or so strings (template types). That doesn't seem like something that would take a minute, or even be particularly slow, but I'm not super confident about that. Then again, the error only shows up on cswiki.

Tgr added a comment.EditedThu, Nov 5, 4:23 AM

All of the errors come from db1129. I don't see anything out of the ordinary there:

tgr@mwmaint1002:~$ sql --wiki=cswiki --host=db1129
wikiadmin@10.64.0.99(cswiki)> show indexes from templatelinks;
+---------------+------------+------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table         | Non_unique | Key_name               | Seq_in_index | Column_name       | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| templatelinks |          0 | PRIMARY                |            1 | tl_from           | A         |     1614647 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          0 | PRIMARY                |            2 | tl_namespace      | A         |     2767967 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          0 | PRIMARY                |            3 | tl_title          | A         |    19375772 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_namespace           |            1 | tl_namespace      | A         |        1956 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_namespace           |            2 | tl_title          | A         |      248407 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_namespace           |            3 | tl_from           | A         |    19375776 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_backlinks_namespace |            1 | tl_from_namespace | A         |        3505 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_backlinks_namespace |            2 | tl_namespace      | A         |        8767 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_backlinks_namespace |            3 | tl_title          | A         |      472579 |     NULL | NULL   |      | BTREE      |         |               |
| templatelinks |          1 | tl_backlinks_namespace |            4 | tl_from           | A         |    19375776 |     NULL | NULL   |      | BTREE      |         |               |
+---------------+------------+------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Tgr added a comment.Thu, Nov 5, 4:30 AM

Running the query from the log (which is truncated at 33 terms instead of the full 250, so, about 12% of the full query):

wikiadmin@10.64.0.99(cswiki)> SELECT ...
37 rows in set (0.02 sec)

wikiadmin@10.64.0.99(cswiki)> EXPLAIN SELECT ...
+------+-------------+---------------+-------+----------------------+--------------+---------+------+------+--------------------------+
| id   | select_type | table         | type  | possible_keys        | key          | key_len | ref  | rows | Extra                    |
+------+-------------+---------------+-------+----------------------+--------------+---------+------+------+--------------------------+
|    1 | SIMPLE      | templatelinks | range | PRIMARY,tl_namespace | tl_namespace | 265     | NULL |  330 | Using where; Using index |
+------+-------------+---------------+-------+----------------------+--------------+---------+------+------+--------------------------+

As Kosta pointed out, the timeout only happens for a small fraction of the queries. Maybe it's waiting on a write lock? But then we'd see another performance warning about some transaction taking long, and I couldn't find any.

Tgr added a comment.Thu, Nov 5, 7:05 AM

Dashboard. It is a fairly rare error (600 times in the last 30 days), cswiki-only, not specific to wmf.14, happening in clusters. Caused by some bot mass-editing pages and piling up write locks on the templatelinks table, maybe? I'm fairly sure our code is the victim and not the cause here, but we'll probably need DBA help to move further.

Thanks for the ping!

These warnings are all followed by a MySQL server has gone away query error like this:

This is because of the query killer kicking in and killing the query, as it takes more than 60 seconds to run.

server_id: 171966563
    stamp: 2020-11-04 15:36:33
    event: wmf_slave_wikiuser_slow (>60)
  content: kill 613947490; SELECT /* GrowthExperiments\NewcomerTasks\TemplateFilter::buildResultsMap  */  tl_from,tl_title  FRO

As @Tgr points out, the entire query is:

1SELECT /* GrowthExperiments\NewcomerTasks\TemplateFilter::buildResultsMap */ tl_from, tl_title FROM `templatelinks` WHERE (tl_from = 670356 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1066533 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1333958 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 921514 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 12783 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 748776 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 690478 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 157039 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 90204 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1196789 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 271933 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1375660 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 331903 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 747961 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1440173 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 707253 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 550807 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 288512 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 203718 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1595758 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1245298 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 438469 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1030239 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 518073 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 956554 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 507361 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 816340 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1179926 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 814624 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1172456 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1035821 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 232592 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1045914 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 459339 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 983838 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 682871 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1253950 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 30787 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 23300 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 557178 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 791260 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 277735 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 227230 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 921950 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 25878 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1118627 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 807129 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 781716 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1302914 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 861117 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 122510 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1388049 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 801948 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1375746 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 169723 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 710154 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 702439 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1386362 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 344849 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 93502 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 641133 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1029555 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 779117 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 601169 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 755147 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1514640 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1620517 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 702168 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1611698 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 802149 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1237797 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1531803 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1113030 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 768350 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 253643 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1103800 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 832574 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 299949 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 80890 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 691232 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 152755 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 731919 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1494837 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 586023 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 390344 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1188404 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 113159 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 579102 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1043245 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 180626 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 710826 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 439719 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 200731 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 830401 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 945893 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 515696 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 918749 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 21067 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 438998 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 80098 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 138897 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 8272 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 318749 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1236844 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1134916 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 15088 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 466685 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1565863 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1123858 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 250116 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 28686 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 629725 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1609685 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 634169 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 956726 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 949043 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 632362 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 966425 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 814054 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1612862 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1540561 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 196737 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 175254 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 913787 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1135215 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 241924 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 629397 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1542646 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 766151 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1135390 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 204710 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 183687 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 999035 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 809691 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 742446 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 944845 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1342621 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 457655 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 753327 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1363995 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 305495 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1542647 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 104913 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1154727 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 235790 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 791841 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 310893 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1051504 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1271159 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1488881 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 232191 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 16318 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1109394 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 650068 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 63752 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 813470 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1167598 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1157226 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 34557 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 22835 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1372739 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 891298 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 445327 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 749957 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 941898 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 764652 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 595544 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 453971 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 11697 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1198541 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 997363 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 34401 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 85969 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 52956 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 755400 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1566125 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 102558 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 975562 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1405073 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1111200 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1369585 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1357995 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 863729 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 4314 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 788955 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1461470 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 911695 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 930917 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1047541 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1306052 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 883852 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1528750 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 645080 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 278423 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1340542 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 476450 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 605941 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 660824 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 159680 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 627294 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1618517 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 595723 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 894889 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 116576 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1372514 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 910719 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 967656 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1007952 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1258410 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 241556 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 491969 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 896834 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 688595 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 83293 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1093214 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 901500 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1031276 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1393710 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 884339 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 26378 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 499612 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 63775 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 138320 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1125851 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 905006 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1207008 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 949524 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 607367 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 987 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 838783 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 338267 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1347851 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 800778 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 987336 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 63085 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 9031 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 712551 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1283491 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 755517 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1277225 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 10653 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 19745 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 371571 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 258302 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 514477 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 1194865 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 788769 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) OR (tl_from = 795497 AND tl_title IN ('Kdy?', 'Kdo?', 'Pravopis', 'Sloh', 'Transkripce', 'Reklama', 'NPOV', 'Kým?', 'Jaký?', 'Který?') AND tl_namespace = 10) /* 4add481ac8c14a02df35cda52595f328 db1129 cswiki 76s */

That query's explain isn't too bad but the query itself takes more than a minute, and hence gets killed by the query killer:

root@db1129.eqiad.wmnet[cswiki]> show explain for 659580894;
+------+-------------+---------------+------+----------------------+--------------+---------+-------+------+--------------------------+
| id   | select_type | table         | type | possible_keys        | key          | key_len | ref   | rows | Extra                    |
+------+-------------+---------------+------+----------------------+--------------+---------+-------+------+--------------------------+
|    1 | SIMPLE      | templatelinks | ref  | PRIMARY,tl_namespace | tl_namespace | 4       | const | 9902 | Using where; Using index |
+------+-------------+---------------+------+----------------------+--------------+---------+-------+------+--------------------------+
1 row in set, 1 warning (0.001 sec)

It is hard to debug such a long query, but from what I can see on the handlers:

Handler_read_next	12759822
Handler_read_next

The number of requests to read the next row in key order. This value is incremented if you are querying an index column with a range constraint or if you are doing an index scan.

Unfortunately, this host is running 10.1, so I cannot enable the optimizer trace there. But I have tried on db1090:3312 (which runs 10.4, and looks like it has the same issue - and the same query plan).
The optimizer trace shows some interesting results:

{
  "considered_execution_plans": [
    {
      "plan_prefix": [],
      "table": "templatelinks",
      "best_access_path": {
        "considered_access_paths": [
          {
            "access_type": "ref",
            "index": "tl_namespace",
            "rows": 10039,
            "cost": 692.95,
            "chosen": true
          },
          {
            "type": "scan",
            "chosen": false,
            "cause": "cost"
          }
        ],
        "chosen_access_method": {
          "type": "ref",
          "records": 10039,
          "cost": 692.95,
          "uses_join_buffering": false
        }
      },
      "rows_for_plan": 10039,
      "cost_for_plan": 2700.7,
      "estimated_join_cardinality": 10039

So the optimizer goes for tl_namespace than going for PRIMARY which looks cheaper here (even if it does a full table scan):

"rows_estimation": [
  {
    "table": "templatelinks",
    "range_analysis": {
      "table_scan": {
        "rows": 19276949,
        "cost": 3.95e6

And this is true if we try a `USE (PRIMARY) on the query shows way better results:

265 rows in set (0.146 sec)

And this is NOT doing USE (PRIMARY) and just leaving the query choosing whatever it prefers (tl_namespace);

265 rows in set (2 min 9.220 sec)

And the explain with the USE is indeed better:

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	templatelinks	range	PRIMARY	PRIMARY	265	NULL	2480	Using where; Using index

This might be an mariadb-optimizer-bug so I am going to report this to mariadb, given that the data on this table is public, I can provide all the queries (this is not the first time we see issues with the optimizer doing silly things)
https://jira.mariadb.org/browse/MDEV-21794
https://jira.mariadb.org/browse/MDEV-21813
https://jira.mariadb.org/browse/MDEV-19579

I will first try to re-generate the table stats to see if the optimizer does something differently.

Mentioned in SAL (#wikimedia-operations) [2020-11-05T07:23:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 T267216', diff saved to https://phabricator.wikimedia.org/P13201 and previous config saved to /var/cache/conftool/dbconfig/20201105-072352-marostegui.json

So interestingly, on db1129 (10.1) the analyze table results on a much realistic explain, which can explain the long query times and why using PK is way faster:

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	templatelinks	ref	PRIMARY,tl_namespace	tl_namespace	4	const	1294531	Using where; Using index

Going to try the same on 10.4

Same thing tried on 10.4, after the analyze, now the original query shows:

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	templatelinks	ref	PRIMARY,tl_namespace	tl_namespace	4	const	1294541	Using where; Using index

And with the hint:

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	templatelinks	range	PRIMARY	PRIMARY	265	NULL	2480	Using where; Using index

And yet the optimizer keeps choosing the wrong index.

mysql:root@localhost [cswiki]> show explain for 73439158;
+------+-------------+---------------+------+----------------------+--------------+---------+-------+---------+--------------------------+
| id   | select_type | table         | type | possible_keys        | key          | key_len | ref   | rows    | Extra                    |
+------+-------------+---------------+------+----------------------+--------------+---------+-------+---------+--------------------------+
|    1 | SIMPLE      | templatelinks | ref  | PRIMARY,tl_namespace | tl_namespace | 4       | const | 1294541 | Using where; Using index |
+------+-------------+---------------+------+----------------------+--------------+---------+-------+---------+--------------------------+
1 row in set, 1 warning (0.000 sec)

Going to file a bug report to mariadb.

Marostegui triaged this task as Medium priority.Thu, Nov 5, 9:28 AM
Marostegui added a project: DBA.
Marostegui moved this task from Triage to Blocked on the DBA board.
Tgr added a comment.Thu, Nov 5, 9:40 AM

I'm still confused why this is only happening on cswiki (the DB server is not actually limited to db1129, that was my mistake).

Given that it looks like an optimizer bug, it could be just that particular set of values for that specific query.

Change 639466 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Force primary index in TemplateFilter

https://gerrit.wikimedia.org/r/639466

Tgr claimed this task.Thu, Nov 5, 7:49 PM

I'm moving this into the QA column for us to keep an eye on over the next couple weeks. Alternatively or in addition to manaul monitoring, as was noted in T240201#6605879, we should set up some alerts in Grafana so we can find out about these issues in an automated way.

Tgr added a comment.Tue, Nov 10, 10:31 PM

Not much point in monitoring if we don't fix it. IMO this should go back to Needs more work. Or do you mean that it should be fine to ignore the issue if it's happening infrequently enough? (It seems to be happening about 20 times a day on cswiki. Not sure what fraction of all homepage views that is.)

@kostajh -- I noticed that @Tgr's question is still here, and I wanted to bring it back up.

Not much point in monitoring if we don't fix it. IMO this should go back to Needs more work. Or do you mean that it should be fine to ignore the issue if it's happening infrequently enough? (It seems to be happening about 20 times a day on cswiki. Not sure what fraction of all homepage views that is.)

I see two related issues:

  • the mariadb-optimizer-bug, which I propose we do not do anything about as it should be fixed upstream
  • the fact that sometimes the grafana charts for suggested edits spikes, and we don't know unless we look manually

I do think we should set up alerting as that will cover much more than just the odd mariadb bug.

For wmf.18 there are quite few similar (at least they look similar to me) issues - T268679: [wmf.18] MentorPageMentorManager::setMentorForUser: Expectation (masterConns <= 0) by MediaWiki::main not met - GrowthExperiments filed .

The last 24 hours look good though:

Tgr added a comment.Wed, Nov 25, 1:04 AM

Filed T268700: Investigate setting up alerts for Growth dashboards about the alerts.

  • the mariadb-optimizer-bug, which I propose we do not do anything about as it should be fixed upstream

I don't think waiting for upstream is a feasible strategy, we are way too far behind (10.5 was released a year ago; we are on a mix of 10.1 and 10.4). We can hope upgrading to 10.4 fixes it, which seemed not to be the case. Otherwise, working around optimizer hiccups is a fairly common thing to do.

We can either choose to not fix it, test if hinting the primary index causes issues elsewhere, put the hint behind some kind of feature flag (that seems rather ugly), or try a different query or break it into smaller queries.

I agree with @Tgr - waiting for MariaDB to fix this might be a long shot. We also don't know whether:

  • They would fix it
  • Fix it and ship it on 10.4
  • Fix it and ship it only on 10.5

From my point of view, working around this in code can be risky, as it is just one query affecting a very concrete wiki. The risk of producing unexpected results can be high. I would prefer to go for the last approach: breaking this query into smaller ones.