Page MenuHomePhabricator

Creating MySQL tables for MediaWiki sometimes stalled on I/O for several minutes
Closed, ResolvedPublic

Description

It tends to be under 5 seconds or between 2 to 5 minutes. I couldn't find a sample of one longer than 3 minutes, but I've definitely seen them.

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6376/consoleFull
(duration: 2s)

00:00:12.249 You can install MediaWiki.
00:00:12.251 Setting up database
00:00:12.355 done
00:00:12.356 Creating tables
00:00:14.843 done

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6389/consoleFull
(duration: 1m17s)

00:00:19.584 You can install MediaWiki.
00:00:19.585 Setting up database
00:00:19.622 done
00:00:19.622 Creating tables
00:01:36.238 done

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6390/consoleFull
(duration: 2m46s)

00:00:24.617 You can install MediaWiki.
00:00:24.618 Setting up database
00:00:24.659 done
00:00:24.661 Creating tables
00:01:01.998 [Thu Apr 16 03:08:28 2015] [hphp] [27994:7f86ecc1abc0:0:000001] [] SlowTimer [11043ms] at runtime/ext_mysql: slow query: CREATE /* DatabaseBase::sourceFile( /mnt/jenkins-workspace/workspace/mediawiki-phpunit-hhvm@2/src/maintenance/tables.sql ) 127.0.0.1 */ INDEX page_timestamp ON `revision` (rev_page,rev_timestamp)
00:01:01.998 
00:01:14.379 [Thu Apr 16 03:08:41 2015] [hphp] [27994:7f86ecc1abc0:0:000002] [] SlowTimer [12380ms] at runtime/ext_mysql: slow query: CREATE /* DatabaseBase::sourceFile( /mnt/jenkins-workspace/workspace/mediawiki-phpunit-hhvm@2/src/maintenance/tables.sql ) 127.0.0.1 */ INDEX user_timestamp ON `revision` (rev_user,rev_timestamp)
00:01:14.379 
00:03:10.550 done

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6392/consoleFull
(duration: 1s)

00:00:07.482 You can install MediaWiki.
00:00:07.483 Setting up database
00:00:07.516 done
00:00:07.516 Creating tables
00:00:08.961 done

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6395/consoleFull
(duration: 36s)

00:00:14.645 You can install MediaWiki.
00:00:14.645 Setting up database
00:00:14.683 done
00:00:14.684 Creating tables
00:00:50.095 done

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/6402/consoleFull
(duration: 2s)

00:00:12.024 You can install MediaWiki.
00:00:12.026 Setting up database
00:00:12.068 done
00:00:12.069 Creating tables
00:00:14.399 done

Event Timeline

Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle subscribed.
Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
Krinkle claimed this task.

I haven't seen any jobs lock for more than 1 minute since T96230 got resolved. It still sometimes takes upto 30 seconds to create the database tables (as opposed to the median 5 seconds), but that seems acceptable for the time being.