Page MenuHomePhabricator
Create Task
Maniphest T161520

HHVM 3.18 crashes when Cirrus tries to fetch another wiki config via maint script
Closed, ResolvedPublic
Actions

Description

After repooling mw1261 with the fix for T161095 being deployed, it was crashing 2-3 times per minute, the backtraces all look like this:

#0  0x00007fc58e3a1a20 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007fc58cc07316 in seekAndWriteFd (fd=16, iOff=iOff@entry=536870912, pBuf=pBuf@entry=0x7fc57a0c3408, nBuf=nBuf@entry=1024, piErrno=piErrno@entry=0x7fc581460220)
    at sqlite3.c:28070
#2  0x00007fc58cc07422 in seekAndWrite (cnt=1024, pBuf=0x7fc57a0c3408, offset=536870912, id=0x7fc581460200) at sqlite3.c:28090
#3  unixWrite (id=0x7fc581460200, pBuf=0x7fc57a0c3408, amt=1024, offset=536870912) at sqlite3.c:28157
#4  0x00007fc58cc2a8f1 in sqlite3OsWrite (offset=<optimized out>, amt=<optimized out>, pBuf=<optimized out>, id=<optimized out>) at sqlite3.c:15974
#5  pager_write_pagelist (pPager=pPager@entry=0x7fc581460088, pList=pList@entry=0x7fc57a0c3840) at sqlite3.c:45130
#6  0x00007fc58cc2ab4b in pagerStress (p=0x7fc581460088, pPg=0x7fc57a0c3840) at sqlite3.c:45339
#7  0x00007fc58cc278cd in sqlite3PcacheFetchStress (ppPage=<synthetic pointer>, pgno=524301, pCache=0x7fc5814601b0) at sqlite3.c:38815
#8  sqlite3PagerAcquire (pPager=0x7fc581460088, pgno=pgno@entry=524301, ppPage=ppPage@entry=0x7ffd57a94890, flags=flags@entry=1) at sqlite3.c:46206
#9  0x00007fc58cc27a3b in btreeGetPage (pBt=pBt@entry=0x7fc581797a68, pgno=524301, ppPage=ppPage@entry=0x7ffd57a949d0, flags=flags@entry=1) at sqlite3.c:53847
#10 0x00007fc58cc2c58c in allocateBtreePage (pBt=0x7fc581797a68, ppPage=0x7ffd57a949d0, pPgno=0x7ffd57a949cc, nearby=524300, eMode=0 '\000') at sqlite3.c:57564
#11 0x00007fc58cc470f9 in fillInCell (pnSize=<optimized out>, nZero=<optimized out>, nData=<optimized out>, pData=<optimized out>, nKey=<optimized out>,
    pKey=<optimized out>, pCell=<optimized out>, pPage=<optimized out>) at sqlite3.c:57924
#12 sqlite3BtreeInsert (pCur=0x10, pKey=0x0, nKey=524300, pData=0x7fc58e3a1a20 <__write_nocancel+7>, nData=11220, nZero=11220, appendBias=1, seekResult=-27)
    at sqlite3.c:59364
#13 0x00007fc58cc6d5ec in sqlite3VdbeExec (p=0x10) at sqlite3.c:73151
#14 0x00007fc58cc75ec7 in sqlite3Step (p=<optimized out>) at sqlite3.c:67780
#15 sqlite3_step (pStmt=0x7fc579dd5fc8) at sqlite3.c:2310
#16 0x0000000001190895 in HPHP::RepoQuery::step (this=this@entry=0x7ffd57a94ea0) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:320
#17 0x0000000001190b15 in exec (this=0x7ffd57a94ea0) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:351
#18 operator() (__closure=<synthetic pointer>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:188
#19 HPHP::RepoTxn::rollback_guard<HPHP::RepoTxn::exec(HPHP::RepoQuery&)::<lambda()> >(HPHP::RepoTxn::<lambda()>, const char *) (this=0x7ffd57a951f0, f=..., f@entry=...,
    func=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:148
#20 0x0000000001190b80 in exec (query=..., this=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:190
#21 HPHP::RepoTxnQuery::exec (this=this@entry=0x7ffd57a94ea0) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo-helpers.cpp:527
#22 0x0000000001270e61 in HPHP::UnitRepoProxy::InsertUnitArrayStmt::insert (this=this@entry=0x7fc5814dcc00, txn=..., unitSn=unitSn@entry=581, arrayId=arrayId@entry=0,
    array="a:902:{s:6:\"aawiki\";s:17:\"php-1.29.0-wmf.17\";s:11:\"aawikibooks\";s:17:\"php-1.29.0-wmf.17\";s:12:\"aawiktionary\";s:17:\"php-1.29.0-wmf.17\";s:6:\"abwiki\";s:17:\"php-1.29.0-wmf.17\";s:12:\"abwiktionary\";s:17:\"ph"...) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/unit-emitter.cpp:982
#23 0x000000000127a2f3 in HPHP::UnitEmitter::insert (this=this@entry=0x7fc57a184800, unitOrigin=unitOrigin@entry=HPHP::File, txn=...)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/unit-emitter.cpp:451
#24 0x000000000127a6f0 in HPHP::UnitEmitter::commit (this=this@entry=0x7fc57a184800, unitOrigin=unitOrigin@entry=HPHP::File)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/unit-emitter.cpp:412
#25 0x0000000001285803 in HPHP::Repo::commitUnit (this=<optimized out>, ue=0x7fc57a184800, unitOrigin=HPHP::File)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/repo.cpp:504
#26 0x0000000000bea9e6 in HPHP::Compiler::hphp_compiler_parse (code=<optimized out>, codeLen=<optimized out>, md5=..., filename=<optimized out>,
    releaseUnit=0x7ffd57a96230) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/compiler/analysis/emitter.cpp:11571
#27 0x0000000000eb3374 in HPHP::(anonymous namespace)::createUnitFromString (path=0x7fc5799867d8 "/srv/mediawiki/wikiversions.php",
    contents=(HPHP::req::ptr<HPHP::StringData>) 0x7fc5815f5400 "<?php
return array(
    "aawiki" => "php-1.29.0-wmf.17",
    "aawikibooks" => "php-1.29.0-wmf.17",

    (...)

    "zhwiktionary" => "php-1.29.0-wmf.17",
    "zuwiki" => "php-1.29.0-wmf.17",
", releaseUnit=releaseUnit@entry=0x7ffd57a96230) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:214
#28 0x00000000021663b6 in createUnitFromFile (releaseUnit=0x7ffd57a96230, path=0x7fc5799867c0) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:232
#29 operator() (__closure=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:293
#30 loadUnitNonRepoAuth (statInfo=..., requestedPath=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:305
#31 lookupUnitNonRepoAuth (statInfo=..., requestedPath=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:335
#32 checkoutFile (statInfo=..., path=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:423
#33 HPHP::lookupUnit (path=0x7fc58159c000, currentDir=<optimized out>, initial_opt=initial_opt@entry=0x7ffd57a96710)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/unit-cache.cpp:525
#34 0x00000000012a0a02 in HPHP::inclOp(unsigned char const*&, HPHP::InclOpFlags, char const*)::{lambda()#2}::operator()() const (__closure=0x7ffd57a968b0)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:5165
#35 0x00000000012a0a02 in iopInclOnce (pc=<optimized out>)
#36 iopWrapper (pc=<optimized out>, fn=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:6226
#37 HPHP::inclOp(unsigned char const*&, HPHP::InclOpFlags, char const*)::{lambda()#2}::operator()() const ()
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:6782
#38 0x00000000021c95d3 in inclOp (opName=<optimized out>, flags=HPHP::Default, pc=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:5166
#39 iopIncl (pc=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:5188
#40 iopWrapper (pc=<optimized out>, fn=<optimized out>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:6226
#41 HPHP::dispatchImpl<false> () at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:6905
#42 0x00000000012b48a3 in dispatch () at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:6922
#43 HPHP::enterVMAtFunc (enterFnAr=enterFnAr@entry=0x7fc579f3ffc0, stk=<optimized out>, varEnv=varEnv@entry=0x7fc58159bad0)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/bytecode.cpp:1512
#44 0x0000000000f44c89 in operator() (__closure=<synthetic pointer>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1650
#45 exception_handler<HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::ActRec*)>::<lambda()> > (action=...) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/vm/unwind-inl.h:30
#46 operator() (__closure=<synthetic pointer>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1580
#47 enterVMCustomHandler<HPHP::enterVM(HPHP::ActRec*, Action) [with Action = HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::ActRec*)>::<lambda()>]::<lambda()> > (action=...,
    ar=0x7fc579f3ffc0) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1571
#48 enterVM<HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::ActRec*)>::<lambda()> > (action=..., ar=0x7fc579f3ffc0)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1580
#49 operator() (ar=0x7fc579f3ffc0, __closure=<synthetic pointer>) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1651
#50 invokeFuncImpl<HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::TypedValue&)>, HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::ActRec*, HPHP::TypedValue&)>, HPHP::ExecutionContext::invokeFunc(const HPHP::Func*, const HPHP::Variant&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags, bool)::<lambda(HPHP::ActRec*)> >
    (doEnterVM=..., doInitArgs=..., doStackCheck=..., useWeakTypes=false, invName=0x0, argc=0, cls=0x0, thiz=0x7fc579961580, f=0x0, this=0x7fc581596010)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1542
#51 HPHP::ExecutionContext::invokeFunc (this=this@entry=0x7fc581596010, f=0x0, args_=..., thiz=0x7fc579961580, thiz@entry=0x0, cls=cls@entry=0x0, varEnv=0x7fc58159bad0,
    invName=0x0, flags=HPHP::ExecutionContext::InvokePseudoMain, useWeakTypes=false) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1655
#52 0x0000000000f45205 in HPHP::ExecutionContext::invokeUnit (this=0x7fc581596010, unit=0x7fc579968a80)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/execution-context.cpp:1272
#53 0x00000000021a3b0c in invoke_file_impl (currentDir=0x21f80b8 "", once=true,
    path=(HPHP::req::ptr<HPHP::StringData>) 0x7fc5815a3940 "/srv/mediawiki/multiversion/MWScript.php", res={ HPHP::KindOfUninit })
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/builtin-functions.cpp:909
#54 HPHP::invoke_file (s=(HPHP::req::ptr<HPHP::StringData>) 0x7fc5815a3940 "/srv/mediawiki/multiversion/MWScript.php", once=once@entry=true,
    currentDir=currentDir@entry=0x21f80b8 "") at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/builtin-functions.cpp:922
#55 0x00000000021a3e49 in HPHP::include_impl_invoke (file=(HPHP::req::ptr<HPHP::StringData>) 0x7fc5815a3940 "/srv/mediawiki/multiversion/MWScript.php",
    once=once@entry=true, currentDir=currentDir@entry=0x21f80b8 "") at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/builtin-functions.cpp:933
#56 0x00000000021a2a63 in HPHP::hphp_invoke (context=0x7fc581596010, cmd="/srv/mediawiki/multiversion/MWScript.php", func=func@entry=false, funcParams=..., funcRet=...,
    reqInitFunc="", reqInitDoc="", error=@0x7ffd57a96d3d: false, errorMsg="", once=true, warmupOnly=false, richErrorMsg=false)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/program-functions.cpp:2254
#57 0x0000000000fd8d44 in HPHP::hphp_invoke_simple (filename="/srv/mediawiki/multiversion/MWScript.php", warmupOnly=warmupOnly@entry=false)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/program-functions.cpp:2208
#58 0x0000000000fe2d1f in HPHP::execute_program_impl (argc=argc@entry=13, argv=argv@entry=0x7ffd57a98a00)
    at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/program-functions.cpp:1827
#59 0x0000000000fe506e in HPHP::execute_program (argc=13, argv=argv@entry=0x7ffd57a98a00) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/program-functions.cpp:1148
#60 0x0000000001032bc0 in HPHP::emulate_zend (argc=argc@entry=9, argv=argv@entry=0x7ffd57a98e88) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/runtime/base/emulate-zend.cpp:283
#61 0x0000000000b56cc3 in main (argc=<optimized out>, argv=0x7ffd57a98e88) at /tmp/buildd/hhvm-3.18.1+dfsg/hphp/hhvm/main.cpp:65

Looking at a different mediawiki server it's likely this MWScript call (which matches timewise):

www-data 142401  0.0  0.0 359868 34988 ?        Ss   14:40   0:00 /usr/bin/php /srv/mediawiki/multiversion/MWScript.php maintenance/getConfiguration.php --wiki enwiki --settings wgCirrusSearchDefaultCluster wgCirrusSearchPreferRecentDefaultDecayPortion wgCirrusSearchPowerSpecialRandom wgCirrusSearchShardCount wgCirrusSearchQueryStringMaxDeterminizedStates wgCirrusSearchMergeSettings wgCirrusSearchCompletionSuggesterUseDefaultSort wgCirrusSearchClusterOverrides wgCirrusSearchLanguageToWikiMap wgCirrusSearchTextcatLanguages wgCirrusSearchFieldTypes wgCirrusSearchMasterTimeout wgCirrusSearchMoreLikeThisConfig wgCirrusSearchBannedPlugins wgCirrusSearchCompletionProfiles wgCirrusSearchSearchShardTimeout wgCirrusSearchTalkNamespaceWeight wgCirrusSearchClientSideSearchTimeout wgCirrusSearchPreferRecentUnspecifiedDecayPortion wgCirrusSearchCrossProjectOrder wgCirrusSearchExtraIndexSettings wgCirrusSearchInterwikiSources wgCirrusSearchLanguageWeight wgCirrusSearchBoostOpening wgCirrusSearchLogElasticRequests wgCirrusSearchDropDelayedJobsAfter wgCirrusSearchBackup wgCirrusSearchEnableCrossProjectSearch wgCirrusSearchRescoreProfiles wgCirrusSearchMoreLikeThisAllowedFields wgCirrusSearchIndexedRedirects wgCirrusSearchUseCompletionSuggester wgCirrusSearchSimilarityProfile wgCirrusSearchExtraBackendLatency wgCirrusSearchPhraseSuggestProfiles wgCirrusSearchUseExperimentalHighlighter wgCirrusSearchInterwikiThreshold wgCirrusSearchExtraIndexes wgCirrusSearchWeights wgCirrusSearchFeedbackLink wgCirrusSearchCreateFrozenIndex wgCirrusSearchConnectionAttempts wgCirrusSearchAllFields wgCirrusSearchMaxShardsPerNode wgCirrusSearchNamespaceMappings wgCirrusSearchAllowLeadingWildcard wgCirrusSearchBoostLinks wgCirrusSearchInterwikiProv wgCirrusSearchFullTextQueryBuilderProfile wgCirrusSearchPrefixSearchStartsWithAnyWord wgCirrusSearchFieldTypeOverrides wgCirrusSearchClientSideUpdateTimeout wgCirrusSearchPhraseSlop wgCirrusSearchPhraseSuggestUseText wgCirrusSearchPhraseSuggestMaxTermFreqHardLimit wgCirrusSearchUseIcuFolding wgCirrusSearchClusters wgCirrusSearchUnlinkedArticlesToUpdate wgCirrusSearchWikimediaExtraPlugin wgCirrusSearchFullTextClusterOverrides wgCirrusSearchPhraseSuggestPrefixLengthHardLimit wgCirrusSearchNamespaceWeights wgCirrusSearchIgnoreOnWikiBoostTemplates wgCirrusSearchUpdateDelay wgCirrusSearchMaintenanceTimeout wgCirrusSearchMoreLikeThisTTL wgCirrusSearchTextcatConfig wgCirrusSearchPhraseRescoreBoost wgCirrusSearchFiletypeAliases wgCirrusSearchNearMatchWeight wgCirrusSearchEnableCrossLanguageSearch wgCirrusSearchPrefixIds wgCirrusSearchMainPageCacheWarmer wgCirrusSearchOptimizeIndexForExperimentalHighlighter wgCirrusSearchPhraseSuggestMaxErrorsHardLimit wgCirrusSearchCompletionSettings wgCirrusSearchPhraseSuggestAllowedMode wgCirrusSearchInterwikiCacheTime wgCirrusSearchDefaultNamespaceWeight wgCirrusSearchEnablePhraseSuggest wgCirrusSearchPhraseSuggestReverseField wgCirrusSearchUpdateConflictRetryCount wgCirrusSearchUserTesting wgCirrusSearchRecycleCompletionSuggesterIndex wgCirrusSearchMoreLikeThisFields wgCirrusSearchBoostTemplates wgCirrusSearchFullTextQueryBuilderProfiles wgCirrusSearchPhraseSuggestUseOpeningText wgCirrusSearchMoreLikeThisMaxQueryTermsLimit wgCirrusSearchRescoreFunctionScoreChains wgCirrusSearchNumCrossProjectSearchResults wgCirrusSearchCompletionDefaultScore wgCirrusSearchUpdateShardTimeout wgCirrusSearchSimilarityProfiles wgCirrusSearchUseIcuTokenizer wgCirrusSearchTextcatModel wgCirrusSearchIndexAllocation wgCirrusSearchMoreAccurateScoringMode wgCirrusSearchPhraseSuggestSettings wgCirrusSearchPoolCounterKey wgCirrusSearchPhraseRescoreWindowSize wgCirrusSearchPreferRecentDefaultHalfLife wgCirrusSearchCompletionSuggesterHardLimit wgCirrusSearchElasticQuirks wgCirrusSearchStemmedWeight wgCirrusSearchRefreshInterval wgCirrusSearchClientSideConnectTimeout wgCirrusSearchFragmentSize wgCirrusSearchIndexBaseName wgCirrusSearchStripQuestionMarks wgCirrusSearchEnableAltLanguage wgCirrusSearchNewCrossProjectPage wgCirrusSearchReplicas wgCirrusSearchRescoreProfile wgCirrusSearchDevelOptions wgCirrusSearchComple

This crash didn't occur last week when mw1261 was pooled to check for potential issues with HHVM 3.18. Maybe the script didn't run back then?

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Mar 27 2017, 2:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2017, 2:49 PM
MoritzMuehlenhoff added a parent task: T158176: Build / migrate to HHVM 3.18.Mar 27 2017, 2:49 PM
dcausse updated the task description. (Show Details)Mar 27 2017, 2:51 PM
dcausse renamed this task from HHVM 3.18 crashes when running Cirrus maintenance script to HHVM 3.18 crashes when Cirrus tries to fetch another wiki config via maint script .Mar 27 2017, 2:58 PM
hashar subscribed.Mar 27 2017, 3:41 PM

Jobs do shell out to mwscript maintenance/getConfiguration.php to get configuration for all the wiki projects. That has previously lead to outage T145819. The idea is roughly:

  • MediaWiki wfShellExec( 'mwscript maintenance/getConfiguration.php' )
  • ulimit for file size is set to 512MBytes or so
  • HHVM attempts to update the bytecode cache
  • ulimit kicks in and abort due to the file cache being larger than the limit

The stacktrace here looks similar with call to HPHP::RepoQuery::step .. sqlite3OsWrite ... seekAndWrite.

Then mwscript is supposedly still hardcoded to use php5 T146285. mw1261 is a regular application server, so some web request apparently trigger: SiteConfiguration::getConfig -> mwscript (under hhvm) -> die.

On mw1261 /var/cache/hhvm/fcgi.hhbc.sq3 is 2.5GBytes which would trigger the ulimit but that is solely for the web application which is not being ulimit. The cli.hhbc.sq3 is at 512MBytes.

mw1261$ ls -lh /var/cache/hhvm/
total 3.1G
-rw-r--r-- 1 www-data www-data 512M Mar 27 14:02 cli.hhbc.sq3
-rw-r--r-- 1 www-data www-data 5.6K Mar 27 14:02 cli.hhbc.sq3-journal
-rw-r--r-- 1 www-data www-data 2.5G Mar 27 14:13 fcgi.hhbc.sq3
-rw-r--r-- 1 www-data www-data  75M Feb  6 12:08 hhvm.hhbc

So I am tempted to say that it is T145819 surfacing again but I dont see why mwscript uses hhvm instead of php5.

hashar mentioned this in T145819: Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MBytes.Mar 27 2017, 3:42 PM
hashar added a comment.Mar 27 2017, 3:52 PM

Sorry the suspected exec is: /usr/bin/php /srv/mediawiki/multiversion/MWScript.php maintenance/getConfiguration.php

And php is HHVM on mw1261:

mw1261:~$ /usr/bin/php --version
HipHop VM 3.18.1 (rel)
Compiler: 3.18.1+dfsg-1+wmf1

MediaWiki invokes php because of includes/DefaultSettings.php:$wgPhpCli = '/usr/bin/php';

And SiteConfiguration::getConfig does:

$cmd = wfShellWikiCmd(  # <-------------------------uses $wgPhpCli
     "$IP/maintenance/getConfiguration.php",
     [
         '--wiki', $wiki,
         '--settings', implode( ' ', $settings ),
         '--format', 'PHP'
     ]
 );

So most probably yes, that is a wfShellExec to the CLI hhvm that is run under with ulimit at 512MBytes eg T145819.

MoritzMuehlenhoff mentioned this in T161598: Monitor HHVM bytecode cache depletion on mediawiki app servers.Mar 28 2017, 10:56 AM
MoritzMuehlenhoff closed this task as Resolved.Apr 5 2017, 10:54 AM
MoritzMuehlenhoff claimed this task.

So this is a generic problem with depleting the HHVM byte code cache and would've happened with 3.12 as well. I've created T161598 to address this problem in general.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits