Page MenuHomePhabricator

Introduce non-voting jobs with quibble+apache
Closed, ResolvedPublic

Description

Quibble defaults to the PHP built-in server, but since release 0.0.46 also includes Apache support. The quibble-apache:0.0.1 image includes the necessary packages.

In this task, we'll create a few non-voting jobs in various repos, which exercise quibble+apache. We want to determine:

  • How does performance compare to the regular quibble job?
  • Do some repos have quirks to address before we can use the apache job?

See this example job which includes the necessary glue. Hopefully this can be simplified a bit.

The major difference is that the apache job can serve requests concurrently, which will expose some issues with our code and require fixes to stabilize tests.

In the long-term, if the results are encouraging we might decide to switch all quibble jobs to use apache.

Related Objects

Event Timeline

The major difference is that the apache job can serve requests concurrently, which will expose some issues with our code and require fixes to stabilize tests.

Have we seen this kind of issue happening in practice, or is it a hunch? If a hunch, it might be worth just flipping the switch for all selenium jobs and see what happens. I struggle to imagine real examples where using Apache could cause tests to start failing, especially with local development mostly happening in VMs or containers that have always used Apache or Nginx with concurrency, and people do run them locally and see it pass.

But that opens some other issues such as race conditions between queries (and potentially locking issue with Sqlite).

Hmm, where is that perceived to be an issue?
I don't think we would run into anything like that for our tests for example.
Setup is generally done serially anyway, the current pinch point seems to be API calls that happen when the UI is actually loaded, and these seem to cause many timeouts! :(

T199393#6702427 is an example, and I am pretty sure we had issue when multiple API calls triggered a localisation cache rebuild which raced for a lock on the sqlite database (got solved by always generating the loalisation cache when installing mediawiki). There must be a bunch of other related races. But we will need to have the jobs and run them to surface them.

Off the top of my head there shouldn't be any more wide spread issues with concurrent requests.
If tests needs to do things in a certain order, they should make sure they do them in a certain order.

I volunteer Wikibase to trial some non voting jobs running this apache varient

The major difference is that the apache job can serve requests concurrently, which will expose some issues with our code and require fixes to stabilize tests.

Have we seen this kind of issue happening in practice, or is it a hunch? If a hunch, it might be worth just flipping the switch for all selenium jobs and see what happens. I struggle to imagine real examples where using Apache could cause tests to start failing, especially with local development mostly happening in VMs or containers that have always used Apache or Nginx with concurrency, and people do run them locally and see it pass.

Yeah, this was one issue (T199393) spotted during test runs (worked around with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/654417) and another one was https://gerrit.wikimedia.org/r/c/654443

I am in favor of flipping the switch too, if we can have some people around to help fix tests quickly.

Change 670965 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add php72-apache flavour to quibble jobs

https://gerrit.wikimedia.org/r/670965

Change 670967 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] non voting quibble-apache job for Wikibase

https://gerrit.wikimedia.org/r/670967

Change 670965 merged by jenkins-bot:
[integration/config@master] jjb: wikimedia Quibble job with Apache

https://gerrit.wikimedia.org/r/670965

Change 670967 merged by jenkins-bot:
[integration/config@master] Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job

https://gerrit.wikimedia.org/r/670967

Change 671167 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] jjb: Add quibble apache job for selenium too

https://gerrit.wikimedia.org/r/671167

Change 671168 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] zuul: Use non voting wmf-quibble-apache-selenium-php72-docker for Wikibase

https://gerrit.wikimedia.org/r/671168

Change 671167 merged by jenkins-bot:
[integration/config@master] jjb: Add quibble apache job for selenium too

https://gerrit.wikimedia.org/r/671167

Change 671168 merged by jenkins-bot:
[integration/config@master] zuul: Use non voting wmf-quibble-apache-selenium-php72-docker for Wikibase

https://gerrit.wikimedia.org/r/671168

Change 671188 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] WIP DNM: data-bridge: dump browser console logs on failure

https://gerrit.wikimedia.org/r/671188

So it looks like every non voting selenium job for Wikibase using apache as the webserver has failed since being introduced.
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/buildTimeTrend

image.png (437×520 px, 31 KB)

A quick scan seems to show that most of these are timeout waiting for elements to be displayed.

Looking at the most recent failure, where the current job passed https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/47/console
The first issue is:

20:00:55 [0-1] Error in "item can add a statement using the keyboard"
20:00:55 element ("div.wikibase-addtoolbar > .wikibase-toolbar-button-add > a") still not displayed after 10000ms

I played around with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/671188 a little more testing videos and trying to log console logs to CI but generally failed and couldn;t debug the failing tests any further..

Will have to try another approach next (maybe locally)

Change 701050 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] zuul: Add quibble-apache-selenium job to experimental group for GrowthExperiments

https://gerrit.wikimedia.org/r/701050

Change 701050 merged by jenkins-bot:

[integration/config@master] zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE

https://gerrit.wikimedia.org/r/701050

So it looks like every non voting selenium job for Wikibase using apache as the webserver has failed since being introduced.
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/buildTimeTrend

image.png (437×520 px, 31 KB)

A quick scan seems to show that most of these are timeout waiting for elements to be displayed.

Looking at the most recent failure, where the current job passed https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/47/console
The first issue is:

20:00:55 [0-1] Error in "item can add a statement using the keyboard"
20:00:55 element ("div.wikibase-addtoolbar > .wikibase-toolbar-button-add > a") still not displayed after 10000ms

I played around with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/671188 a little more testing videos and trying to log console logs to CI but generally failed and couldn;t debug the failing tests any further..

Will have to try another approach next (maybe locally)

I can't reproduce the error locally.

I'd be interested to look at the VM running the tests and just see if it is even more stuff like T281122: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs and VM memory compaction etc making things freeze up

I'd be interested to look at the VM running the tests and just see if it is even more stuff like T281122: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs and VM memory compaction etc making things freeze up

I'm able to reproduce the error in a VM on DigitalOcean (Ubuntu OS), using the same Docker command that we use in CI. In the same container, starting the php built-in server and telling Selenium to use that as the URL allows the test to pass.

Here are the request logs for npm run selenium-test:repo -- --spec repo/tests/selenium/specs/tainted-ref.js. In apache, the test fails with:

[0-0] Error in "the Tainted icon should appear and disappear correctly"
Can't call click on element with selector ".wikibase-toolbar-button-edit" because element wasn't found
[0-0] FAILED in chrome - /repo/tests/selenium/specs/tainted-ref.js

 "spec" Reporter:
------------------------------------------------------------------
[Chrome Headless 90.0.4430.93 linux #0-0] Spec: /workspace/src/extensions/Wikibase/repo/tests/selenium/specs/tainted-ref.js
[Chrome Headless 90.0.4430.93 linux #0-0] Running: Chrome Headless (v90.0.4430.93) on linux
[Chrome Headless 90.0.4430.93 linux #0-0] Session ID: efbb6e37-1a40-4785-9a32-a9a4d49f8a23
[Chrome Headless 90.0.4430.93 linux #0-0]
[Chrome Headless 90.0.4430.93 linux #0-0] the Tainted icon
[Chrome Headless 90.0.4430.93 linux #0-0]    ✖ should appear and disappear correctly
[Chrome Headless 90.0.4430.93 linux #0-0]
[Chrome Headless 90.0.4430.93 linux #0-0] 1 failing (8.1s)
[Chrome Headless 90.0.4430.93 linux #0-0]
[Chrome Headless 90.0.4430.93 linux #0-0] 1) the Tainted icon should appear and disappear correctly
[Chrome Headless 90.0.4430.93 linux #0-0] Can't call click on element with selector ".wikibase-toolbar-button-edit" because element wasn't found
[Chrome Headless 90.0.4430.93 linux #0-0] Error: Can't call click on element with selector ".wikibase-toolbar-button-edit" because element wasn't found
[Chrome Headless 90.0.4430.93 linux #0-0]     at ItemPage.clickEditOnStatementElement (/workspace/src/extensions/Wikibase/node_modules/wdio-wikibase/pagesections/main.statement.section.js:188:70)
[Chrome Headless 90.0.4430.93 linux #0-0]     at ItemPage.clickEditOnStatement (/workspace/src/extensions/Wikibase/node_modules/wdio-wikibase/pagesections/main.statement.section.js:176:8)
[Chrome Headless 90.0.4430.93 linux #0-0]     at ItemPage.editStatementValue (/workspace/src/extensions/Wikibase/node_modules/wdio-wikibase/pagesections/main.statement.section.js:200:8)
[Chrome Headless 90.0.4430.93 linux #0-0]     at Context.it (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/tainted-ref.js:49:12)

This is the screenshot

should-appear-and-disappear-correctly.png (900×1 px, 129 KB)
where you can see that the edit toolbar item is visible.

What's odd is that if I view the URL in my browser, the item is definitely present, and clickable. And if I change the line above the .click() statement to use waitForClickable(), webdriver tells me that the item is clickable.

php built-in server:

[Thu Jul  1 12:32:31 2021] 127.0.0.1:52938 [200]: //api.php?format=json
[Thu Jul  1 12:32:31 2021] 127.0.0.1:52940 [200]: //api.php?format=json
[Thu Jul  1 12:32:31 2021] 127.0.0.1:52942 [200]: //api.php?format=json
[Thu Jul  1 12:32:31 2021] 127.0.0.1:52944 [200]: //api.php?format=json
[Thu Jul  1 12:32:31 2021] 127.0.0.1:52946 [301]: //index.php?title=Special%3AEntityPage%2FQ9
[Thu Jul  1 12:32:32 2021] 127.0.0.1:52948 [200]: /index.php/Item:Q9
[Thu Jul  1 12:32:32 2021] 127.0.0.1:52950 [200]: /load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector
[Thu Jul  1 12:32:33 2021] 127.0.0.1:52952 [200]: /load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector
[Thu Jul  1 12:32:33 2021] 127.0.0.1:52968 [200]: /load.php?lang=en&modules=ext.visualEditor.core.utils.parsing%7Cext.visualEditor.desktopArticleTarget.init%7Cext.visualEditor.progressBarWidget%2CsupportCheck%2CtargetLoader%2CtempWikitextEditorWidget%2Ctrack%2Cve&skin=vector&version=164jy
[Thu Jul  1 12:32:34 2021] 127.0.0.1:52970 [200]: /load.php?lang=en&modules=dataValues%2Cjquery%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Csite%2CvalueFormatters%2CvalueParsers%2Cvue%2Cwikibase%7CdataValues.DataValue%2CTimeValue%2Cvalues%7Cext.cx.eventlogging.campaigns%7Cext.eventLogging%2CnavigationTiming%7Cext.uls.common%2Ccompactlinks%2Ci18n%2Cinterface%2Clanguagenames%2Cmediawiki%2Cmessages%2Cpreferences%2Cwebfonts%7Cjquery.animateWithEvent%2Cclient%2Ccookie%2Ci18n%2Cinputautoexpand%2Cspinner%2CtextSelection%2Ctipsy%2Cui%2Culs%2Cvalueview%7Cjquery.event.special.eachchange%7Cjquery.spinner.styles%7Cjquery.ui.commonssuggester%2Clanguagesuggester%2Csuggester%7Cjquery.uls.data%2Cgrid%7Cjquery.util.getDirectionality%7Cjquery.valueview.Expert%2CExpertExtender%2CExpertStore%7Cjquery.valueview.experts.CommonsMediaType%2CEmptyValue%2CGeoShape%2CGlobeCoordinateInput%2CMonolingualText%2CQuantityInput%2CStringValue%2CTabularData%2CTimeInput%2CUnDeserializableValue%7Cjquery.wikibase.entityselector%2Cwbtooltip%7Cmediawiki.ForeignApi%2CString%2CTitle%2CUri%2Capi%2Cbase%2Ccldr%2Ccookie%2CjqueryMsg%2Clanguage%2Cstorage%2Cuser%2Cutil%7Cmediawiki.ForeignApi.core%7Cmediawiki.editfont.styles%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%7Cmediawiki.page.watch.ajax%7Cmediawiki.ui.button%7Cmw.config.values.wbDataBridgeConfig%2CwbDataTypes%2CwbRefTabsEnabled%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.icons%2Cstyles%7Coojs-ui-toolbars.icons%7Coojs-ui-widgets.icons%7Coojs-ui-windows.icons%7Coojs-ui.styles.indicators%7Cskins.vector.legacy.js%7Cuser.defaults%7Cutil.ContentLanguages%2CExtendable%2CMessageProvider%2CMessageProviders%2CNotifier%2ChighlightSubstring%2Cinherit%7CvalueParsers.ValueParserStore%2Cparsers%7Cwikibase.EntityInitializer%2CSite%2CWikibaseContentLanguages%2CbuildErrorOutput%2Cdatamodel%7Cwikibase.api.RepoApi%2CValueCaller%7Cwikibase.client.data-bridge.init%7Cwikibase.entityChangers.EntityChangersFactory%7Cwikibase.entityPage.entityLoaded%7Cwikibase.experts.Entity&skin=vector&version=d3cez
[Thu Jul  1 12:32:34 2021] 127.0.0.1:52972 [200]: /load.php?lang=en&modules=wikibase.experts.Item%2CProperty%2C__namespace%2Cmodules%7Cwikibase.formatters.ApiValueFormatter%7Cwikibase.getLanguageNameByCode%2CgetUserLanguages%2Cserialization%2Csites%2Ctainted-ref%2Ctemplates%7Cwikibase.ui.entityViewInit%2Centitysearch%7Cwikibase.utilities.ClaimGuidGenerator%7Cwikibase.view.ControllerViewFactory%2CReadModeViewFactory%2C__namespace&skin=vector&version=irwsu
[Thu Jul  1 12:32:34 2021] 127.0.0.1:52974 [200]: /index.php/Special:EntityData/Q9.json?revision=28
[Thu Jul  1 12:32:34 2021] 127.0.0.1:52976 [200]: /load.php?lang=en&modules=wikibase.mediainfo.getDeserializer%7Cwikibase.mediainfo.serialization.MediaInfoDeserializer&skin=vector&version=i9esr
[Thu Jul  1 12:32:34 2021] 127.0.0.1:52978 [200]: /load.php?lang=en&modules=ext.uls.webfonts.fonts%2Crepository%7Cjquery.webfonts&skin=vector&version=gs7vw
[Thu Jul  1 12:32:35 2021] 127.0.0.1:52980 [200]: /api.php?action=wbparsevalue&format=json&parser=wikibase-entityid&values=%1FP9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:35 2021] 127.0.0.1:52982 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22ExampleValue%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fplain&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:35 2021] 127.0.0.1:52984 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22refString%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fplain&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:35 2021] 127.0.0.1:52986 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%7B%22id%22%3A%22P9%22%7D%2C%22type%22%3A%22wikibase-entityid%22%7D&generate=text%2Fplain&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:35 2021] 127.0.0.1:52988 [200]: /load.php?lang=en&modules=ext.uls.ime%7Cjquery.ime&skin=vector&version=o8jpg
[Thu Jul  1 12:32:36 2021] 127.0.0.1:52992 [200]: /load.php?lang=en&modules=ext.uls.displaysettings%2Clanguagesettings%7Cmediawiki.ui.checkbox%2Cradio&skin=vector&version=1q8jt
[Thu Jul  1 12:32:36 2021] 127.0.0.1:52994 [200]: /api.php?action=wbparsevalue&format=json&parser=string&values=%1FnewValue0.8461497974592946-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:36 2021] 127.0.0.1:52996 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22newValue0.8461497974592946-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fhtml%3B%20disposition%3Dverbose-preview&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:36 2021] 127.0.0.1:52998 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22newValue0.8461497974592946-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fplain&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:36 2021] 127.0.0.1:53000 [200]: /api.php?action=query&format=json&meta=tokens&type=csrf
[Thu Jul  1 12:32:36 2021] 127.0.0.1:53002 [200]: /api.php
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53004 [200]: /api.php?action=wbparsevalue&format=json&parser=wikibase-entityid&values=%1FP9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53006 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22refString%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fhtml%3B%20disposition%3Dverbose-preview&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53010 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22refString%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fhtml%3B%20disposition%3Dverbose-preview&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53012 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%7B%22id%22%3A%22P9%22%7D%2C%22type%22%3A%22wikibase-entityid%22%7D&generate=text%2Fhtml&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53014 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22newValue0.8461497974592946-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fplain&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53016 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22refString%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fplain&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[Thu Jul  1 12:32:37 2021] 127.0.0.1:53022 [200]: /api.php?action=wbformatvalue&format=json&datavalue=%7B%22value%22%3A%22refString%22%2C%22type%22%3A%22string%22%7D&generate=text%2Fhtml%3B%20disposition%3Dverbose-preview&property=P9&errorformat=plaintext&uselang=en&options=%7B%22lang%22%3A%22en%22%7D
[

apache:

2021-07-01T13:28:26	34418	59	POST	http://127.0.0.1:9413/api.php?format=json	application/json	-	mwbot/1.0.10	-
2021-07-01T13:28:26	153370	143	POST	http://127.0.0.1:9413/api.php?format=json	application/json	-	mwbot/1.0.10	-
2021-07-01T13:28:26	81856	59	POST	http://127.0.0.1:9413/api.php?format=json	application/json	-	mwbot/1.0.10	-
2021-07-01T13:28:26	210159	789	POST	http://127.0.0.1:9413/api.php?format=json	application/json	-	mwbot/1.0.10	-
2021-07-01T13:28:27	136402	20	GET	http://127.0.0.1:9413/index.php?title=Special%3AEntityPage%2FQ11	text/html	-	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	188591	6084	GET	http://127.0.0.1:9413/index.php?title=Item:Q11	text/html	-	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	168	22188	GET	http://127.0.0.1:9413/resources/assets/wiki.png?4d48b	image/png	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	166660	13828	GET	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	text/css	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	173	159	GET	http://127.0.0.1:9413/skins/Vector/resources/common/images/bullet-icon.svg?d4515	image/svg+xml	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	169	717	GET	http://127.0.0.1:9413/extensions/UniversalLanguageSelector/lib/jquery.uls/images/language.svg?80b0b	image/svg+xml	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	165	254	GET	http://127.0.0.1:9413/skins/Vector/resources/common/images/user-avatar.svg?b7f58	image/svg+xml	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	161	280	GET	http://127.0.0.1:9413/skins/Vector/resources/common/images/search.svg?bbf78	image/svg+xml	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	167	271	GET	http://127.0.0.1:9413/extensions/Wikibase/client/resources/images/edit.svg?34bc5	image/svg+xml	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.pt%7Cext.visualEditor.desktopArticleTarget.noscript%7Cjquery.wikibase.toolbar.styles%7Cskins.vector.styles.legacy%7Cwikibase.client.data-bridge.externalModifiers%7Cwikibase.client.init%7Cwikibase.common&only=styles&skin=vector	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	169	3839	GET	http://127.0.0.1:9413/resources/assets/poweredby_mediawiki_88x31.png	image/png	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:27	616474	19948	GET	http://127.0.0.1:9413/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector	text/javascript	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:28	163162	15520	GET	http://127.0.0.1:9413/load.php?lang=en&modules=ext.visualEditor.core.utils.parsing%7Cext.visualEditor.desktopArticleTarget.init%7Cext.visualEditor.progressBarWidget%2CsupportCheck%2CtargetLoader%2CtempWikitextEditorWidget%2Ctrack%2Cve&skin=vector&version=164jy	text/javascript	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:28	243621	104809	GET	http://127.0.0.1:9413/load.php?lang=en&modules=wikibase.experts.Item%2CProperty%2C__namespace%2Cmodules%7Cwikibase.formatters.ApiValueFormatter%7Cwikibase.getLanguageNameByCode%2CgetUserLanguages%2Cserialization%2Csites%2Ctainted-ref%2Ctemplates%7Cwikibase.ui.entityViewInit%2Centitysearch%7Cwikibase.utilities.ClaimGuidGenerator%7Cwikibase.view.ControllerViewFactory%2CReadModeViewFactory%2C__namespace&skin=vector&version=irwsu	text/javascripthttp://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:28	441987	419214	GET	http://127.0.0.1:9413/load.php?lang=en&modules=dataValues%2Cjquery%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Csite%2CvalueFormatters%2CvalueParsers%2Cvue%2Cwikibase%7CdataValues.DataValue%2CTimeValue%2Cvalues%7Cext.cx.eventlogging.campaigns%7Cext.eventLogging%2CnavigationTiming%7Cext.uls.common%2Ccompactlinks%2Ci18n%2Cinterface%2Clanguagenames%2Cmediawiki%2Cmessages%2Cpreferences%2Cwebfonts%7Cjquery.animateWithEvent%2Cclient%2Ccookie%2Ci18n%2Cinputautoexpand%2Cspinner%2CtextSelection%2Ctipsy%2Cui%2Culs%2Cvalueview%7Cjquery.event.special.eachchange%7Cjquery.spinner.styles%7Cjquery.ui.commonssuggester%2Clanguagesuggester%2Csuggester%7Cjquery.uls.data%2Cgrid%7Cjquery.util.getDirectionality%7Cjquery.valueview.Expert%2CExpertExtender%2CExpertStore%7Cjquery.valueview.experts.CommonsMediaType%2CEmptyValue%2CGeoShape%2CGlobeCoordinateInput%2CMonolingualText%2CQuantityInput%2CStringValue%2CTabularData%2CTimeInput%2CUnDeserializableValue%7Cjquery.wikibase.entityselector%2Cwbtooltip%7Cmediawiki.ForeignApi%2CString%2CTitle%2CUri%2Capi%2Cbase%2Ccldr%2Ccookie%2CjqueryMsg%2Clanguage%2Cstorage%2Cuser%2Cutil%7Cmediawiki.ForeignApi.core%7Cmediawiki.editfont.styles%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%7Cmediawiki.page.watch.ajax%7Cmediawiki.ui.button%7Cmw.config.values.wbDataBridgeConfig%2CwbDataTypes%2CwbRefTabsEnabled%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.icons%2Cstyles%7Coojs-ui-toolbars.icons%7Coojs-ui-widgets.icons%7Coojs-ui-windows.icons%7Coojs-ui.styles.indicators%7Cskins.vector.legacy.js%7Cuser.defaults%7Cutil.ContentLanguages%2CExtendable%2CMessageProvider%2CMessageProviders%2CNotifier%2ChighlightSubstring%2Cinherit%7CvalueParsers.ValueParserStore%2Cparsers%7Cwikibase.EntityInitializer%2CSite%2CWikibaseContentLanguages%2CbuildErrorOutput%2Cdatamodel%7Cwikibase.api.RepoApi%2CValueCaller%7Cwikibase.client.data-bridge.init%7Cwikibase.entityChangers.EntityChangersFactory%7Cwikibase.entityPage.entityLoaded%7Cwikibase.experts.Entity&skin=vector&version=rctay	text/javascript	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:28	188287	493	GET	http://127.0.0.1:9413/index.php?title=Special:EntityData/Q11.json&revision=33	application/json	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US
2021-07-01T13:28:29	103289	5556	GET	http://127.0.0.1:9413/load.php?lang=en&modules=ext.uls.webfonts.fonts%2Crepository%7Cjquery.webfonts&skin=vector&version=gs7vw	text/javascript	http://127.0.0.1:9413/index.php?title=Item:Q11	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.93 Safari/537.36	en-US

Comparing index.php?title=Item:Q2 on Apache and PHP built-in server for the tainted-ref.js test failure. I see that the JS editing UI isn't loading in the Apache version for the statement:

image.png (988×1 px, 139 KB)

But with PHP built-in server it is loaded:

image.png (1×1 px, 577 KB)

Comparing the network requests, the only difference I see is that PHP built-in server has a final call to: /load.php?lang=en&modules=wikibase.mediainfo.getDeserializer%7Cwikibase.mediainfo.serialization.MediaInfoDeserializer&skin=vector&version=i9esr

I don't know why that doesn't load for our Quibble + Apache setup in integration/config.

Comparing the network requests, the only difference I see is that PHP built-in server has a final call to: /load.php?lang=en&modules=wikibase.mediainfo.getDeserializer%7Cwikibase.mediainfo.serialization.MediaInfoDeserializer&skin=vector&version=i9esr

If this should be happening, and is not happening, this could indeed be the reason for the editing UI not loading.

There must be some difference in setups, or some other failure somewhere?

This ends up being required from https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/f13b4df1f0c5096d6dd338c80685b7e1db97affb/repo/resources/Resources.php#130 but there is nothing special there

I wonder if the other deps are loaded?

		'wikibase.EntityInitializer' => $moduleTemplate + [
			'packageFiles' => [
				'wikibase.EntityInitializer.js',


			'dependencies' => [
				'wikibase',
				'wikibase.serialization'
			]

After debugging the Wikibase issues @Krinkle and @kostajh found this is related to T282154

Change 701899 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@master] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/701899

Change 701899 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@master] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/701899

This fixes the Wikibase tests. The next failures are with AbuseFilter, see T285464: HTMLForms using POST might be broken if the form action is a URL containing "?title=" for that one.

Change 701899 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/701899

Change 702938 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] selenium: Restart php7.2-fpm after modifying LocalSettings.php

https://gerrit.wikimedia.org/r/702938

Change 702904 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@master] [DNM] Test for T276428

https://gerrit.wikimedia.org/r/702904

Change 702825 had a related patch set uploaded (by Addshore; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@REL1_35] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/702825

Change 702946 had a related patch set uploaded (by Addshore; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@REL1_36] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/702946

Change 702938 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] selenium: Restart php7.2-fpm after modifying LocalSettings.php

https://gerrit.wikimedia.org/r/702938

With this one, all the Selenium tests pass https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2017/consoleFull

Change 702825 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_35] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/702825

Change 702938 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] selenium: Restart php7.2-fpm after modifying LocalSettings.php

https://gerrit.wikimedia.org/r/702938

Can someone help diagnose (and ideally fix) this error:
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2841/console

11:31:14 [0-0] RUNNING in chrome - /tests/selenium/specs/switch_views.js
11:31:16 [0-0] Error in "Switch between views: desktop to mobile"
11:31:27 element ("#footer-places-mobileview a") still not existing after 5000ms
11:31:27 [0-0] FAILED in chrome - /tests/selenium/specs/switch_views.js

Incidentally I see that a build for a different commit succeeds:
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2842/

It seems like the test may be flaky.

Change 671188 abandoned by Kosta Harlan:

[mediawiki/extensions/Wikibase@master] WIP DNM: Selenium: videos for repo & bridge using wdio-video-reporter

Reason:

Ic0785e6810dfd70

https://gerrit.wikimedia.org/r/671188

Change 702946 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_36] EntityDataRequestHandler: Remove ob_clean() call

https://gerrit.wikimedia.org/r/702946

Can someone help diagnose (and ideally fix) this error:
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2841/console

11:31:14 [0-0] RUNNING in chrome - /tests/selenium/specs/switch_views.js
11:31:16 [0-0] Error in "Switch between views: desktop to mobile"
11:31:27 element ("#footer-places-mobileview a") still not existing after 5000ms
11:31:27 [0-0] FAILED in chrome - /tests/selenium/specs/switch_views.js

Incidentally I see that a build for a different commit succeeds:
https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2842/

It seems like the test may be flaky.

Marked as skipped in rEMFR29a34de17a6c: [Selenium] Skip desktop to mobile test

I think that is done, we had experimental jobs running including a wmf-quibble gating job triggered from Wikibase. I have switched the Selenium Quibble jobs to use Apache and that seems to work ;)

Change 702904 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] [DNM] Test for T276428

Reason:

I assume this is no longer needed

https://gerrit.wikimedia.org/r/702904