Page MenuHomePhabricator
Create Task
Maniphest T198201

Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server)
Closed, DeclinedPublic
Actions

Assigned To
None
Authored By
hashar
Jun 26 2018, 11:40 AM
Tags
Referenced Files
F22682597: shows-page-creation.png
Jun 26 2018, 1:00 PM
F22682600: should-show-page-with-new-edit.png
Jun 26 2018, 1:00 PM
Subscribers
Aklapper
awight
Etonkovidova
gerritbot
hashar
Krinkle
Ladsgroup
View All 10 Subscribers
Tokens
"Insectivore" token, awarded by zeljkofilipin.

Description

When running the selenium browser tests for ORES, the CI build fails with:

  ORES
      1) filters are present
      

1 failing (3s)

1) ORES filters are present:
false == true
AssertionError: false == true

The exception log shows:

/index.php?action=history&title=0.45299226547790816-%C3%B6%C3%A4%C3%BC-%E2%99%A0%E2%99%A3%E2%99%A5%E2%99%A6   RuntimeException from line 94 of extensions/ORES/includes/ORESService.php: Failed to make ORES request to [https://ores.wikimedia.org/v3/scores/wikidb/?models=damaging%7Cgoodfaith&revids=4&precache=1&format=json], There was a problem during the HTTP request: 404 NOT FOUND
#0 extensions/ORES/includes/ScoreFetcher.php(57): ORES\ORESService->request(array, array)
#1 extensions/ORES/includes/FetchScoreJob.php(89): ORES\ScoreFetcher->getScores(integer, array, boolean, array)
#2 includes/jobqueue/JobRunner.php(296): ORES\FetchScoreJob->run()
#3 includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(ORES\FetchScoreJob, Wikimedia\Rdbms\LBFactorySimple, BufferingStatsdDataFactory, integer)
#4 includes/MediaWiki.php(1007): JobRunner->run(array)
#5 includes/MediaWiki.php(990): MediaWiki->triggerSyncJobs(integer, MediaWiki\Logger\LegacyLogger)
#6 includes/MediaWiki.php(914): MediaWiki->triggerJobs()
#7 includes/MediaWiki.php(729): MediaWiki->restInPeace(string, boolean)
#8 includes/MediaWiki.php(752): MediaWiki->{closure}()
#9 includes/MediaWiki.php(557): MediaWiki->doPostOutputShutdown(string)
#10 index.php(42): MediaWiki->run()
#11 {main}

Which is due to the CI setup trying to reach the default ORES:

extension.json
{
    "config": {
        "OresBaseUrl": {
            "value": "https://ores.wikimedia.org/"
        },
}

Thus the wiki tries to reach https://ores.wikimedia.org/v3/scores/wikidb/ which obviously yields nothing.

I don't think we can easily spawn an ORES backend in CI, at least not today. But surely we can find a way to skip the test entirely when being the webdriver tests are run from a patchset upload.

Repro: check experimental on dummy change https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ORES/+/436523/

Details

SubjectRepoBranchLines +/-
Disable failing test that prevents merges across WMF-deployed reposmediawiki/extensions/ORESREL1_31+2 -3
Inject tests/selenium/LocalSettings.phpmediawiki/coremaster+36 -1
(DO NOT SUBMIT) Test change for T198201mediawiki/extensions/ORESmaster+0 -0
Change environment when running Selenium testsintegration/quibblemaster+31 -18
Migrate ORES to Quibbleintegration/configmaster+1 -2
Disable failing test that prevents merges across WMF-deployed reposmediawiki/extensions/ORESmaster+2 -2
Inject tests/selenium/LocalSettings.phpintegration/quibblemaster+20 -0
selenium: skip test on CImediawiki/extensions/ORESmaster+9 -0
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Jun 26 2018, 11:40 AM
Restricted Application added a project: Machine-Learning-Team. · View Herald TranscriptJun 26 2018, 11:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar mentioned this in T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.Jun 26 2018, 12:12 PM
hashar added a parent task: T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.
gerritbot subscribed.Jun 26 2018, 12:40 PM

Change 442085 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/ORES@master] selenium: skip test on CI

https://gerrit.wikimedia.org/r/442085

gerritbot added a project: Patch-For-Review.Jun 26 2018, 12:40 PM
gerritbot added a comment.Jun 26 2018, 12:56 PM

Change 442085 abandoned by Hashar:
selenium: skip test on CI

Reason:
So that skips the test when being run under Jenkins and if triggered from Zuul. Then some mediawiki/core selenium tests end up failling

  • Special:RecentChanges shows page creation
  • Special:Watchlist should show page with new edit

So I guess instead we need to have ORES to be disabled by default until a proper URL is set up.

https://gerrit.wikimedia.org/r/442085

hashar added a comment.Jun 26 2018, 1:00 PM

The patch would skip the ORES selenium tests just fine however mediawiki/core tests end up failing.

Special:RecentChanges shows page creation
Special:Watchlist should show page with new edit

shows-page-creation.png (894×1 px, 162 KB)

should-show-page-with-new-edit.png (894×1 px, 175 KB)

So I guess we need to find a way to disable ORES by default. Of course for wikimedia that would be need to be explicit set in operations/mediawiki-config and if we have a daily browser test, it would need a tests/selenium/LocalSettings.php to set the URL.

awight edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Jun 26 2018, 1:05 PM
awight subscribed.

@hashar Thanks for the investigation! Can you assign this an urgency? I don't understand whether we're currently breaking CI for everyone, if it's a warning or error, etc.

awight added a comment.Jun 26 2018, 1:06 PM

See also our task to integrate a working ORES service under docker: T195073. If that would be a solution, I'd prefer to put our energy into finishing that integration.

hashar added a comment.Jun 26 2018, 1:41 PM

Right now patches sent to mediawiki/extensions/ORES do not run the Selenium tests. I am in a process of migrating every extensions to use a new set of jobs. They run under Docker and do run the Selenium tests (mediawiki/core ones + extensions ones). The migration task is T183512.

The new job can be triggered by using check experimental. The container installs MediaWiki and the extension, spawns a webserver then run the Selenium tests against the local webserver. And of course, there is no ORES server available.

I guess if somehow had $wgOresBaseUrl to default to null/false and thus entirely skip ORES until it is properly setup, that would work around my use case :]

Adding end to end tests of ORES extensions with an ORES backend is a different task. That will be accomplished as part of new deployment pipeline ( T170481 ), currently scope limited to Math / Mathoid.

awight added a comment.Jun 26 2018, 2:22 PM

@hashar Okay I see, we're blocking the migration, maybe we have a few weeks to act?

So does it make sense for us to integrate our dockerfile with your work, or do you have a universal docker image which will be used for all extensions?

zeljkofilipin added a project: User-zeljkofilipin.Jun 27 2018, 9:14 AM
Ladsgroup claimed this task.Jun 27 2018, 4:59 PM

this is actually easily fixable.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJun 27 2018, 4:59 PM
Ladsgroup added a comment.Jun 27 2018, 5:14 PM

It seems tests/selenium/LocalSettings.php is not being loaded, otherwise It would hit https://ores.wikimedia.org/v3/scores/teswiki and not https://ores.wikimedia.org/v3/scores/wikidb (See the related codebase)

Ladsgroup removed Ladsgroup as the assignee of this task.Jun 27 2018, 7:24 PM
MusikAnimal subscribed.Jun 27 2018, 8:23 PM

Could this be what's causing https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6154/console ? Related patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/440018

Fails on "Special:RecentChanges shows page creation". In previous patchsets we were able to make this test pass after rechecking.

hashar added a comment.Jun 27 2018, 8:29 PM
In T198201#4315627, @awight wrote:

@hashar Okay I see, we're blocking the migration, maybe we have a few weeks to act?

So does it make sense for us to integrate our dockerfile with your work, or do you have a universal docker image which will be used for all extensions?

We are only starting the grand plan to migrate services to Docker with Math/Mathoid being the first use case. Each service will have its own docker image which will be build/used by CI as well eventually. But all of that will take a few more months/quarters to achieve.

In T198201#4319865, @Ladsgroup wrote:

It seems tests/selenium/LocalSettings.php is not being loaded, otherwise It would hit https://ores.wikimedia.org/v3/scores/teswiki and not https://ores.wikimedia.org/v3/scores/wikidb (See the related codebase)

Yup that is it. Quibble does not load tests/selenium/LocalSettings.php! I need to implement support for that file.

hashar added a comment.Jun 27 2018, 8:35 PM
In T198201#4320439, @MusikAnimal wrote:

Could this be what's causing https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6154/console ? Related patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/440018

Fails on "Special:RecentChanges shows page creation". In previous patchsets we were able to make this test pass after rechecking.

The job triggered for PageTriage does not have ORES injected as a dependency. So that is absolutely unrelated. That "Special:RecentChanges shows page creation" has a different cause and deserves a different task :]

hashar claimed this task.Jun 27 2018, 8:36 PM
hashar added a project: Quibble.

That is a Quibble issue. Thank you @Ladsgroup !

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 27 2018, 8:36 PM
hashar renamed this task from ORES webdriver.io selenium test fail on CI due to lack of ORES server to Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server).Jun 27 2018, 8:36 PM
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.Jun 28 2018, 9:32 AM
• Niharika triaged this task as High priority.Jun 29 2018, 9:55 PM
• Niharika subscribed.

Triaging this as high priority. We're being forced to V:2 every patch for PageTriage extension to deploy it.

Halfak edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Jul 2 2018, 4:50 PM
Halfak moved this task from Unsorted to Backlog/Lift Wing on the Machine-Learning-Team board.
gerritbot added a comment.Jul 12 2018, 10:15 AM

Change 445380 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Inject tests/selenium/LocalSettings.php

https://gerrit.wikimedia.org/r/445380

hashar mentioned this in rQUIBBLEe500d486d644: Inject tests/selenium/LocalSettings.php.Jul 12 2018, 10:17 AM
hashar added a comment.EditedJul 16 2018, 8:53 AM

I have a patch for Quibble to make it include the tests/selenium/LocalSettings.php file which injects:

<?php
$wgOresWikiId = 'testwiki';
$wgOresModels = [
	'damaging' => [ 'enabled' => true ],
	'goodfaith' => [ 'enabled' => true ],
	'reverted' => [ 'enabled' => false ],
	'wp10' => [ 'enabled' => false ],
	'draftquality' => [ 'enabled' => false ]
];

When running the Selenium test, debug log shows it is hitting https://ores.wikimedia.org/v3/scores/testwiki/?format=json which yields:

{
    "testwiki": {
        "models": {
            "damaging": { "version": "0.0.3" },
            "goodfaith": { "version": "0.0.3" },
            "reverted": { "version": "0.0.3" }
        }
    }
}

but eventually in the logs:

[ORES] Error in ChangesListHookHandler: No model available for [damaging]

It comes from the SqlModelLookup class and apparently the model has not been populated in the database when running update.php:

oresm_idoresm_nameoresm_versionoresm_is_current
1reverted0.0.31
hashar mentioned this in T199644: Selenium failures on wmf-quibble-vendor-mysql-*-docker jobs for gate and submit.Jul 16 2018, 9:19 AM
hashar mentioned this in rQUIBBLEe94268ebab95: Inject tests/selenium/LocalSettings.php.Jul 16 2018, 10:18 AM
gerritbot added a comment.Jul 16 2018, 10:22 AM

Change 445971 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Inject tests/selenium/LocalSettings.php

https://gerrit.wikimedia.org/r/445971

gerritbot added a comment.Jul 16 2018, 10:23 AM

Change 445380 abandoned by Hashar:
Inject tests/selenium/LocalSettings.php

Reason:
Better have it done in mediawiki/core via includes/DevelopmentSettings.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /445971

https://gerrit.wikimedia.org/r/445380

hashar added a comment.Jul 16 2018, 10:33 AM

I thought about loadig test/selenium/LocalSettings.php files from Quibble, but it is much better to do it via mediawiki/core includes/DevelopmentSettings.php:

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/445971/

So we can then craft a patch for the ORES extension that has a Depends-On: I71013125987bb49426a4807dcefc1c06175b6663.

Note: we have the same use case for VisualEditor to set a $wg variable. That is for T199644 which needs VisualEditor to ship a LocalSettings.php ( https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/VisualEditor/+/445963/ ).

gerritbot added a comment.Jul 16 2018, 10:54 AM

Change 445976 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/ORES@master] (DO NOT SUBMIT) Test change for T198201

https://gerrit.wikimedia.org/r/445976

SBisson subscribed.Jul 16 2018, 3:56 PM

I think this is causing the failure we get on this patch: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageTriage/+/443856/

Is there a workaround for now?

hashar added a comment.Jul 16 2018, 4:04 PM

@SBisson nop it is broken for good. There are a set of patches that are pending review / merge though.

gerritbot added a comment.Jul 16 2018, 7:41 PM

Change 446095 had a related patch set uploaded (by MaxSem; owner: MaxSem):
[mediawiki/extensions/ORES@master] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/446095

gerritbot added a comment.Jul 16 2018, 7:50 PM

Change 446095 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/446095

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)).Jul 16 2018, 8:00 PM
hashar mentioned this in T196960: Quibble should have a way for extensions to opt out of core selenium browser tests.Jul 17 2018, 4:31 PM
gerritbot added a comment.Jul 18 2018, 10:39 PM

Change 446701 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Change environment when running Selenium tests

https://gerrit.wikimedia.org/r/446701

hashar mentioned this in rQUIBBLE99cfbf6a9bd1: Change environment when running Selenium tests.Jul 18 2018, 10:48 PM
hashar moved this task from Backlog to In progress on the Quibble board.Jul 19 2018, 2:49 PM
gerritbot added a comment.Aug 24 2018, 11:29 AM

Change 455141 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate ORES to Quibble

https://gerrit.wikimedia.org/r/455141

gerritbot added a comment.Aug 24 2018, 11:31 AM

Change 455141 merged by jenkins-bot:
[integration/config@master] Migrate ORES to Quibble

https://gerrit.wikimedia.org/r/455141

gerritbot added a comment.Aug 27 2018, 5:03 PM

Change 446701 abandoned by Hashar:
Change environment when running Selenium tests

https://gerrit.wikimedia.org/r/446701

zeljkofilipin mentioned this in T188742: Run tests daily targeting beta cluster for all repositories with Selenium tests.Sep 13 2018, 4:23 PM
hashar removed hashar as the assignee of this task.Sep 24 2018, 4:53 PM

The test has been backed up, the reason is Quibble does not include tests/browser/LocalSettings.php, it also run wdio tests from mediawiki/core. There is a work in progress to phase out tests/browser/LocalSettings.php and to run wdio directly from the root of the extension.

Meanwhile it is no more a blocker to have ORES patches merged by CI, I lack time to investigate a proper solution right now. Really the proper solution would be to install the ORES backend or at least a fake server and then on CI configure the extension to point to that (instead of production).

zeljkofilipin awarded a token.Sep 25 2018, 11:06 AM
gerritbot added a comment.Oct 31 2018, 2:19 PM

Change 445976 abandoned by Hashar:
(DO NOT SUBMIT) Test change for T198201

https://gerrit.wikimedia.org/r/445976

gerritbot added a comment.Oct 31 2018, 2:27 PM

Change 445971 abandoned by Hashar:
Inject tests/selenium/LocalSettings.php

Reason:
I don't think this is needed anymore.

https://gerrit.wikimedia.org/r/445971

• Niharika unsubscribed.Oct 31 2018, 6:26 PM
gerritbot added a comment.Dec 18 2018, 5:23 PM

Change 480544 had a related patch set uploaded (by Paladox; owner: MaxSem):
[mediawiki/extensions/ORES@REL1_31] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/480544

gerritbot added a comment.Dec 18 2018, 5:37 PM

Change 480544 merged by jenkins-bot:
[mediawiki/extensions/ORES@REL1_31] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/480544

Krinkle removed a parent task: T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.Feb 7 2019, 1:40 AM
Krinkle closed this task as Declined.Feb 7 2019, 1:55 AM
Krinkle subscribed.

After T199116, this can probably be solved relatively easily by making using wdio config hooks to perform an action before the tests begin, and after they end; namely to add/remove a line to the LocalSettings.php file. This would be similar to how we do this in PHPUnit and QUnit already, and does indeed require the tests to run from the same host as where the MediaWiki source resides.

This could later be expanded upon to by using a maintenance script start and end a mock database, which would resolve lots of other issues as well; such as the problem that all identifiers are non-deterministic and potentially clash due to rand. Alternatively we could have it as now without mock db, and use only a maintenance script to clean up afterwards (e.g. tracing all page, revisions and user accounts and hard-deleting them), but that seems hard to even just for core, and wouldn't doesn't scale well to all extensions and their own test cases.

Alternatively, rather than injecting LocalSettings, it might also be possible to phase out these overrides in favour of better defaults and automatic discovery of sibling services at run-time. See T199939. If after that we still find bits that are impossible or too hard to quickly migrate, we can use wgWikimediaJenkinsCI which is slightly less bad than having to inject PHP code (T200650).

Given that we've managed to survive for a year without feature in wdio, I assume this is no longer causing breakage or preventing anything.

Any extensions looking to write new tests that might need certain settings can use the above approaches. Declining as such, since per Hashar it seems there are no plans to add this feature from the ruby stack to the wdio stack.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL