Page MenuHomePhabricator

Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server)
Closed, DeclinedPublic

Description

When running the selenium browser tests for ORES, the CI build fails with:

  ORES
      1) filters are present
      

1 failing (3s)

1) ORES filters are present:
false == true
AssertionError: false == true

The exception log shows:

/index.php?action=history&title=0.45299226547790816-%C3%B6%C3%A4%C3%BC-%E2%99%A0%E2%99%A3%E2%99%A5%E2%99%A6   RuntimeException from line 94 of extensions/ORES/includes/ORESService.php: Failed to make ORES request to [https://ores.wikimedia.org/v3/scores/wikidb/?models=damaging%7Cgoodfaith&revids=4&precache=1&format=json], There was a problem during the HTTP request: 404 NOT FOUND
#0 extensions/ORES/includes/ScoreFetcher.php(57): ORES\ORESService->request(array, array)
#1 extensions/ORES/includes/FetchScoreJob.php(89): ORES\ScoreFetcher->getScores(integer, array, boolean, array)
#2 includes/jobqueue/JobRunner.php(296): ORES\FetchScoreJob->run()
#3 includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(ORES\FetchScoreJob, Wikimedia\Rdbms\LBFactorySimple, BufferingStatsdDataFactory, integer)
#4 includes/MediaWiki.php(1007): JobRunner->run(array)
#5 includes/MediaWiki.php(990): MediaWiki->triggerSyncJobs(integer, MediaWiki\Logger\LegacyLogger)
#6 includes/MediaWiki.php(914): MediaWiki->triggerJobs()
#7 includes/MediaWiki.php(729): MediaWiki->restInPeace(string, boolean)
#8 includes/MediaWiki.php(752): MediaWiki->{closure}()
#9 includes/MediaWiki.php(557): MediaWiki->doPostOutputShutdown(string)
#10 index.php(42): MediaWiki->run()
#11 {main}

Which is due to the CI setup trying to reach the default ORES:

extension.json
{
    "config": {
        "OresBaseUrl": {
            "value": "https://ores.wikimedia.org/"
        },
}

Thus the wiki tries to reach https://ores.wikimedia.org/v3/scores/wikidb/ which obviously yields nothing.

I don't think we can easily spawn an ORES backend in CI, at least not today. But surely we can find a way to skip the test entirely when being the webdriver tests are run from a patchset upload.

Repro: check experimental on dummy change https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ORES/+/436523/

Event Timeline

hashar created this task.Jun 26 2018, 11:40 AM
Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptJun 26 2018, 11:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 442085 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/ORES@master] selenium: skip test on CI

https://gerrit.wikimedia.org/r/442085

Change 442085 abandoned by Hashar:
selenium: skip test on CI

Reason:
So that skips the test when being run under Jenkins and if triggered from Zuul. Then some mediawiki/core selenium tests end up failling

  • Special:RecentChanges shows page creation
  • Special:Watchlist should show page with new edit

So I guess instead we need to have ORES to be disabled by default until a proper URL is set up.

https://gerrit.wikimedia.org/r/442085

The patch would skip the ORES selenium tests just fine however mediawiki/core tests end up failing.

Special:RecentChanges shows page creation
Special:Watchlist should show page with new edit

So I guess we need to find a way to disable ORES by default. Of course for wikimedia that would be need to be explicit set in operations/mediawiki-config and if we have a daily browser test, it would need a tests/selenium/LocalSettings.php to set the URL.

awight added a subscriber: awight.

@hashar Thanks for the investigation! Can you assign this an urgency? I don't understand whether we're currently breaking CI for everyone, if it's a warning or error, etc.

See also our task to integrate a working ORES service under docker: T195073. If that would be a solution, I'd prefer to put our energy into finishing that integration.

Right now patches sent to mediawiki/extensions/ORES do not run the Selenium tests. I am in a process of migrating every extensions to use a new set of jobs. They run under Docker and do run the Selenium tests (mediawiki/core ones + extensions ones). The migration task is T183512.

The new job can be triggered by using check experimental. The container installs MediaWiki and the extension, spawns a webserver then run the Selenium tests against the local webserver. And of course, there is no ORES server available.

I guess if somehow had $wgOresBaseUrl to default to null/false and thus entirely skip ORES until it is properly setup, that would work around my use case :]

Adding end to end tests of ORES extensions with an ORES backend is a different task. That will be accomplished as part of new deployment pipeline ( T170481 ), currently scope limited to Math / Mathoid.

@hashar Okay I see, we're blocking the migration, maybe we have a few weeks to act?

So does it make sense for us to integrate our dockerfile with your work, or do you have a universal docker image which will be used for all extensions?

this is actually easily fixable.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJun 27 2018, 4:59 PM

It seems tests/selenium/LocalSettings.php is not being loaded, otherwise It would hit https://ores.wikimedia.org/v3/scores/teswiki and not https://ores.wikimedia.org/v3/scores/wikidb (See the related codebase)

Ladsgroup removed Ladsgroup as the assignee of this task.Jun 27 2018, 7:24 PM

Could this be what's causing https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6154/console ? Related patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/440018

Fails on "Special:RecentChanges shows page creation". In previous patchsets we were able to make this test pass after rechecking.

@hashar Okay I see, we're blocking the migration, maybe we have a few weeks to act?

So does it make sense for us to integrate our dockerfile with your work, or do you have a universal docker image which will be used for all extensions?

We are only starting the grand plan to migrate services to Docker with Math/Mathoid being the first use case. Each service will have its own docker image which will be build/used by CI as well eventually. But all of that will take a few more months/quarters to achieve.

It seems tests/selenium/LocalSettings.php is not being loaded, otherwise It would hit https://ores.wikimedia.org/v3/scores/teswiki and not https://ores.wikimedia.org/v3/scores/wikidb (See the related codebase)

Yup that is it. Quibble does not load tests/selenium/LocalSettings.php! I need to implement support for that file.

Could this be what's causing https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6154/console ? Related patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/440018

Fails on "Special:RecentChanges shows page creation". In previous patchsets we were able to make this test pass after rechecking.

The job triggered for PageTriage does not have ORES injected as a dependency. So that is absolutely unrelated. That "Special:RecentChanges shows page creation" has a different cause and deserves a different task :]

hashar claimed this task.

That is a Quibble issue. Thank you @Ladsgroup !

hashar renamed this task from ORES webdriver.io selenium test fail on CI due to lack of ORES server to Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server).Jun 27 2018, 8:36 PM
Niharika triaged this task as High priority.Jun 29 2018, 9:55 PM
Niharika added a subscriber: Niharika.

Triaging this as high priority. We're being forced to V:2 every patch for PageTriage extension to deploy it.

Change 445380 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Inject tests/selenium/LocalSettings.php

https://gerrit.wikimedia.org/r/445380

hashar added a comment.EditedJul 16 2018, 8:53 AM

I have a patch for Quibble to make it include the tests/selenium/LocalSettings.php file which injects:

<?php
$wgOresWikiId = 'testwiki';
$wgOresModels = [
	'damaging' => [ 'enabled' => true ],
	'goodfaith' => [ 'enabled' => true ],
	'reverted' => [ 'enabled' => false ],
	'wp10' => [ 'enabled' => false ],
	'draftquality' => [ 'enabled' => false ]
];

When running the Selenium test, debug log shows it is hitting https://ores.wikimedia.org/v3/scores/testwiki/?format=json which yields:

{
    "testwiki": {
        "models": {
            "damaging": { "version": "0.0.3" },
            "goodfaith": { "version": "0.0.3" },
            "reverted": { "version": "0.0.3" }
        }
    }
}

but eventually in the logs:

[ORES] Error in ChangesListHookHandler: No model available for [damaging]

It comes from the SqlModelLookup class and apparently the model has not been populated in the database when running update.php:

oresm_idoresm_nameoresm_versionoresm_is_current
1reverted0.0.31

Change 445971 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Inject tests/selenium/LocalSettings.php

https://gerrit.wikimedia.org/r/445971

Change 445380 abandoned by Hashar:
Inject tests/selenium/LocalSettings.php

Reason:
Better have it done in mediawiki/core via includes/DevelopmentSettings.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /445971

https://gerrit.wikimedia.org/r/445380

I thought about loadig test/selenium/LocalSettings.php files from Quibble, but it is much better to do it via mediawiki/core includes/DevelopmentSettings.php:

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/445971/

So we can then craft a patch for the ORES extension that has a Depends-On: I71013125987bb49426a4807dcefc1c06175b6663.

Note: we have the same use case for VisualEditor to set a $wg variable. That is for T199644 which needs VisualEditor to ship a LocalSettings.php ( https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/VisualEditor/+/445963/ ).

Change 445976 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/ORES@master] (DO NOT SUBMIT) Test change for T198201

https://gerrit.wikimedia.org/r/445976

I think this is causing the failure we get on this patch: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageTriage/+/443856/

Is there a workaround for now?

@SBisson nop it is broken for good. There are a set of patches that are pending review / merge though.

Change 446095 had a related patch set uploaded (by MaxSem; owner: MaxSem):
[mediawiki/extensions/ORES@master] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/446095

Change 446095 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/446095

Change 446701 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Change environment when running Selenium tests

https://gerrit.wikimedia.org/r/446701

hashar moved this task from Backlog to In progress on the Quibble board.Jul 19 2018, 2:49 PM

Change 455141 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate ORES to Quibble

https://gerrit.wikimedia.org/r/455141

Change 455141 merged by jenkins-bot:
[integration/config@master] Migrate ORES to Quibble

https://gerrit.wikimedia.org/r/455141

Change 446701 abandoned by Hashar:
Change environment when running Selenium tests

https://gerrit.wikimedia.org/r/446701

hashar removed hashar as the assignee of this task.Sep 24 2018, 4:53 PM

The test has been backed up, the reason is Quibble does not include tests/browser/LocalSettings.php, it also run wdio tests from mediawiki/core. There is a work in progress to phase out tests/browser/LocalSettings.php and to run wdio directly from the root of the extension.

Meanwhile it is no more a blocker to have ORES patches merged by CI, I lack time to investigate a proper solution right now. Really the proper solution would be to install the ORES backend or at least a fake server and then on CI configure the extension to point to that (instead of production).

Change 445976 abandoned by Hashar:
(DO NOT SUBMIT) Test change for T198201

https://gerrit.wikimedia.org/r/445976

Change 445971 abandoned by Hashar:
Inject tests/selenium/LocalSettings.php

Reason:
I don't think this is needed anymore.

https://gerrit.wikimedia.org/r/445971

Niharika removed a subscriber: Niharika.Oct 31 2018, 6:26 PM

Change 480544 had a related patch set uploaded (by Paladox; owner: MaxSem):
[mediawiki/extensions/ORES@REL1_31] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/480544

Change 480544 merged by jenkins-bot:
[mediawiki/extensions/ORES@REL1_31] Disable failing test that prevents merges across WMF-deployed repos

https://gerrit.wikimedia.org/r/480544

Krinkle closed this task as Declined.Thu, Feb 7, 1:55 AM
Krinkle added a subscriber: Krinkle.

After T199116, this can probably be solved relatively easily by making using wdio config hooks to perform an action before the tests begin, and after they end; namely to add/remove a line to the LocalSettings.php file. This would be similar to how we do this in PHPUnit and QUnit already, and does indeed require the tests to run from the same host as where the MediaWiki source resides.

This could later be expanded upon to by using a maintenance script start and end a mock database, which would resolve lots of other issues as well; such as the problem that all identifiers are non-deterministic and potentially clash due to rand. Alternatively we could have it as now without mock db, and use only a maintenance script to clean up afterwards (e.g. tracing all page, revisions and user accounts and hard-deleting them), but that seems hard to even just for core, and wouldn't doesn't scale well to all extensions and their own test cases.

Alternatively, rather than injecting LocalSettings, it might also be possible to phase out these overrides in favour of better defaults and automatic discovery of sibling services at run-time. See T199939. If after that we still find bits that are impossible or too hard to quickly migrate, we can use wgWikimediaJenkinsCI which is slightly less bad than having to inject PHP code (T200650).

Given that we've managed to survive for a year without feature in wdio, I assume this is no longer causing breakage or preventing anything.

Any extensions looking to write new tests that might need certain settings can use the above approaches. Declining as such, since per Hashar it seems there are no plans to add this feature from the ruby stack to the wdio stack.