Page MenuHomePhabricator

selenium tests are flaky under php7 on TemplateWizard
Closed, ResolvedPublic

Description

Browser tests from T199580 failing very often on php7 jobs, but always running on hhvm.
There is on pass on php7.1 in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TemplateWizard/+/476076/

1) TemplateWizard basic use of TemplateWizard:
1 == 0
running chrome
AssertionError: 1 == 0
    at assertVisibleElementCount (/workspace/src/extensions/TemplateWizard/tests/selenium/specs/TemplateWizard.js:23:9)
    at Context.<anonymous> (/workspace/src/extensions/TemplateWizard/tests/selenium/specs/TemplateWizard.js:59:3)
    at Promise.F (/workspace/src/node_modules/core-js/library/modules/_export.js:36:28)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I've had a look at this a couple of times but have not yet figured anything out. I've not been able to replicate it locally. We could just remove the failing bits, so that unrelated patches aren't repeatedly held up by this.

I can replicate this pretty reliably on my local VM (Debian Stretch; PHP 7.0.30).

After creating a template with parameters it can take a long time (sometimes over 1 minute) for those parameters to appear when searching for the template using the "Insert a template" wizard. You just see "No parameters could be determined for this template due to absence of TemplateData documentation. You may insert the template without parameters."

I have seen this both when running the Selenium scripts but also when creating a template normally via the browser.

I have attached what was written to $wgDebugLogFile during the last failed selenium run, if that helps. Let me know if there is any other information I could collect.

This seems to be stalled. Pinging @zeljkofilipin in case he has any ideas.

Just to make sure I've understood the problem: for TemplateWizard patches (example: 496350), quibble-vendor-mysql-php70-docker (example: 18061) and quibble-vendor-mysql-php71-docker (example: 7413) Jenkins jobs fail. At the same time quibble-vendor-mysql-hhvm-docker (example: 39575) and quibble-vendor-mysql-php72-docker (example: 7162) jobs pass. All mentioned jobs run the same tests.

TemplateWizard basic use of TemplateWizard test fails with AssertionError: 1 == 0.

Tests passing on one platform and failing on another are usually (but not always) a timing problem.

From a quick look at the test, I would suggest refactoring it. It's not following the structure I've documented at Selenium/Node.js/Write. There's also plenty of examples at Selenium/Node.js#write-tests.

I would suggest that a member of the team pairs with me on refactoring and fixing the test.

Thanks @zeljkofilipin! We'll need to prioritize this within our team. We are finishing up a couple of projects in the next few weeks. I wonder if @Niharika thinks we might have time to have @Samwilson pair up on this?

If anybody is coming to Wikimania-Hackathon-2019 I'll be there and available for pairing.

If anybody is coming to Wikimania-Hackathon-2019 I'll be there and available for pairing.

Did you mean the Wikimedia Hackathon? That will be a good opportunity to pair on this task, if Sam wants to.

I'll be at the hackathon, and that sounds great @zeljkofilipin! Thanks.

Change 502390 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] View test template in Selenium test, to confirm its contents

https://gerrit.wikimedia.org/r/502390

This seems to be an issue with the job queue, and if we load the test template's page before doing anything else then the templatedata is populated correctly and the rest of the tests pass.

Maybe this is a hack, but it does sort of make sense. The above patch adds a check for the templatedata on the test template's page.

I was wrong. The templatedata is still not being returned by the API when the TemplateWizard test is run. It works locally, and when I set $wgJobRunRate = 0; it fails with the same error that's occurring in CI.

Change 502399 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] Remove failing part of test

https://gerrit.wikimedia.org/r/502399

Change 502399 merged by jenkins-bot:
[mediawiki/extensions/TemplateWizard@master] Remove failing part of test

https://gerrit.wikimedia.org/r/502399

Change 503262 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] Restore front-end tests

https://gerrit.wikimedia.org/r/503262

Change 502390 merged by Samwilson:
[mediawiki/extensions/TemplateWizard@master] View test template in Selenium test, to confirm its contents

https://gerrit.wikimedia.org/r/502390

Change 503262 merged by jenkins-bot:
[mediawiki/extensions/TemplateWizard@master] Restore front-end tests

https://gerrit.wikimedia.org/r/503262