Page MenuHomePhabricator
Create Task
Maniphest T211470

selenium tests are flaky under php7 on TemplateWizard
Closed, ResolvedPublic
Actions

Assigned To
Samwilson
Authored By
Umherirrender
Dec 8 2018, 1:28 PM
Tags
Referenced Files
F28128031: templatewizard_selenium_dwalden.log
Feb 5 2019, 6:11 PM
Subscribers
• aezell
Aklapper
dom_walden
Jdforrester-WMF
kaldari
Mooeypoo
• Niharika
View All 11 Subscribers
Tokens
"The World Burns" token, awarded by zeljkofilipin.

Description

Browser tests from T199580 failing very often on php7 jobs, but always running on hhvm.
There is on pass on php7.1 in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TemplateWizard/+/476076/

1) TemplateWizard basic use of TemplateWizard:
1 == 0
running chrome
AssertionError: 1 == 0
    at assertVisibleElementCount (/workspace/src/extensions/TemplateWizard/tests/selenium/specs/TemplateWizard.js:23:9)
    at Context.<anonymous> (/workspace/src/extensions/TemplateWizard/tests/selenium/specs/TemplateWizard.js:59:3)
    at Promise.F (/workspace/src/node_modules/core-js/library/modules/_export.js:36:28)

Details

SubjectRepoBranchLines +/-
Restore front-end testsmediawiki/extensions/TemplateWizardmaster+50 -3
View test template in Selenium test, to confirm its contentsmediawiki/extensions/TemplateWizardmaster+4 -0
Remove failing part of testmediawiki/extensions/TemplateWizardmaster+2 -49
Customize query in gerrit

Related Objects

Event Timeline

Umherirrender created this task.Dec 8 2018, 1:28 PM
Restricted Application added a project: Community-Tech. · View Herald TranscriptDec 8 2018, 1:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle moved this task from Backlog to Wikimedia production on the PHP 7.0 support board.Dec 9 2018, 6:46 PM
Krinkle moved this task from Backlog to Wikimedia production on the PHP 7.1 support board.
xSavitar subscribed.Dec 18 2018, 5:37 PM
• Niharika moved this task from New & TBD Tickets to Engineering Backlog on the Community-Tech board.Jan 15 2019, 5:59 AM
• Niharika added subscribers: Samwilson, Mooeypoo, • Niharika.

@Samwilson @Mooeypoo Flagging this ticket for you both.

Samwilson added a comment.Jan 15 2019, 6:38 PM

I've had a look at this a couple of times but have not yet figured anything out. I've not been able to replicate it locally. We could just remove the failing bits, so that unrelated patches aren't repeatedly held up by this.

dom_walden subscribed.Feb 5 2019, 6:11 PM

I can replicate this pretty reliably on my local VM (Debian Stretch; PHP 7.0.30).

After creating a template with parameters it can take a long time (sometimes over 1 minute) for those parameters to appear when searching for the template using the "Insert a template" wizard. You just see "No parameters could be determined for this template due to absence of TemplateData documentation. You may insert the template without parameters."

I have seen this both when running the Selenium scripts but also when creating a template normally via the browser.

I have attached what was written to $wgDebugLogFile during the last failed selenium run, if that helps. Let me know if there is any other information I could collect.

templatewizard_selenium_dwalden.log1 MBDownload

kaldari added subscribers: zeljkofilipin, kaldari.Mar 13 2019, 4:29 AM

This seems to be stalled. Pinging @zeljkofilipin in case he has any ideas.

zeljkofilipin added a comment.Mar 15 2019, 3:12 PM

Just to make sure I've understood the problem: for TemplateWizard patches (example: 496350), quibble-vendor-mysql-php70-docker (example: 18061) and quibble-vendor-mysql-php71-docker (example: 7413) Jenkins jobs fail. At the same time quibble-vendor-mysql-hhvm-docker (example: 39575) and quibble-vendor-mysql-php72-docker (example: 7162) jobs pass. All mentioned jobs run the same tests.

TemplateWizard basic use of TemplateWizard test fails with AssertionError: 1 == 0.

Tests passing on one platform and failing on another are usually (but not always) a timing problem.

From a quick look at the test, I would suggest refactoring it. It's not following the structure I've documented at Selenium/Node.js/Write. There's also plenty of examples at Selenium/Node.js#write-tests.

I would suggest that a member of the team pairs with me on refactoring and fixing the test.

zeljkofilipin added a project: User-zeljkofilipin.Mar 15 2019, 3:13 PM
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
• aezell subscribed.Mar 15 2019, 4:32 PM

Thanks @zeljkofilipin! We'll need to prioritize this within our team. We are finishing up a couple of projects in the next few weeks. I wonder if @Niharika thinks we might have time to have @Samwilson pair up on this?

zeljkofilipin added a comment.Mar 15 2019, 5:35 PM

If anybody is coming to Wikimania-Hackathon-2019 I'll be there and available for pairing.

zeljkofilipin moved this task from Deep work 🌊 to Waiting 🏹 on the User-zeljkofilipin board.Mar 15 2019, 5:35 PM
Jdforrester-WMF subscribed.Mar 16 2019, 9:03 PM
• Niharika added a comment.Mar 16 2019, 9:47 PM
In T211470#5027629, @zeljkofilipin wrote:

If anybody is coming to Wikimania-Hackathon-2019 I'll be there and available for pairing.

Did you mean the Wikimedia Hackathon? That will be a good opportunity to pair on this task, if Sam wants to.

zeljkofilipin added a comment.Mar 18 2019, 9:28 AM

Oops, correct. I will be at Wikimedia-Hackathon-2019. I will not be at Wikimania-Hackathon-2019.

Samwilson added a comment.Mar 31 2019, 11:59 PM

I'll be at the hackathon, and that sounds great @zeljkofilipin! Thanks.

zeljkofilipin awarded a token.Apr 1 2019, 10:17 AM
kaldari added a comment.Apr 1 2019, 6:49 PM

@Samwilson - Do you want to disable these tests in the meantime? They seem to be causing trouble: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TemplateWizard/+/499879/

gerritbot added a comment.Apr 9 2019, 3:21 AM

Change 502390 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] View test template in Selenium test, to confirm its contents

https://gerrit.wikimedia.org/r/502390

gerritbot added a project: Patch-For-Review.Apr 9 2019, 3:21 AM
Samwilson added a comment.Apr 9 2019, 3:40 AM

This seems to be an issue with the job queue, and if we load the test template's page before doing anything else then the templatedata is populated correctly and the rest of the tests pass.

Maybe this is a hack, but it does sort of make sense. The above patch adds a check for the templatedata on the test template's page.

Samwilson added a project: Community-Tech-Sprint.Apr 9 2019, 3:41 AM
Samwilson claimed this task.Apr 9 2019, 4:16 AM
Samwilson moved this task from Ready to Needs Review/Feedback on the Community-Tech-Sprint board.
Samwilson moved this task from Needs Review/Feedback to In Development on the Community-Tech-Sprint board.EditedApr 9 2019, 5:04 AM

I was wrong. The templatedata is still not being returned by the API when the TemplateWizard test is run. It works locally, and when I set $wgJobRunRate = 0; it fails with the same error that's occurring in CI.

gerritbot added a comment.Apr 9 2019, 7:02 AM

Change 502399 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] Remove failing part of test

https://gerrit.wikimedia.org/r/502399

Samwilson moved this task from In Development to Needs Review/Feedback on the Community-Tech-Sprint board.Apr 9 2019, 7:49 AM
gerritbot added a comment.Apr 9 2019, 8:48 AM

Change 502399 merged by jenkins-bot:
[mediawiki/extensions/TemplateWizard@master] Remove failing part of test

https://gerrit.wikimedia.org/r/502399

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.25; 2019-04-09).Apr 9 2019, 9:01 AM
Samwilson moved this task from Needs Review/Feedback to In Development on the Community-Tech-Sprint board.Apr 12 2019, 12:36 AM
Krinkle moved this task from Untriaged to Other on the PHP 7.2 support board.Apr 16 2019, 12:35 AM
• Niharika removed a project: Community-Tech-Sprint.Apr 18 2019, 11:16 PM
zeljkofilipin moved this task from Waiting 🏹 to Backlog 🪒 on the User-zeljkofilipin board.Jul 16 2019, 1:08 PM
Umherirrender added a project: ci-test-error.Aug 4 2019, 6:07 PM
Krinkle removed projects: PHP 7.1 support, PHP 7.0 support.Sep 25 2019, 4:56 PM
gerritbot added a comment.Oct 8 2019, 8:27 AM

Change 503262 had a related patch set uploaded (by Samwilson; owner: Samwilson):
[mediawiki/extensions/TemplateWizard@master] Restore front-end tests

https://gerrit.wikimedia.org/r/503262

gerritbot added a comment.Oct 10 2019, 1:12 AM

Change 502390 merged by Samwilson:
[mediawiki/extensions/TemplateWizard@master] View test template in Selenium test, to confirm its contents

https://gerrit.wikimedia.org/r/502390

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.2; 2019-10-15).Oct 10 2019, 2:00 AM
gerritbot added a comment.Oct 10 2019, 6:00 PM

Change 503262 merged by jenkins-bot:
[mediawiki/extensions/TemplateWizard@master] Restore front-end tests

https://gerrit.wikimedia.org/r/503262

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2019, 6:10 PM
Jdforrester-WMF closed this task as Resolved.Oct 10 2019, 10:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL