Investigate QuickSurveys browser tests failures
Closed, ResolvedPublic

Description

For example
https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/11/testReport/junit/(root)/External%20surveys/External_survey_is_not_present_when_dismissed/

I wonder if this is failing because now there is a third survey (drink-survey) that is not being declared as dismissed. Worth someone investigating. I'm concerned there may be an issue with our sampling as that quick survey is not meant to show at all (0 sample)

You'll probably need to run this browser test locally various times until you can replicate the issue.

Jdlrobson moved this task to To Do on the Reading-Web-Sprint-56-Four Lions board.
Jdlrobson added a subscriber: Jdlrobson.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2015, 9:27 PM
phuedx added a subscriber: phuedx.Sep 24 2015, 4:04 PM

I haven't been able to get this test to fail locally… yet.

Jdlrobson added a subscriber: dduvall.

Tested against beta labs and no luck @dduvall any ideas? Jenkins is not giving us much to work on with the broken Sauce Labs urls..

I really have no clue what's going on here. T105589 needs to get fixed asap to unblock us.

greg changed the title from "Investigate beta labs QuickSurveys browser tests failures" to "Investigate QuickSurveys browser tests failures".Oct 1 2015, 3:18 PM
phuedx added a comment.EditedOct 13 2015, 2:02 PM

Might this be caused by T114485?

Change 246801 had a related patch set uploaded (by Jdlrobson):
QA: Don't ask. Just merge.

https://gerrit.wikimedia.org/r/246801

Tested against beta labs and no luck @dduvall any ideas? Jenkins is not giving us much to work on with the broken Sauce Labs urls..

Bumping since this is still an issue. cc: @dduvall

It's hard for me to comment without knowing more about the case that you're testing and at least a bit of the underlying implementation. Is there a way we can simulate the xhr latency so as to reproduce this locally?

I have created a Jenkins job to test 246801 (diff from the original job).

Results:

I still do not agree with using sleep, see T48887: Remove "sleep" from tests.

phuedx added a comment.Nov 6 2015, 1:44 PM

@zeljkofilipin: All I see is "No permission to view config history". Is this because you're using super secret sauce to get the test passing? ;)

@phuedx I think you have to be logged in to jenkins to see job config history, but this is the only change in the job shell script (applying the gerrit patch):

git fetch https://gerrit.wikimedia.org/r/mediawiki/extensions/QuickSurveys refs/changes/01/246801/1 && git checkout FETCH_HEAD
phuedx added a comment.Nov 6 2015, 2:03 PM

I think you have to be logged in to jenkins to see job config history…

Ha! Of course I do.

phuedx changed the task status from "Open" to "Stalled".Dec 15 2015, 1:22 PM
phuedx added a subscriber: zeljkofilipin.

@Jdlrobson: The commit message of 246801 needs rewording and there's a sleep command in there that @zeljkofilipin doesn't agree with. The test does pass though…

Jdlrobson changed the task status from "Stalled" to "Open".Dec 17 2015, 10:53 PM

Came back to look at the patch https://gerrit.wikimedia.org/r/246801
No rest for the wicked.

Would be great to have a review but please do not lose any sleep over it, I know this bug has been tiring.

Change 246801 merged by jenkins-bot:
QA: Ensure survey code loaded before testing if survey is visible

https://gerrit.wikimedia.org/r/246801

bmansurov closed this task as "Resolved".Dec 21 2015, 12:26 PM
bmansurov added a subscriber: bmansurov.

Closing since the last QuickSurvey browser test failure happened on December 15, a day before the above patch was merged.