Page MenuHomePhabricator

Add basic browser tests: external surveys
Closed, ResolvedPublic2 Story Points

Related Objects

Event Timeline

Jdlrobson claimed this task.
Jdlrobson removed Jdlrobson as the assignee of this task.
Jdlrobson raised the priority of this task from to High.
Jdlrobson updated the task description. (Show Details)
bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-54-28-Days-Later board.

Change 233933 had a related patch set uploaded (by Bmansurov):
Add browser tests for external surveys

https://gerrit.wikimedia.org/r/233933

Change 233930 had a related patch set uploaded (by Bmansurov):
Add an option to load a specific survey

https://gerrit.wikimedia.org/r/233930

Change 234710 had a related patch set uploaded (by Bmansurov):
Add browser tests for external surveys

https://gerrit.wikimedia.org/r/234710

Change 234710 abandoned by Bmansurov:
Add browser tests for external surveys

Reason:
https://gerrit.wikimedia.org/r/#/c/233933/2

https://gerrit.wikimedia.org/r/234710

All tests pass locally when I disable the existing step "And the page has fully loaded". I'm not sure why that step doesn't work on my computer. And remotely, I get different errors. Does anyone know howto fix it? @phuedx, @rmoen, @Jdlrobson?

(I'm in a planning meeting all morning so will look at this later but it's possible you've just uncovered a new bug unrelated to your patch - I'd suggest you try adding the browser tests before your patch to see if the expected tests fail in those circumstance)

@Jdlrobson, thanks. Either the bucketing logic seems off or I'm not understanding it correctly. Say, my example survey has 0.1 coverage, then I'm always assigned to the control bucket, even though I should be assigned to the bucket A 10% of the time. No?

It's possible. Write some unit tests to test it more thoroughly. I believe in you Baha! You'll work it out :)

phuedx added a comment.Sep 1 2015, 9:58 AM

Either the bucketing logic seems off or I'm not understanding it correctly. Say, my example survey has 0.1 coverage, then I'm always assigned to the control bucket, even though I should be assigned to the bucket A 10% of the time. No?

The current bucketing mechanism operates like "10% of users see the survey all of the 100% of the time – until they interact with it".

phuedx added a comment.Sep 1 2015, 2:55 PM

@Jdlrobson: What needs more work?

See my review: https://gerrit.wikimedia.org/r/#/c/233933/ - issues with browser tests

Thanks, I've replied to your comment, @Jdlrobson.
Thanks for the clarification, @phuedx.

Jdlrobson edited a custom field.Sep 2 2015, 4:33 PM

I can replicate the issue that Jenkins has locally. I will post a follow-up patch for your inspection when you wake up :)

@bmansurov see https://gerrit.wikimedia.org/r/235618 - primarily goal here is to make the feature files as readable as possible. I hope things become clearer with this patch. Feel free to squash it/take ideas from it/whatever :)

@Jdlrobson, thanks! I've squashed the patches.

Change 233930 merged by jenkins-bot:
Add an option to load a specific survey

https://gerrit.wikimedia.org/r/233930

(Will move back if it doesn't merge for whatever reason)

Change 233933 merged by jenkins-bot:
Add browser tests for external surveys

https://gerrit.wikimedia.org/r/233933

Nice work both.

Jdlrobson closed this task as Resolved.Sep 4 2015, 12:04 AM
Jdlrobson moved this task from Ready for Signoff to Done on the Reading-Web-Sprint-55-π board.

Thanks @bmansurov!