Page MenuHomePhabricator

CentralNotice bucket improvements, step 3 bis part 2: moar better cross-browser tests
Closed, ResolvedPublic1 Story Points

Description

Enable existing cross-browser tests on all available platforms, and check that we’re covering all the platforms that we have to.
Status:
More platforms now enabled, some still missing: https://gerrit.wikimedia.org/r/#/c/178714/ https://gerrit.wikimedia.org/r/#/c/178710/
Mobile test failing, needs mobile platform enabled and mobile redirect disabled for RecordImpression and BannerLoader. See #2317.
Add more cross-browser tests to all or most CentralNotice code paths.
Create an instance and write instance setup code to do cross-browser tests of CentralNotice code using shared fixtures. (This is required by the previous point. Also, if we find that we need an instance for testing actual banners, campaigns and Javascript not in CentralNotice, we could use the same one; see #2229.)
Bonus: we now have a CN cross-browser testing dashboard: https://integration.wikimedia.org/ci/view/BrowserTests/view/CentralNotice/

Event Timeline

atgo created this task.Jan 8 2015, 12:22 AM
atgo assigned this task to AndyRussG.
atgo raised the priority of this task from to Normal.
atgo updated the task description. (Show Details)
atgo edited a custom field.
atgo added subscribers: Aklapper, atgo, ellery.

Update:

  • I have a nefariastuous plan for cross-browser tests that use our test fixtures: write a cucumber test that runs and verifies the result of our QUnit tests! Bwahahahahaha... In theory we can run QUnit on test2.wikipedia.org, but there's an apparently bug blocking that right now. If we can't fix that, we can set up a simple labs instance of our own.

I think said plan is better than trying to create a test instance with the actual campaigns defined by the test fixtures--It'd have to have a lot of MW installs, since the test cases aren't meant to be set up simultaneously on a single infrastructure wiki, and it'd be complicated. Plus, I don't see any disadvantages to the plan, other than its evident evilosity. (I could very easily be wrong, though!)

Change 193361 had a related patch set uploaded (by AndyRussG):
WIP Created the first Android centralNotice Jenkins job

https://gerrit.wikimedia.org/r/193361

Change 193556 had a related patch set uploaded (by AndyRussG):
CentralNotice browser tests: even more platforms and browsers

https://gerrit.wikimedia.org/r/193556

Change 193552 had a related patch set uploaded (by AndyRussG):
Browser tests: add mobile tags

https://gerrit.wikimedia.org/r/193552

Change 193552 merged by Ssmith:
Browser tests: add mobile tags

https://gerrit.wikimedia.org/r/193552

Current status:

  • In the the CR pipeline are now patches to enabling a full, or quite-full-enough, slate of platforms for browser tests, including iphone, android and additional browsers on Windows and OSX.
  • I talked to @Krinkle about running our QUnit tests on multiple platforms. He's actually been working on a system specifically for doing exactly that via karma, Jenkins and SauceLabs, so no need make any cucumber tests. To ride that train, the main thing we have to do is make get grunt qunit to work for CentralNotice tests.

Since the second point will let us run through all the test fixtures on multiple browser platforms, I suggest that once those both points are through, we may call this card a day.

Change 193361 merged by jenkins-bot:
Created the first Android CentralNotice Jenkins job

https://gerrit.wikimedia.org/r/193361

Change 194773 had a related patch set uploaded (by AndyRussG):
CentralNotice: enable qunit-karma job

https://gerrit.wikimedia.org/r/194773

Change 197967 had a related patch set uploaded (by AndyRussG):
BrowserTests: support iphone

https://gerrit.wikimedia.org/r/197967

Change 197967 merged by jenkins-bot:
BrowserTests: support iphone

https://gerrit.wikimedia.org/r/197967

Change 193556 merged by jenkins-bot:
CentralNotice browser tests: even more platforms and browsers

https://gerrit.wikimedia.org/r/193556

Now closing this card, since we have a full house of browser tests, and failures will in theory be announced by e-mail to the fr-tech mailing list.

I'd also hope to see QUnit tests also run on a suite of browsers on a schedule, and e-mail about failures, just like browser tests. Since that's not standard practice (rather, they run only on Chrome on Linux, when patches are sent for review), we'll need to coordinate with folks working on CI (who have been moving quickly on this stuff, in fact, much appreciated!). Anyway, I'll make another card. :)

AndyRussG closed this task as Resolved.Apr 15 2015, 6:18 PM

(The idea of running QUnit on many browsers [see above] would take care of the remaining point in this card's description, that is, running our test fixtures cross-browser.)

greg moved this task from Inbox to Done on the Browser-Tests-Infrastructure board.Apr 16 2015, 4:14 PM

Change 194773 abandoned by JanZerebecki:
CentralNotice: enable qunit-karma job

Reason:
The generic qunit job for this extension is already enabled and was switched to karma in the meantime. I.e. this patches objectives were solved in other ones.

Please restore if you think otherwise.

https://gerrit.wikimedia.org/r/194773

mmodell removed a subscriber: awight.Jun 22 2017, 9:38 PM