Page MenuHomePhabricator

[Quarterly Success Metric] By team test history
Closed, ResolvedPublic

Description

As part of our overall effort to improve the visibility and shared ownership of browser tests, we'd like to provide teams a simplified view into the current and historical state of their browser tests. It's not yet clear whether this can be achieved with the existing capabilities in Jenkins or whether a new dashboard will need to be created elsewhere—such as a standalone dashboard similar to https://integration.wikimedia.org/zuul/ or a Phabricator application.

Requirements

  1. Current test results should be displayed in channels that are used in everyday team communication so as not to be missed or ignored.
  2. Historical results should be available as a timeline to show feature coverage and pass/failure trends.
  3. It should be easy to correlate test failure with commit history so that bugs may be quickly squashed or tests refactored.
  4. Results should link to verbose build logs in Jenkins and/or job replays in SauceLabs.
  5. Displaying the current status by feature and scenario, instead of by Jenkins build, may promote use in acceptance testing.

Some Options

Jenkins Dashboards + Team IRC Notifications

We may not need to build out something completely new here. Jenkins already provides dashboard views that can be customized to show current and historical browser test trends, and we're already looking to make failures more visible by notifying team directly through IRC (see T89375). This solution paired with some best practices communication to PMs might satisfy all but requirement #5 albeit it not in the cleanest/prettiest fashion, but it would entail a lot less work and involve less risk of duplication.

Standalone Dashboard using Logged Metrics and Results

A custom dashboard that consumes metrics (perhaps via elasticsearch) would grant us a clean slate for presenting more usable/understandable graphs and results than what we currently have in Jenkins. Requirement #5 might still be difficult, however, as the Cucumber results would remain unstructured as Jenkins text logs, unless we can log a structured version of them to elasticsearch and consume/present them via the dashboard.

Publish to Custom Phabricator Application

One (very ambitious) idea was to write a custom Phabricator application that provides an endpoint for test results to be submitted and presents them under corresponding project pages. One advantage to this approach over the others would be that the test result data can be easily correlated with responsible commits (important as long as browser tests remain post-merge/non-voting) and possibly even presented alongside the Diffusion commit tree. Another advantage would be the visibility and close proximity to tools that teams use for workflow, an important aspect for facilitating future acceptance testing efforts—browser tests at that point could represent true acceptance tests and integrate with acceptance criteria in Phabricator.

Some big disadvantages to this option include the initial build investment and ongoing maintenance burden that comes along with building a Phab plugin. There's also the risk of building with a focus on acceptance testing when the practice of ATTD hasn't really been pursued in earnest by any team as of yet (that ol' cart before the horse thing).

Event Timeline

greg created this task.Feb 5 2015, 6:26 PM
greg assigned this task to dduvall.
greg raised the priority of this task from to Medium.
greg updated the task description. (Show Details)
greg added a project: Staging.
greg added a project: Release-Engineering-Team.
greg set Security to None.
greg added subscribers: mmodell, greg, Aklapper.
dduvall updated the task description. (Show Details)Feb 27 2015, 10:53 PM
dduvall updated the task description. (Show Details)Mar 5 2015, 12:15 AM
dduvall updated the task description. (Show Details)Mar 5 2015, 12:45 AM
mmodell moved this task from Backlog to In Progress on the Staging board.Mar 12 2015, 4:59 PM
thcipriani moved this task from In Progress to Backlog on the Staging board.May 27 2015, 3:04 PM
dduvall closed this task as Resolved.Jun 9 2015, 6:57 PM

I did a terrible job at communicating the progress on this, but the initial project can be considered done. I'll create tasks for improvements so they can be prioritized.

Raita lives at http://raita.wmflabs.org for now. The ops/puppet patch has been cherry-picked on integration-puppetmaster but is still pending ops review.

https://gerrit.wikimedia.org/r/#/c/208024/

@dduvall you can post to wikitech-l / engineering / qa lists to show case raita :-}

greg added a comment.Jun 9 2015, 9:19 PM

@dduvall you can post to wikitech-l / engineering / qa lists to show case raita :-}

we're also going to be reviving the tech talk/lightning talks, and this could be a good candidate for that :)