Page MenuHomePhabricator

Theme: Testing
Closed, ResolvedPublic

Description

Theme program committee

@gregGreg GrossmeierWMF Director of Engineering. Program Committee Chair
@WMDE-leszekLeszek ManickiWMDE Engineering Manager

Theme proposal

Providing helpful, actionable, and timely feedback on developer code will enable long term health and stability of our platform and ultimately make Wikimedia developers more productive. This track will focus on testing practices (including local testing), tooling, infrastructure, and culture.

Session proposals

See sub-tasks

Event Timeline

Rfarrand created this task.Sep 27 2019, 6:27 PM
Rfarrand triaged this task as Medium priority.Sep 27 2019, 6:33 PM
Rfarrand reassigned this task from Rfarrand to greg.Sep 27 2019, 6:38 PM
Restricted Application added a project: User-greg. · View Herald TranscriptSep 27 2019, 6:38 PM
Rfarrand updated the task description. (Show Details)Oct 4 2019, 12:18 AM
Rfarrand updated the task description. (Show Details)Oct 4 2019, 12:24 AM

The session proposals will be added to this task tomorrow and an email will go out to techconf attendees asking for feedback on theme and session proposals before they are finalized.

Rfarrand updated the task description. (Show Details)Oct 4 2019, 6:50 PM
hashar updated the task description. (Show Details)Oct 7 2019, 1:17 PM
hashar awarded a token.Oct 7 2019, 2:50 PM

Anything missing (topics, problems, workshops, reports, case studies, ideas, tools, etc) from the area of Testing that you'd like to see at the Technical Conference 2019? Please comment here, reach out to @greg or me!

hashar added a subscriber: hashar.Oct 7 2019, 8:25 PM

It'd be great to have a short discussion on the overall QA role in a development process. A working title may be - Dev processes and best QA practices - why QA is not limited to testing code?

greg updated the task description. (Show Details)Oct 8 2019, 8:28 PM

I offered to co-lead the API integration testing session but I’m also happy to co-lead or lead the testing pyramid or the how to run tests session.

Tgr added a subscriber: Tgr.Oct 13 2019, 8:29 PM

Anything missing (topics, problems, workshops, reports, case studies, ideas, tools, etc) from the area of Testing that you'd like to see at the Technical Conference 2019? Please comment here, reach out to @greg or me!

Visual testing? It is used in some parts of our codebase (e.g. Parsoid) and might have wider applicability (wouldn't it be nice if automated tests could catch things like T234380?).

UI / DOM testing? I remember this gave us a lot of problems with MediaViewer, Selenium is too large and unwieldy for the task, for QUnit the lack of isolation and lack of established patterns caused lots of issues.

(Testing would not be my top choice of track to attend though, so I'm just kibitzing.)

greg added a comment.Oct 15 2019, 10:22 PM

I offered to co-lead the API integration testing session but I’m also happy to co-lead or lead the testing pyramid or the how to run tests session.

Can I put you down for leading the testing pyramid one? :)

Can I put you down for leading the testing pyramid one? :)

@greg sure. And if anyone wants to co-lead please point them my way.

greg added a comment.Nov 15 2019, 9:51 PM

Recap notes:

Testing Theme

Greg re-cap

  • System level, patterns, API integration, Test pyramids, running test locally
  • From the API Integration testing - was this morning
    • Going through potential examples on how to do this - async aweight
    • Mate: API integration and infrastructue with frameworks ??? black box - come up with some example user stories
  • System level testing patterns and anti patterns
    • Made people sad. Some repos showed many anti patterns. Be careful and communicate considerately
    • Create a wiki page
    • create cleanup patches
    • Sumbit first patches to get the ball running - get a boilerplate in place
    • Reach out and find out why some teams didn't find the Selenium documentation or if they did why it wasn't useful
    • reach out to teams and offer assistance - reviews 
    • Quality and testing team needs to put themselves out there more and be more proactivity
  • Test pyramid and speeding up ???
    • reasonable model to use
    • some variability in how it's applies
    • not intended to demonizing any area of testing
    • good progress breaking into unit tests
    • Many repos fit this model today
    • Broader understanding that education is needed
      • contextualized knowledge sharing
    • propogatin of this model and more outreach as an action item
  • Running tests locally
    • Something about zul merger, quibble
    • For testing generally - guidance around how and why we shoud test. Not best practices, but policies (but not that word, becuase it's too strong). Maybe better onboarding in respect to testing.

Group 1

  • Anything else from your experiences you wanted to share?
    • Documentation as well as outreach and education
      • instruction manuals and best practices, and review of state of the art and what others are doing
      • Wall of fame

Group 2

  • System level testing patterns and anti patterns
  • API integration
  • Test pyramids,
    • suprised to see good progress already made. 
    • How did you know about this?
    • blog posts and other communication have enabled and inspired others
    • incentivization of change
  • running test locally 
  • Anything else from your experiences you wanted to share?
  •  Do we have documentaion on how to write tests?
    • Yes some. On wiki, but only partially. Some work to do.
      •  maybe that would be a good session - bootstrapping test writing - how to start and how to write them

Group 3

  • System level testing patterns and anti patterns
  • API integration
    • Daniel: More user stories and workflows
  • Test pyramids
    • providing the first patch, or sending out a message to wikitech, maybe be more proactive in communicating 
    • shaming people was also effective - making it a wall of fame
    • pair programming as a way of learning unit testing
    • onboarding as an opporunity to teach best behaviors
      • best initiative is with new staff
    • One thing that was missing was guidance on what type of tests you want to write ahead of time or knowldge of what the standard is. Varies from team to team.
      • A set of recommendations.
      • When do we need to use them?
    • Try to do some thing across teams for onboarding - sharing that 
      • can't fit everyone, bu shared best practices
        • how deployments happen, how Y happens, etc. 
        • have this done by other team members not on their team.
        • Onboarding is part of mentoring - establishing cultural norms
  • running test locally

Group 4

  • System level testing patterns and anti patterns
  • API integration
  • Test pyramids
  • running test locally
  • Big part was education and how big that field is. Many people aware testing should be done on many levels. 
    • API testing - very expansive knowledge on that and shoud think about it more.
    • Testing requires a coordinated effort - recommendations can help (policies, guidance)
  • Would have been interesting to hear how other teams integrate testing into thier work
    • there was a session! Elena's - see notes in phab
  • Many folks had thoughts on how things might be structured, that is great. Quibble session was a good refresher. It needs help in documentation, but good to see activity. Test pyramid was interesting, looked at some repo's and good to see adoption
    • would a dashboard showing the pyramid be useful? Don't want it to become a metric people use. 
    • if we made coverage more discoverable, coverage increased. Folks motivated if they can have visibility 
  • Is there anything missing from this selection of topics?
    • Silence. :)

Group 5

  • System level testing patterns and anti patterns
    • not much about acceptance criteria - would have liked more of that
    • during the hosting ??? it was mentioned that there are daily browser tests running against that. I was under the impressiont hat Selenium is self contained. Why can't tests run against prodution instead of using beta. 
      • There are some tests that run dialy ....??
    • Documentation and learning - an dobservation is that most the information is out there but it is not a very coherent story that tells people how to do their testing properly. 
      • In one of the sessions we mentioned tutorials, videos, use cases, etc. 
        • ??? [Something about the nuance of testing]
      • We could write clickbait headlines - "The top 5 ways not to screw up your environment!"
  • API integration
  • Test pyramids
  • running test locally
greg closed this task as Resolved.Dec 17 2019, 11:18 PM