Page MenuHomePhabricator

Create exploratory test cases for Event Metrics QA
Closed, ResolvedPublic

Description

We have tested Event Metrics one feature at a time. But as we move towards release, we should do some more holistic, exploratory testing. Below are suggestions for some use cases that seem worth checking out.

  • Large event, one wiki: EM can theoretically track a per-wiki maximum of 50k pages, Commons files or wikidata items. But what is its real limit? Create an event with a number approaching that limit on 1 wiki. This should be fairly easy to do by combining certain large categories. The Wiki Loves Monuments categories, for example, are conveniently grouped by year. One might also do this by combining some of our most into an event (there's a report of Top Editors on Wikistats 2.0).
    • Can the event successfully process or does it time out?
    • If the latter, how close to the limit do we seem to be able to get?
    • What happens when users try to download a Pages Created report with many thousands of results?
    • Do very large numbers get abbreviated and displayed properly on the Event Summary page? E.g., what happens when Pageviews or Bytes hit the billions?
  • Large event, multiple wikis: The 50k limit is per wiki. What happens when an event combines multiple wikis and tries to track a large number of items on each?
  • Category with Participants filtering: For T205734 we will have tested Category filtering without participants. But do these filters work properly when combined? Results should be for thie intersection of the two, using this logic: Participants AND (Category1 OR Category2 OR Category3).

Event Timeline

Hi @dom_walden, I defined a few exploratory QA use cases above. I'll add to this list as I think of more.

  • Large event, one wiki: EM can theoretically track a per-wiki maximum of 50k pages, Commons files or wikidata items. But what is its real limit? Create an event with a number approaching that limit on 1 wiki. This should be fairly easy to do by combining certain large categories. The Wiki Loves Monuments categories, for example, are conveniently grouped by year. One might also do this by combining some of our most into an event (there's a report of Top Editors on Wikistats 2.0).
    • Can the event successfully process or does it time out?

It is possible to calculate data for events which go up to (and possibly slightly over) 50k created/improved, and get the Event Summary reports (CSV, Wikitext, HTML).

  • If the latter, how close to the limit do we seem to be able to get?

Events with around 1-2k pages/wikidata items improved are more manageable, for reasons stated below.

  • What happens when users try to download a Pages Created report with many thousands of results?

Near to the 50k limit, it becomes almost impossible to run Pages Created/Improved reports as they take such a long time (CSV and Wikitext).

EDIT: I tried to run a pages improved wikitext report for 13.7k wikidata items, and eventually returned the error: "Maximum execution time of 30 seconds exceeded".

On staging, I find that while waiting for such a report I cannot do anything else on Event Metrics (pages stop loading entirely). I don't know whether this is a sign of EM being overloaded or some sort of feature of wmflabs to prevent a particular user hogging all the resources.

I could attempt to repeat on production but don't want to potentially cause a slow down.

I also find that, for events with 50k pages created, if I run the Pages Created report (CSV or Wikidata) all it returns is the header with no data. This does not occur with Pages Improved.

  • Large event, multiple wikis: The 50k limit is per wiki. What happens when an event combines multiple wikis and tries to track a large number of items on each?

This event is about within the manageable range (i.e. you can load reports without it breaking, it just takes a while): https://eventmetrics-dev.wmflabs.org/programs/142/events/355 (In My Programs > dwalden Category filter > Large event, multiple wikis).

Participants 999, Pages created 26, Pages improved 1.9k, Edits 6.3k, Uploaded files 1.6k, Bytes changed 1.8M.

jmatazzoni moved this task from QA to Done on the Community-Tech-Sprint board.