Page MenuHomePhabricator

[EPIC] Extend client-side error logging coverage to include English Wikipedia
Closed, ResolvedPublic

Description

This task tracks the work being done to scale up the client-side error logging effort to cover more projects and more platforms.

Phase 1

  • Hebrew

Phase 2

  • metawiki
  • Wikimedia Commons

Phase 3

  • All Wikipedias (besides enwiki)
  • Wikidata

Phase 4

  • English Wikipedia

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedovasileva
Resolvedovasileva
Resolvedovasileva
Resolved jlinehan
ResolvedTgr
ResolvedNone
Resolved jlinehan
Resolved jlinehan
ResolvedOttomata
ResolvedOttomata
ResolvedSpikeJdlrobson
Resolved jlinehan
OpenNone
Resolved jlinehan
Resolved jlinehan
Resolved jlinehan
Resolved jlinehan
ResolvedJdlrobson
Resolved jlinehan
Resolved jlinehan
ResolvedTgr
Resolvedcolewhite
DeclinedNone
ResolvedNone
Resolved jlinehan
ResolvedTgr
ResolvedBUG REPORTTgr
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 622620 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enables error logging on Hebrew Wikipedia

https://gerrit.wikimedia.org/r/622620

Change 622620 merged by jenkins-bot:
[operations/mediawiki-config@master] Enables error logging on Hebrew Wikipedia

https://gerrit.wikimedia.org/r/622620

Mentioned in SAL (#wikimedia-operations) [2020-08-26T17:41:25Z] <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable client side error logging on hewiki (T255585) (duration: 01m 04s)

Jdlrobson updated the task description. (Show Details)

Change 623392 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enables MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Change 623392 abandoned by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

Reason:

https://gerrit.wikimedia.org/r/623392

Change 623587 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623587

Change 623587 abandoned by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

Reason:
Abandoning because it didn't fix the issue

https://gerrit.wikimedia.org/r/623587

Change 623392 restored by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Change 623392 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Mentioned in SAL (#wikimedia-operations) [2020-09-09T18:12:54Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 85e36ae12e7467a559e3d52c58cc3a71ffd09ded: Enable MediaWiki client errors on commonswiki and metawiki (T255585) (duration: 01m 06s)

Restricted Application added a subscriber: Huji. · View Herald TranscriptSep 9 2020, 6:20 PM

Change 627575 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enable MediaWiki client errors on frwiki

https://gerrit.wikimedia.org/r/627575

Change 627575 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MediaWiki client errors on frwiki

https://gerrit.wikimedia.org/r/627575

Mentioned in SAL (#wikimedia-operations) [2020-09-15T18:12:52Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 1d3456570b80b1d8af1d2b71975496e54f87b24e: Enable MediaWiki client errors on frwiki (T255585) (duration: 00m 57s)

Adding some info from various discussions for visibility.

With group0, hawiki, cawiki, hewiki, metawiki, and commonswiki, we're seeing a per-pageview error rate of 0.00013333333,. Extrapolating, we'd be looking at 33,293 daily errors for ja+es+de+fr+ru+it+zh+pt, and 72,293 with the above + en. Considering the long tail of other projects, we can still estimate we'll be < 100k daily errors at total coverage, if the current rate holds. There is reason to assume that the rate will drop on larger projects, due to the fact that "normal" users and clients, which will experience fewer or no errors, may make up a larger population.

I carried this number to #wikimedia-observablity

hip 12:34
so the question is, is this the kind of system where 100k/daily is a big number, or a small number, or in between?

herron 12:51
I'd say the smaller side. ingesting 100k more events per day wouldn't be a problem. we have capacity for bursts in log volume ~10x above our normal amount and sufficient storage as well. as long as we're not expecting this to introduce sustained significant increases in error volume (on the order of thousands per second) it should be fine

cdanis 12:56
hip: by way of comparison, the past 24h of logstash is >34M events

So in short, we should be fine to spin up to larger wikis from here, starting with frwiki.

Change 629174 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] clientError: enable on ja,es,de,ru,it,zh,pt wikipedias

https://gerrit.wikimedia.org/r/629174

Change 629174 merged by jenkins-bot:
[operations/mediawiki-config@master] clientError: enable on ja,es,de,ru,it,zh,pt wikipedias

https://gerrit.wikimedia.org/r/629174

Mentioned in SAL (#wikimedia-operations) [2020-09-22T23:27:36Z] <legoktm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: clientError: enable on ja,es,de,ru,it,zh,pt wikipedias (T255585) (duration: 01m 04s)

Change 630908 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] clientError: Expand coverage to all Wikipedias besides enwiki.

https://gerrit.wikimedia.org/r/630908

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@WMDE-leszek that sounds great, it will be enabled on Wikidata in a few hours.

Change 630908 merged by jenkins-bot:
[operations/mediawiki-config@master] clientError: Enable on Wikidata + all Wikipedias besides enwiki.

https://gerrit.wikimedia.org/r/630908

Mentioned in SAL (#wikimedia-operations) [2020-09-30T18:14:17Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable clientError on Wikidata and all Wikipedias except enwiki (T255585) (duration: 00m 58s)

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@WMDE-leszek that sounds great, it will be enabled on Wikidata in a few hours.

thanks @jlinehan !

Thanks, +1 for removing client_ip before we proceed with wider rollout of client error logging.

This is so weird! @Aklapper: for some reason Phabricator won't allow both the Instrument-ClientError and #product-infrastructure-data tags. It keeps removing one whenever I add the other.

@mpopov: That is correct and expected behavior - Instrument-ClientError is a subproject of #product-infrastructure-data.
(In general, feel free to bring up general questions on https://www.mediawiki.org/wiki/Talk:Phabricator/Help - thanks!)

Change 658356 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enables MediaWiki client error instrument on English Wikipedia

https://gerrit.wikimedia.org/r/658356

Change 658356 merged by jenkins-bot:
[operations/mediawiki-config@master] Enables MediaWiki client error instrument on English Wikipedia

https://gerrit.wikimedia.org/r/658356

Mentioned in SAL (#wikimedia-operations) [2021-01-25T19:25:05Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:658356|Enables MediaWiki client error instrument on English Wikipedia (T255585)]] (duration: 01m 01s)

We are live and it's been pretty quiet so far and error lates have been surprisingly low which is good! Will check in again later today and tomorrow.

Jdlrobson updated the task description. (Show Details)