Page MenuHomePhabricator

[EPIC] Extend client-side error logging coverage to include English Wikipedia
Open, MediumPublic

Description

This task tracks the work being done to scale up the client-side error logging effort to cover more projects and more platforms.

Phase 1

  • Hebrew

Phase 2

  • metawiki
  • Wikimedia Commons

Phase 3

  • All Wikipedias (besides enwiki)
  • Wikidata

Phase 4

  • English Wikipedia

Related Objects

StatusSubtypeAssignedTask
OpenNone
Openovasileva
Openjlinehan
Openjlinehan
ResolvedNone
Resolvedjlinehan
Resolvedjlinehan
ResolvedOttomata
ResolvedOttomata
ResolvedSpikeJdlrobson
Resolvedjlinehan
Openjlinehan
Resolvedjlinehan
Resolvedjlinehan
Resolvedjlinehan
Resolvedjlinehan
ResolvedJdlrobson
Resolvedjlinehan
Openjlinehan
Openjlinehan
OpenNone
OpenNone
OpenNone
Openjlinehan
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 16 2020, 4:50 PM

Filing under the previous umbrella task for visibility. Parent may need a description refresh.

Nirmos added a subscriber: Nirmos.Jun 16 2020, 5:05 PM
LGoto triaged this task as Medium priority.Jul 1 2020, 3:45 PM
fgiunchedi moved this task from Inbox to Radar on the observability board.Jul 20 2020, 12:57 PM
jlinehan renamed this task from Enable client-side error logging on production wikis to Extend client-side error logging coverage.Aug 5 2020, 1:34 PM
jlinehan updated the task description. (Show Details)
jlinehan added subscribers: dcipoletti, Mholloway.
jlinehan added a project: Epic.

Change 622620 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enables error logging on Hebrew Wikipedia

https://gerrit.wikimedia.org/r/622620

Change 622620 merged by jenkins-bot:
[operations/mediawiki-config@master] Enables error logging on Hebrew Wikipedia

https://gerrit.wikimedia.org/r/622620

Mentioned in SAL (#wikimedia-operations) [2020-08-26T17:41:25Z] <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable client side error logging on hewiki (T255585) (duration: 01m 04s)

Jdlrobson updated the task description. (Show Details)Aug 27 2020, 9:00 PM
Jdlrobson updated the task description. (Show Details)

Change 623392 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enables MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Change 623392 abandoned by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

Reason:

https://gerrit.wikimedia.org/r/623392

Change 623587 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623587

Change 623587 abandoned by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

Reason:
Abandoning because it didn't fix the issue

https://gerrit.wikimedia.org/r/623587

Change 623392 restored by Jason Linehan:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Change 623392 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MediaWiki client errors on commonswiki and metawiki

https://gerrit.wikimedia.org/r/623392

Mentioned in SAL (#wikimedia-operations) [2020-09-09T18:12:54Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 85e36ae12e7467a559e3d52c58cc3a71ffd09ded: Enable MediaWiki client errors on commonswiki and metawiki (T255585) (duration: 01m 06s)

jlinehan updated the task description. (Show Details)Sep 9 2020, 6:14 PM
jlinehan updated the task description. (Show Details)Sep 9 2020, 6:20 PM
Restricted Application added a subscriber: Huji. · View Herald TranscriptSep 9 2020, 6:20 PM
jlinehan updated the task description. (Show Details)Sep 9 2020, 6:25 PM

Change 627575 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] Enable MediaWiki client errors on frwiki

https://gerrit.wikimedia.org/r/627575

Change 627575 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MediaWiki client errors on frwiki

https://gerrit.wikimedia.org/r/627575

Mentioned in SAL (#wikimedia-operations) [2020-09-15T18:12:52Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 1d3456570b80b1d8af1d2b71975496e54f87b24e: Enable MediaWiki client errors on frwiki (T255585) (duration: 00m 57s)

Adding some info from various discussions for visibility.

With group0, hawiki, cawiki, hewiki, metawiki, and commonswiki, we're seeing a per-pageview error rate of 0.00013333333,. Extrapolating, we'd be looking at 33,293 daily errors for ja+es+de+fr+ru+it+zh+pt, and 72,293 with the above + en. Considering the long tail of other projects, we can still estimate we'll be < 100k daily errors at total coverage, if the current rate holds. There is reason to assume that the rate will drop on larger projects, due to the fact that "normal" users and clients, which will experience fewer or no errors, may make up a larger population.

I carried this number to #wikimedia-observablity

hip 12:34
so the question is, is this the kind of system where 100k/daily is a big number, or a small number, or in between?

herron 12:51
I'd say the smaller side. ingesting 100k more events per day wouldn't be a problem. we have capacity for bursts in log volume ~10x above our normal amount and sufficient storage as well. as long as we're not expecting this to introduce sustained significant increases in error volume (on the order of thousands per second) it should be fine

cdanis 12:56
hip: by way of comparison, the past 24h of logstash is >34M events

So in short, we should be fine to spin up to larger wikis from here, starting with frwiki.

jlinehan updated the task description. (Show Details)Sep 15 2020, 6:21 PM

Change 629174 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] clientError: enable on ja,es,de,ru,it,zh,pt wikipedias

https://gerrit.wikimedia.org/r/629174

Change 629174 merged by jenkins-bot:
[operations/mediawiki-config@master] clientError: enable on ja,es,de,ru,it,zh,pt wikipedias

https://gerrit.wikimedia.org/r/629174

Mentioned in SAL (#wikimedia-operations) [2020-09-22T23:27:36Z] <legoktm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: clientError: enable on ja,es,de,ru,it,zh,pt wikipedias (T255585) (duration: 01m 04s)

Change 630908 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[operations/mediawiki-config@master] clientError: Expand coverage to all Wikipedias besides enwiki.

https://gerrit.wikimedia.org/r/630908

WMDE-leszek added a comment.EditedSep 29 2020, 5:13 PM

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@WMDE-leszek that sounds great, it will be enabled on Wikidata in a few hours.

Change 630908 merged by jenkins-bot:
[operations/mediawiki-config@master] clientError: Enable on Wikidata + all Wikipedias besides enwiki.

https://gerrit.wikimedia.org/r/630908

Mentioned in SAL (#wikimedia-operations) [2020-09-30T18:14:17Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable clientError on Wikidata and all Wikipedias except enwiki (T255585) (duration: 00m 58s)

@jlinehan: I am not sure if saying it here on the ticket is a right forum, please redirect me if it is not. WMDE would be very interested in having client-side error logging enabled on Wikidata too!

@WMDE-leszek that sounds great, it will be enabled on Wikidata in a few hours.

thanks @jlinehan !

Thanks, +1 for removing client_ip before we proceed with wider rollout of client error logging.

mpopov added a subscriber: mpopov.Oct 23 2020, 6:15 PM

This is so weird! @Aklapper: for some reason Phabricator won't allow both the Instrument-ClientError and Product-Infrastructure-Data tags. It keeps removing one whenever I add the other.