Page MenuHomePhabricator

Enable verbose logging without installing the WikimediaDebug extension
Open, MediumPublic

Description

WikimediaDebug allows the user to set verbose logging mode, in which case all logs are recorded regardless of the levels configured in $wmgMonologChannels. This is very useful but has some limitations:

  • Installing a browser extension is fine for developers, but might be too much effort for users who report a hard-to-reproduce bug and are asked to help diagnose it.
  • The browser extension is only supported for Firefox and Chrome.
  • Mobile Chrome does not allow installing extensions.
  • The extension routes all traffic to a debug host. This is useful for debugging, but not always good for diagnosing problems (which might be related to cross-DC requests etc).

We should have a lightweight way to trigger verbose logging, probably via a cookie.

Event Timeline

Given potential for abuse, and for skewing server latency and perf metrics (due to log overhead), I think it might be a feature to limit this to mwdebug hosts. However, that isn't mutually exclusive with the goal to have this work with (only) a cookie.

The routing logic for mwdebug is a Lua script at the CDN layer (ATS), and we could read cookies from there as well (we do this already for multi-dc, so you can review that as example of how to do something similar). E.g. a certain cookie could serve as alternative way to enable a (subset) of WikimediaDebug features.

One thing I would recommend is to make this cookie really hard to set in a way that would apply for longer periods of time, for the same reasons we automatically stop WikimediaDebug after 15min (namely, to protect user privacy, and to avoid creating noise for devs doing other work on the same server). For example, it should not be possible to lazlyy set this cookie by hand via $.cookie and then have it stay for 24 hours.

We could document a copy-pastable 3-line JS snippet that encodes some kind of timestamp in the cookie name, so that the routing logic can automatically ignore ones that within ± 15min from the current time (or 16min, if you go for comparing without the 3 digits from epoch).

The main thing I want to avoid is some kind of snippet that if executed results in a cookie that remains applicable for a long duration of time. For the mobile use case, I imagine people will need a gadget or user script of some sort, so we could provide one for them that adds a button, and when clicked runs those 3 lines.

I was thinking of creating a special page where users can set/unset the cookie (maybe it can be shoehorned into WikimediaEvents). I agree proliferation is a potential concern; making the cookie name dynamic is one way to handle it, or we could require the expiry to be part of the cookie value and MediaWiki could unset the cookie if it's expired or is too long-lived.

Cookies can only be set for the registrable domain, so for testing cross-wiki features users would have to set the cookie separately on each wiki. That's annoying, but there is not much we can do about it.

[…] I agree proliferation is a potential concern; making the cookie name dynamic is one way to handle it, or we could require the expiry to be part of the cookie value and MediaWiki could unset the cookie if it's expired or is too long-lived.

That sounds good to me also.

Change 1000306 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/WikimediaEvents@master] Add X-Wikimedia-Debug special page

https://gerrit.wikimedia.org/r/1000306

Change 1000307 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Support cookies in XWikimediaDebug

https://gerrit.wikimedia.org/r/1000307

Change 1000307 merged by jenkins-bot:

[operations/mediawiki-config@master] Support cookies in XWikimediaDebug

https://gerrit.wikimedia.org/r/1000307

Mentioned in SAL (#wikimedia-operations) [2024-03-14T21:34:21Z] <krinkle@deploy2002> Synchronized src/XWikimediaDebug.php: Support cookies in XWikimediaDebug, I5e33e90fd, T350094 (duration: 12m 08s)

Change 1000306 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/WikimediaEvents@master] Add X-Wikimedia-Debug special page

https://gerrit.wikimedia.org/r/1000306

Current draft:

LandingActive
Screenshot 2024-03-14 at 16.59.45.png (1×2 px, 340 KB)
Screenshot 2024-03-14 at 16.59.47.png (1×2 px, 319 KB)

Proposed improvement:

LandingActive
Screenshot 2024-03-14 at 16.59.53.png (1×2 px, 346 KB)
Screenshot 2024-03-14 at 16.59.56.png (1×2 px, 324 KB)

Change 1000306 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Add X-Wikimedia-Debug special page

https://gerrit.wikimedia.org/r/1000306

Krinkle triaged this task as Medium priority.Mon, Apr 15, 3:27 PM

Change #1024932 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Profiler: Handle X-Wikimedia-Debug cookie

https://gerrit.wikimedia.org/r/1024932