Gilles (Gilles Dubuc)
Senior Performance Engineer, WMF

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 8:24 AM (223 w, 3 d)
Availability
Available
IRC Nick
gilles
LDAP User
Gilles
MediaWiki User
Unknown

Recent Activity

Today

Gilles closed T214015: Create gerrit bot for git pushes to specific repo from prod machines as Resolved.

Alright, I think that answers it, it's more straightforward than I anticipated. The bot needs I have are actually much smaller than I first thought, and I'll probably be running it off of WMCS.

Fri, Jan 18, 11:52 AM · Release-Engineering-Team, Performance-Team
Gilles closed T214015: Create gerrit bot for git pushes to specific repo from prod machines, a subtask of T209857: Create Autonomous Systems ranking based on RUM data, as Resolved.
Fri, Jan 18, 11:52 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles closed T214025: "The authentication plugin denied the password change." on wikitech as Invalid.

It worked with a different name. I guess the name "performanceteam" was problematic. It could be that there's a unix user group named that way or something.

Fri, Jan 18, 11:43 AM · Performance-Team, wikitech.wikimedia.org
Gilles added a comment to T214025: "The authentication plugin denied the password change." on wikitech.

I'll try again, calling it something different

Fri, Jan 18, 11:41 AM · Performance-Team, wikitech.wikimedia.org
Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

As expected, here it is: https://analytics.wikimedia.org/datasets/performance/autonomoussystems/

Fri, Jan 18, 10:18 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles added a comment to T214025: "The authentication plugin denied the password change." on wikitech.

For the desired shell name in the signup form, if that's what it is, I'm pretty sure I put "performanceteam". Since I'm stuck in this limbo of temp password, I can't log into the account to look anything up.

Fri, Jan 18, 10:17 AM · Performance-Team, wikitech.wikimedia.org
Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

I've created a performance/autonomoussystems folder on /srv/published-datasets on stat1004, which should get picked up and published on https://analytics.wikimedia.org/datasets/ eventually.

Fri, Jan 18, 10:13 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles added a comment to T214025: "The authentication plugin denied the password change." on wikitech.

Where do I look that up?

Fri, Jan 18, 10:12 AM · Performance-Team, wikitech.wikimedia.org
Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

So I've figured out how I can have a bot push arbitrary file contents to a gerrit change on performance/docroot, but there is ont big caveat: it's not creating a git commit per se, and thus it's not running the git hooks that get jekyll to run... I don't think it's reasonable to have node and all the potentially risky dependencies it's going to pull run on the stat machines merely to run jekyll and generate the change we need.

Fri, Jan 18, 9:21 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles triaged T214149: Backport pygerrit2 to Debian Stretch as Normal priority.
Fri, Jan 18, 9:08 AM · Operations, Release-Engineering-Team, Performance-Team
Gilles claimed T214015: Create gerrit bot for git pushes to specific repo from prod machines.
Fri, Jan 18, 9:06 AM · Release-Engineering-Team, Performance-Team
Gilles added a comment to T214015: Create gerrit bot for git pushes to specific repo from prod machines.

Here's how simple the code is for publishing the change to gerrit with that library:

Fri, Jan 18, 9:06 AM · Release-Engineering-Team, Performance-Team
Gilles added a comment to T214015: Create gerrit bot for git pushes to specific repo from prod machines.

I've figured out how to do that locally with the pygerrit2 python library and it's fairly straightforward. Unfortunately that library is only available in buster, dammit, I will have to backport it.

Fri, Jan 18, 8:53 AM · Release-Engineering-Team, Performance-Team

Yesterday

Gilles added a comment to T214025: "The authentication plugin denied the password change." on wikitech.

User id 13687 on wikitech

Thu, Jan 17, 3:00 PM · Performance-Team, wikitech.wikimedia.org
Gilles created T214025: "The authentication plugin denied the password change." on wikitech.
Thu, Jan 17, 11:49 AM · Performance-Team, wikitech.wikimedia.org
Gilles triaged T214015: Create gerrit bot for git pushes to specific repo from prod machines as Normal priority.
Thu, Jan 17, 9:15 AM · Release-Engineering-Team, Performance-Team
Gilles renamed T209857: Create Autonomous Systems ranking based on RUM data from Create ISP ranking based on RUM data to Create Autonomous Systems ranking based on RUM data.
Thu, Jan 17, 8:59 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles updated subscribers of T209857: Create Autonomous Systems ranking based on RUM data.

@Nuria my plan is to have the report generated by a cron python script on a stat machine, and have the resulting CSV then git-pushed to the performance/docroot repo (static site where the report will be viewable). Does that sound sane to you? Is there any precedent to doing something like this?

Thu, Jan 17, 8:58 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team

Tue, Jan 15

Gilles added a comment to T212124: Consider adding decoding=async to our img tags.

Looking at synthetic and RUM data, it doesn't look like this had a visible effect on global metrics. This was an expected possible outcome, as it's possible that the browser's heuristics were already doing the right thing.

Tue, Jan 15, 3:46 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team
Gilles created T213833: 403 on http://wpt.wmftest.org.
Tue, Jan 15, 3:36 PM · Upstream, WebPageTest, Performance-Team
Gilles closed T213674: Possible first paint regression on mobile as Resolved.

The explanation is that I reset the sampling rate of navigation timing for ruwiki back to the default on 2019-01-10. This was after making it 10x its normal rate on 2018-12-20. Which has an inverse effect on metrics, that went unnoticed. It's easier to see the effect when looking at the desktop metrics at those dates.

Tue, Jan 15, 3:32 PM · Performance-Team
Gilles added a subtask for T170817: Upgrade Thumbor servers to Stretch: T213771: First page of a specific PDF files on Commons does not render a preview.
Tue, Jan 15, 3:22 PM · serviceops, User-jijiki, Patch-For-Review, User-fgiunchedi, Performance-Team (Radar), Operations, Thumbor
Gilles added a parent task for T213771: First page of a specific PDF files on Commons does not render a preview: T170817: Upgrade Thumbor servers to Stretch.
Tue, Jan 15, 3:22 PM · serviceops, User-jijiki, Thumbor
Gilles added a comment to T213771: First page of a specific PDF files on Commons does not render a preview.

Failing thumbnails tend to be costly to reattempt, which means repeated requests to those get rate-limited (429). Generally speaking, short of a software upgrade, a failing thumbnail isn't going to work the next time it's requested, hence the use of poolcounter or memcache-based throttling.

Tue, Jan 15, 3:21 PM · serviceops, User-jijiki, Thumbor

Mon, Jan 14

Gilles moved T213674: Possible first paint regression on mobile from Inbox to Doing on the Performance-Team board.
Mon, Jan 14, 8:54 PM · Performance-Team
Gilles claimed T213674: Possible first paint regression on mobile.
Mon, Jan 14, 8:53 PM · Performance-Team

Fri, Jan 11

Gilles added a comment to T204281: Stop prioritizing peering over transit.

Following @ayounsi's request, I've put together per-DC real user monitoring performance metrics using the following Hive query:

Fri, Jan 11, 12:51 PM · Performance-Team (Radar), Operations, netops
Gilles added a comment to T213398: Move performance team dashboards to /performance/ folder in Grafana.

@Krinkle's suggestions sound good to me

Fri, Jan 11, 8:23 AM · Performance-Team

Thu, Jan 10

Gilles added a comment to T212304: Ask the eswiki community whether we can run the performance perception survey there.
  • We need to ask the eswiki community's permission before running the perceived performance microsurvey on that wiki.
Thu, Jan 10, 2:28 PM · QuickSurveys (Surveys), CommRel-Specialists-Support (Jan-Mar-2019), Readers-Web-Backlog (Tracking), Performance-Team
Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

On staff IRC there was a discussion of rural vs urban. However I think that since mobile is worth focusing on, there's no mapping between IP addresses and city location for mobile. Without asking users to share their location (which we won't do), there's no way to assess whether they're in a rural or urban area. This means that we'll have to stick with national rankings, which shouldn't inform local decisions about which ISP is best, since that can vary greatly based on location, but if an effort is made by an ISP to improve their service, it should surface in our updated rankings month-to-month or year-to-year.

Thu, Jan 10, 12:42 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

Initial results show that this might work, but I need to wait until the extra data has been collected before I can claim success on devising a fair ranking. Right now for January I only have 12-151 samples per ISP for the US for example (which is logical since pretty much only ruwiki was contributing to the dataset so far) and with that amount I'm seeing big variations in median transferSize. The amount of samples to generate the ranking needs to be large enough that the median transferSize is in the same ballpark for all ranked ISPs. If we don't get there with more data, it's a peculiar finding that users might look at bigger articles depending on which ISP they're subscribed to (could be a rural vs urban thing if that's the case).

Thu, Jan 10, 12:35 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles removed a project from T209857: Create Autonomous Systems ranking based on RUM data: Patch-For-Review.
Thu, Jan 10, 12:33 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles added a comment to T213398: Move performance team dashboards to /performance/ folder in Grafana.

Maybe some subfolders for the ones we have a lot of dashboard for, like NavTiming?

Thu, Jan 10, 8:38 AM · Performance-Team

Mon, Jan 7

Gilles moved T212304: Ask the eswiki community whether we can run the performance perception survey there from Inbox to Blocked on the Performance-Team board.
Mon, Jan 7, 9:05 PM · QuickSurveys (Surveys), CommRel-Specialists-Support (Jan-Mar-2019), Readers-Web-Backlog (Tracking), Performance-Team
Gilles added a comment to T212124: Consider adding decoding=async to our img tags.

I thought about Parsoid, but first let's see if it's beneficial at all for merely reading articles rendered by MediaWiki on a web browser?

Mon, Jan 7, 11:06 AM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team

Thu, Dec 20

Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

I wasn't aware that the latest plan was to use ATS for TLS termination.

Thu, Dec 20, 4:03 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

Could be a puppet variable too, to make the filtering block conditional.

Thu, Dec 20, 3:43 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

Is there an nginx "site" or config specific to varnish termination?

Thu, Dec 20, 3:39 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles updated subscribers of T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

@BBlack would you miss x-cache, x-cache-status and x-varnish if those were completely removed at the TLS termination level? Some of that information is now exposed as part of the Server-Timing header (which unlike others, can be parsed by JS).

Thu, Dec 20, 3:24 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

Plain nginx config has the ability to remove the headers, but it can't do so conditionally...

Thu, Dec 20, 3:05 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles changed the status of T27611: Support optimized WebP thumbnails as alternative to JPEG, PNG from Open to Stalled.
Thu, Dec 20, 9:51 AM · Performance-Team, Commons, Multimedia, MediaWiki-File-management
Gilles changed the status of T27611: Support optimized WebP thumbnails as alternative to JPEG, PNG, a subtask of T114791: Explore lazy-loading WebP thumbnails to supporting browsers (eg Android Chrome), from Open to Stalled.
Thu, Dec 20, 9:51 AM · Readers-Web-Backlog (Tracking), Performance-Team, Commons, MediaWiki-File-management, MobileFrontend, Multimedia, Performance
Gilles added a comment to T27611: Support optimized WebP thumbnails as alternative to JPEG, PNG.

Expanding the amount of WebP thumbnails we served is blocked on T211661: Automatically clean up unused thumbnails in Swift

Thu, Dec 20, 9:51 AM · Performance-Team, Commons, Multimedia, MediaWiki-File-management
Gilles added a comment to T209886: Assess Thumbor upgrade options.

If the files are already on Beta, purge them, make sure your browser cache is cleared for these images, and you'll get thumbnails generated with librsvg 2.40.20-3

Thu, Dec 20, 8:05 AM · Patch-For-Review, serviceops, User-jijiki, Performance-Team (Radar), Operations, Thumbor

Dec 19 2018

Gilles triaged T212304: Ask the eswiki community whether we can run the performance perception survey there as Normal priority.
Dec 19 2018, 3:22 PM · QuickSurveys (Surveys), CommRel-Specialists-Support (Jan-Mar-2019), Readers-Web-Backlog (Tracking), Performance-Team
Gilles lowered the priority of T210484: Only serve debug HTTP headers when x-wikimedia-debug is present from Normal to Low.
Dec 19 2018, 3:03 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

After brainstorming this more, since Nginx TLS termination is going to remain for the foreseeable future, even after we move backend caches to ATS, it makes this effort simpler to blacklist specific debugging header at the Nginx level and have the debug "gate" there. This way the list of filtered headers is defined in one central location and is guaranteed not to mess with Varnish log data collection.

Dec 19 2018, 3:02 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles removed a project from T209857: Create Autonomous Systems ranking based on RUM data: Patch-For-Review.
Dec 19 2018, 3:00 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles closed T207862: Use Server-Timing to collect Varnish hit/miss information in NavigationTiming as Resolved.

The data is making it into the ServerTiming schema as expected now, as verified on Hive.

Dec 19 2018, 2:59 PM · MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Patch-For-Review, Performance-Team
Gilles closed T211667: Study the compression performance of single-frame AV1 as Resolved.

Will post blog post soon, either on the perf calendar or our own blog if it doesn't get picked up for that.

Dec 19 2018, 2:41 PM · Performance-Team

Dec 18 2018

Gilles added a comment to T212124: Consider adding decoding=async to our img tags.

It sounds like by default depending on some heuristics the browser might block text on image decoding, presumably to flush them both in the same paint. Imho this sounds like async decoding would be a win, as it would allow text or other things to get flushed in situations where it would have been blocked by image decoding.

Dec 18 2018, 9:26 AM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team

Dec 17 2018

Gilles moved T212124: Consider adding decoding=async to our img tags from Inbox to Backlog: Small / Maintenance on the Performance-Team board.
Dec 17 2018, 9:15 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team
Gilles claimed T212124: Consider adding decoding=async to our img tags.
Dec 17 2018, 9:13 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team
Gilles updated the task description for T212124: Consider adding decoding=async to our img tags.
Dec 17 2018, 3:36 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team
Gilles created T212124: Consider adding decoding=async to our img tags.
Dec 17 2018, 2:41 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Patch-For-Review, MediaWiki-General-or-Unknown, MediaWiki-extensions-General, Parsing-Team, Performance-Team
Gilles added a comment to T212081: PNG thumbnail of SVG file not rendered correctly, mostly black.

If they're happening in production, yes.

Dec 17 2018, 12:33 PM · Multimedia, MediaWiki-File-management, Commons

Dec 14 2018

Gilles added a comment to T211667: Study the compression performance of single-frame AV1.

Emailed the 2 Google engineers who wrote most of the WebP container spec about the idea of extending WebP to allow AV1 as a codec to encode frames. One of them seems to be working on AV1 anyway. Based on the spec, it seems like a very straightforward thing to do, and Chrome already has AV1 decoding for video.

Dec 14 2018, 8:59 AM · Performance-Team
Gilles added a comment to T201409: Harmonise the identification of requests across our stack.

@Imarlier https://translate.google.com/#view=home&op=translate&sl=et&tl=en&text=krinkle

Dec 14 2018, 8:45 AM · Core Platform Team Backlog (Designing), User-CDanis, TechCom-RFC (TechCom-Approved), Performance-Team (Radar), Operations, Services (designing), User-mobrovac, Traffic
Gilles added a comment to T211667: Study the compression performance of single-frame AV1.

I've figured out a way to test this with the latest ffmpeg and libaom, and holy crap the hype is real. When I hit the same DSSIM fidelity for the AV1 "thumbnail" as the JPG and WEBP ones from production, I get the following. Note that the embedded images in this comment are lossless PNG conversions of the actual thumbnails, provided here to eyeball visual differences.

Dec 14 2018, 5:42 AM · Performance-Team

Dec 13 2018

Gilles added a comment to T211906: Expose PHP7/HHVM to NavTiming in a header, send with navtiming beacon so we can use it as a dimension.

If there's already a cookie planned to enable that, I think that's the easiest thing to use here. Server-Timing has limited cross-browser support at this point: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing

Dec 13 2018, 8:01 PM · MediaWiki-extensions-NavigationTiming, Performance-Team

Dec 11 2018

Gilles closed T197607: Add ability to oversample specific pages as Resolved.

NavigationTiming doesn't store article id, but it stores the revision id. For a popular article like https://ru.wikipedia.org/wiki/Россия which is part of the oversampled articles, it gives a recognisable revision id. Last edit was on the 8th, with revision id 96715436. Let's look at how often navtiming was recorded for it on the 9th and on the 10th once per-page oversampling was enabled (mid-day):

Dec 11 2018, 10:22 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Performance-Team, MediaWiki-extensions-NavigationTiming
Gilles added a comment to T197607: Add ability to oversample specific pages.

I'm not sure that it worked or that the 5 articles picked made a big difference, there's no visible uptake in ruwiki survey responses:

Dec 11 2018, 9:44 AM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Performance-Team, MediaWiki-extensions-NavigationTiming
Gilles triaged T211667: Study the compression performance of single-frame AV1 as Normal priority.
Dec 11 2018, 9:40 AM · Performance-Team
Gilles created T211667: Study the compression performance of single-frame AV1.
Dec 11 2018, 9:40 AM · Performance-Team
fgiunchedi awarded T211661: Automatically clean up unused thumbnails in Swift a Yellow Medal token.
Dec 11 2018, 9:29 AM · Traffic, media-storage, Operations, Performance-Team
Gilles triaged T211661: Automatically clean up unused thumbnails in Swift as Normal priority.
Dec 11 2018, 9:27 AM · Traffic, media-storage, Operations, Performance-Team
Gilles created T211661: Automatically clean up unused thumbnails in Swift.
Dec 11 2018, 9:27 AM · Traffic, media-storage, Operations, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

Analytics needs x-analytics in every request, not only in debugging ones but we don't need to include it in the response headers. We don't know if downstream that header is needed.

Dec 11 2018, 9:18 AM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

what about ?debug=true ? We already vary on that right ? might as well vary which set of headers is let true...

Dec 11 2018, 9:17 AM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.

@Anomie very good point, I think it will be very hard for someone to find out about such a whitelist. Things will work for them on Vagrant and possibly on Beta, leading to unexpected breakage upon production deployment when their new header is just missing.

Dec 11 2018, 9:15 AM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team

Dec 10 2018

Gilles claimed T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.
Dec 10 2018, 9:07 PM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles added a comment to T209886: Assess Thumbor upgrade options.

Looking good!

Dec 10 2018, 12:27 PM · Patch-For-Review, serviceops, User-jijiki, Performance-Team (Radar), Operations, Thumbor
Gilles added a comment to T207862: Use Server-Timing to collect Varnish hit/miss information in NavigationTiming.

The object isn't sealed, isn't frozen, is extensible, and if I output it after modifying it, it has the new property. But the new property doesn't survive JSON.stringify, because the new property isn't included by the object's toJSON():

Dec 10 2018, 10:49 AM · MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Patch-For-Review, Performance-Team

Dec 5 2018

Gilles moved T27611: Support optimized WebP thumbnails as alternative to JPEG, PNG from Blocked to Doing on the Performance-Team board.
Dec 5 2018, 4:01 PM · Performance-Team, Commons, Multimedia, MediaWiki-File-management
Gilles moved T194640: Leverage Priority Hints from Backlog: Small / Maintenance to Blocked on the Performance-Team board.
Dec 5 2018, 4:01 PM · Performance-Team
Gilles committed rTHMBREXT1632a81283e6: Use webp -exact option on Stretch (authored by Gilles).
Use webp -exact option on Stretch
Dec 5 2018, 3:46 PM

Nov 30 2018

Gilles closed T208897: PHP Fatal Error: Argument passed to MessageCache::isMainCacheable() must be array as Resolved.

Please reopen if it's seen again.

Nov 30 2018, 12:29 PM · Performance-Team, MediaWiki-Cache, Wikimedia-production-error
Gilles added a comment to T207862: Use Server-Timing to collect Varnish hit/miss information in NavigationTiming.

Doesn't work, I see nothing in hive and this happens a bunch:

Nov 30 2018, 11:43 AM · MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Patch-For-Review, Performance-Team

Nov 29 2018

Gilles closed T210141: Test our production stack's HTTP/2 priority support as Resolved.
Nov 29 2018, 12:47 PM · Performance-Team
Gilles closed T207615: Varnish won't purge thumbnails of specific file as Declined.

Sure, 'till next time ;)

Nov 29 2018, 12:14 PM · Traffic, Operations

Nov 27 2018

Gilles added a comment to T209857: Create Autonomous Systems ranking based on RUM data.

In order to allow future work on this based on the CPU benchmark results, I need to expand the scope of the CPU benchmark beyond the perception survey. I think a sub-sampling ratio of NavTiming samples would make sense.

Nov 27 2018, 1:56 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles added a comment to T208897: PHP Fatal Error: Argument passed to MessageCache::isMainCacheable() must be array.

Unless I'm querying logstash incorrectly, I don't see any occurence of the original fatal in the last 7 days.

Nov 27 2018, 1:46 PM · Performance-Team, MediaWiki-Cache, Wikimedia-production-error
Gilles triaged T208015: Preload top article image on desktop as Low priority.
Nov 27 2018, 1:27 PM · Performance-Team
Gilles removed a project from T27611: Support optimized WebP thumbnails as alternative to JPEG, PNG: Patch-For-Review.
Nov 27 2018, 1:26 PM · Performance-Team, Commons, Multimedia, MediaWiki-File-management
Gilles removed a project from T197607: Add ability to oversample specific pages: Patch-For-Review.
Nov 27 2018, 1:26 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Performance-Team, MediaWiki-extensions-NavigationTiming
Gilles removed a project from T207862: Use Server-Timing to collect Varnish hit/miss information in NavigationTiming: Patch-For-Review.
Nov 27 2018, 1:26 PM · MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Patch-For-Review, Performance-Team
Gilles removed a project from T209857: Create Autonomous Systems ranking based on RUM data: Patch-For-Review.
Nov 27 2018, 1:25 PM · Patch-For-Review, MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), Performance-Team
Gilles removed a project from T210141: Test our production stack's HTTP/2 priority support: Patch-For-Review.
Nov 27 2018, 1:25 PM · Performance-Team
Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

Alright, I think this is the closest we can get to running something like the future ATS setup:

Nov 27 2018, 1:14 PM · Performance-Team
Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

Uploaded the test images to mediawiki.org, so that we can have them served from upload.wikimedia.org on a separate connection:

Nov 27 2018, 12:02 PM · Performance-Team
Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

There's already an ATS cluster we can hit internally. The config in ATS is unified for text and upload.

Nov 27 2018, 10:06 AM · Performance-Team
Gilles created T210484: Only serve debug HTTP headers when x-wikimedia-debug is present.
Nov 27 2018, 9:53 AM · Patch-For-Review, Operations, Analytics, Traffic, Performance-Team
Gilles reassigned T207941: Spike of DBTransactionSizeError exceptions from /w/api.php from Special:Watchlist from Gilles to Catrope.
Nov 27 2018, 8:31 AM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Growth-Team (Current Sprint), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error

Nov 26 2018

Gilles added a comment to T207941: Spike of DBTransactionSizeError exceptions from /w/api.php from Special:Watchlist.

Looking at potential callers, it seems to be coming from the "Mark all changes as seen" button on Special:Watchlist. Which, when clicked, after firing the API call we're seeing, triggers an update of the UI by fetching the changes list from the server. I'm guessing that this is probably the reason why the API call tries to do all this work synchronously. I think this calls for the API call doing chunking and still being synchronous, if we want to avoid drastic UI+API changes.

Nov 26 2018, 11:39 AM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Growth-Team (Current Sprint), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error
Gilles added a comment to T207941: Spike of DBTransactionSizeError exceptions from /w/api.php from Special:Watchlist.

If I'm following the code correctly based on the API parameters, it ends up calling setNotificationTimestampsForUser in WatchedItemStore, which seems to update all rows at once for the user calling this API in the watchlist table, setting the wl_notificationtimestamp column to null. It's doing the whole thing inside the POST, as the return status of the API call seems to depend on the success or failure of that update call.

Nov 26 2018, 9:23 AM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Growth-Team (Current Sprint), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error
Gilles updated subscribers of T207941: Spike of DBTransactionSizeError exceptions from /w/api.php from Special:Watchlist.

@Anomie you were the last to touch this API (ApiSetNotificationTimestamp) for MCR. Does it look to you like something that could be caused by recent MCR changes?

Nov 26 2018, 9:07 AM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Growth-Team (Current Sprint), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error
Gilles added a comment to T207941: Spike of DBTransactionSizeError exceptions from /w/api.php from Special:Watchlist.

Some recent instances in api.log:

Nov 26 2018, 9:02 AM · MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), Growth-Team (Current Sprint), Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error

Nov 22 2018

Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

There's one thing that's definitely bad in our results, though, which is that since it's interleaving everything, the high priority stuff is sharing bandwidth with low priority things. Some of that is happening in Pat's "good" example as well, though, with the first hidden image competing with the high priority things.

Nov 22 2018, 2:01 PM · Performance-Team
Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

It looks like we're doing fine: https://www.webpagetest.org/result/181122_MY_402f279db0f28ab1957bb10b5d551d61/3/details/#waterfall_view_step1

Nov 22 2018, 1:56 PM · Performance-Team
Gilles added a comment to T210141: Test our production stack's HTTP/2 priority support.

https://www.webpagetest.org/result/181122_MY_402f279db0f28ab1957bb10b5d551d61/

Nov 22 2018, 1:38 PM · Performance-Team