Gilles (Gilles Dubuc)
Senior Performance Engineer, WMF

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 8:24 AM (176 w, 1 d)
Availability
Available
IRC Nick
gilles
LDAP User
Gilles
MediaWiki User
Unknown

Recent Activity

Today

Gilles updated the task description for T165272: Review research on performance perception.
Wed, Feb 21, 10:00 AM · Performance-Team
Gilles added a comment to T165272: Review research on performance perception.

"The Impact of Waiting Time Distributions on QoE of Task-Based Web Browsing Sessions" Islam Nazrul, Elepe Vijaya John David 2014

Wed, Feb 21, 9:59 AM · Performance-Team
Gilles closed T187167: Give mw:thumbor read/write access to all private Swift containers as Resolved.
Wed, Feb 21, 8:59 AM · Performance-Team (Radar), User-fgiunchedi, Thumbor
Gilles closed T187167: Give mw:thumbor read/write access to all private Swift containers, a subtask of T169144: Serve thumb.php requests with Thumbor, as Resolved.
Wed, Feb 21, 8:59 AM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor

Yesterday

Gilles triaged T187822: Have Thumbor use a different Swift user when dealing with private containers as Low priority.
Tue, Feb 20, 7:05 PM · Performance-Team, Thumbor
Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.

Running this on Terbium would probably give mw:thumbor access to the containers of all private wikis:

Tue, Feb 20, 5:55 PM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.

For Beta I'm going to use Wikisource for testing: https://upload.beta.wmflabs.org/wikisource/en/thumb/6/62/Wind_in_the_Willows_%281913%29.djvu/page1-862px-Wind_in_the_Willows_%281913%29.djvu.jpg

Tue, Feb 20, 3:24 PM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles updated the task description for T165272: Review research on performance perception.
Tue, Feb 20, 2:11 PM · Performance-Team
Gilles updated the task description for T184510: Ideas for performance perception studies.
Tue, Feb 20, 2:08 PM · Performance-Team
Gilles added a comment to T116132: missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs.

legacy HTTP/1 UAs may suffer due to UA limits

Tue, Feb 20, 1:49 PM · Performance-Team (Radar), Operations, Performance, Traffic
Gilles added a comment to T187765: Replace the Nginx fronting Thumbor with a reverse proxy capable of queuing requests.

It looks like this might be achievable with a combination of Apache, MPM prefork and mod_wsgi. Apache would run a fixed amount of child processes (eg. as many as there are cores), not use worker threads, and mod_wsgi would run a copy of the thumbor app in each process. Number or processes capped. When it's reached, Apache should be waiting until the next available process to treat the next request.

Tue, Feb 20, 11:42 AM · Performance-Team, Thumbor
Gilles lowered the priority of T187203: Modify upstream Thumbor to allow true async engines from Low to Lowest.
Tue, Feb 20, 10:24 AM · Performance-Team, Thumbor
Gilles triaged T187765: Replace the Nginx fronting Thumbor with a reverse proxy capable of queuing requests as Low priority.
Tue, Feb 20, 9:07 AM · Performance-Team, Thumbor
Gilles created T187765: Replace the Nginx fronting Thumbor with a reverse proxy capable of queuing requests.
Tue, Feb 20, 9:06 AM · Performance-Team, Thumbor

Mon, Feb 19

Gilles moved T116132: missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs from Inbox to Radar on the Performance-Team board.
Mon, Feb 19, 9:57 PM · Performance-Team (Radar), Operations, Performance, Traffic
Gilles moved T187167: Give mw:thumbor read/write access to all private Swift containers from Inbox to Radar on the Performance-Team board.
Mon, Feb 19, 9:37 PM · Performance-Team (Radar), User-fgiunchedi, Thumbor
Gilles moved T187203: Modify upstream Thumbor to allow true async engines from Inbox to Backlog on the Performance-Team board.
Mon, Feb 19, 9:36 PM · Performance-Team, Thumbor
Gilles moved T187299: User-perceived page load performance study from Inbox to Next-up on the Performance-Team board.
Mon, Feb 19, 9:36 PM · Performance-Team
Gilles moved T187350: Add thumbor and thumbor-plugins versions to log entries/errors from Inbox to Backlog on the Performance-Team board.
Mon, Feb 19, 9:36 PM · Performance-Team, Thumbor
Gilles added a comment to T187203: Modify upstream Thumbor to allow true async engines.
Mon, Feb 19, 9:08 PM · Performance-Team, Thumbor
Gilles added a comment to T187684: Define key metrics (3-5 total) that we're tracking and go public with the metrics.

I can't find the definition of last visual change, does it include things below the fold?

Mon, Feb 19, 7:55 PM · Performance-Team
Gilles added a project to T116132: missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs: Performance-Team.

Now that HTTP/2 is a thing and Zero won't be anymore, could we put connection coalescing for upload.wikimedia.org back on the table?

Mon, Feb 19, 7:37 PM · Performance-Team (Radar), Operations, Performance, Traffic
Gilles updated the task description for T123582: Use "preconnect" resource hint for thumbnail host.
Mon, Feb 19, 5:49 PM · Performance-Team (Radar), MediaWiki-General-or-Unknown, Multimedia
Gilles added a comment to T179977: Change use of preconnect Link to header.

I'll add a note to T123582: Use "preconnect" resource hint for thumbnail host to state that the header variant should be used if this gets implemented.

Mon, Feb 19, 5:48 PM · Performance-Team

Wed, Feb 14

Gilles added a comment to T187277: Should it be possible for a schema to override DNT in exceptional circumstances?.

The problem is that the mechanics of EventLogging aren't collecting just aggregate counts, they're keeping individual records about requests. If the virtual pageviews were being sent to a backend that only keeps aggregate counts, incrementing on each hit, then that might be fine. You essentially want to turn a blind eye to the detailed data EventLogging records because you won't use it, but it's there, it's being recorded. The definition of DNT is loose enough that people keep making compromises to them. But IMHO in this case if you want to record virtual pageviews and only care about aggregate counts, you should be sending the data to a backend that only records aggregates.

Wed, Feb 14, 6:34 PM · Analytics, Analytics-EventLogging
Gilles closed T187159: Add prefixes to custom Thumbor headers as Resolved.
Wed, Feb 14, 5:53 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles updated the task description for T187203: Modify upstream Thumbor to allow true async engines.
Wed, Feb 14, 5:52 PM · Performance-Team, Thumbor
Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.
  • Use a separate Swift user for private containers access. This will allow avoiding access leak mistakes where a new private wiki is created and the whitelist in puppet not updated.
    • Check that the wiki creation maintenance script doesn't give access to the regular Thumbor user when it's a private wiki being created. Instead, give access to the private-specific thumbor user
Wed, Feb 14, 5:41 PM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles closed T187336: Thumbor: TypeError: 'NoneType' object is not iterable as Resolved.
Wed, Feb 14, 5:26 PM · Performance-Team, Thumbor
Gilles closed T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file' as Resolved.
Wed, Feb 14, 5:26 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles triaged T187350: Add thumbor and thumbor-plugins versions to log entries/errors as Low priority.
Wed, Feb 14, 5:24 PM · Performance-Team, Thumbor
Gilles created T187350: Add thumbor and thumbor-plugins versions to log entries/errors.
Wed, Feb 14, 5:24 PM · Performance-Team, Thumbor
Gilles updated the task description for T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file'.
Wed, Feb 14, 4:26 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles lowered the priority of T187336: Thumbor: TypeError: 'NoneType' object is not iterable from High to Normal.
Wed, Feb 14, 4:24 PM · Performance-Team, Thumbor
Gilles added a commit to T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file': rTHMBREXT4c7ef3c4cb55: Version bump.
Wed, Feb 14, 4:15 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles committed rTHMBREXT4c7ef3c4cb55: Version bump (authored by Gilles).
Version bump
Wed, Feb 14, 4:15 PM
Gilles updated the task description for T187336: Thumbor: TypeError: 'NoneType' object is not iterable.
Wed, Feb 14, 4:12 PM · Performance-Team, Thumbor
Gilles updated the task description for T187336: Thumbor: TypeError: 'NoneType' object is not iterable.
Wed, Feb 14, 4:11 PM · Performance-Team, Thumbor
Gilles closed T179954: Thumbor errors should contain a trackable request id as Resolved.
Wed, Feb 14, 4:10 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles closed T179954: Thumbor errors should contain a trackable request id, a subtask of T157850: Interacting with Wikimedia logs should be a pleasant experience, as Resolved.
Wed, Feb 14, 4:10 PM · Epic, Wikimedia-General-or-Unknown
Gilles closed T179954: Thumbor errors should contain a trackable request id, a subtask of T179200: Not showing my own image of nokia 3310, as Resolved.
Wed, Feb 14, 4:10 PM · Thumbor, Multimedia
Gilles updated the task description for T187336: Thumbor: TypeError: 'NoneType' object is not iterable.
Wed, Feb 14, 4:08 PM · Performance-Team, Thumbor
Gilles updated the task description for T187336: Thumbor: TypeError: 'NoneType' object is not iterable.
Wed, Feb 14, 4:07 PM · Performance-Team, Thumbor
Gilles triaged T187336: Thumbor: TypeError: 'NoneType' object is not iterable as High priority.
Wed, Feb 14, 4:07 PM · Performance-Team, Thumbor
Gilles created T187336: Thumbor: TypeError: 'NoneType' object is not iterable.
Wed, Feb 14, 4:07 PM · Performance-Team, Thumbor
Gilles committed rTHMBREXT0ece18d8d482: Fix wikimedia_original_file AttributeError (authored by Gilles).
Fix wikimedia_original_file AttributeError
Wed, Feb 14, 4:06 PM
Gilles added a commit to T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file': rTHMBREXT0ece18d8d482: Fix wikimedia_original_file AttributeError.
Wed, Feb 14, 4:06 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles triaged T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file' as High priority.
Wed, Feb 14, 4:03 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles created T187335: Thumbor: AttributeError: Context instance has no attribute 'wikimedia_original_file'.
Wed, Feb 14, 4:03 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles closed T187088: Thumbor won't render some SVGs whose namespace is too far into the file as Resolved.
Wed, Feb 14, 4:02 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles committed rTHMBREXT5be43e413989: Version bump (authored by Gilles).
Version bump
Wed, Feb 14, 10:26 AM
Gilles added a commit to T179954: Thumbor errors should contain a trackable request id: rTHMBREXT5be43e413989: Version bump.
Wed, Feb 14, 10:26 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a commit to T187088: Thumbor won't render some SVGs whose namespace is too far into the file: rTHMBREXT5be43e413989: Version bump.
Wed, Feb 14, 10:25 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a commit to T187159: Add prefixes to custom Thumbor headers: rTHMBREXT5be43e413989: Version bump.
Wed, Feb 14, 10:25 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles committed rTHMBREXTfc9cc9dc48ef: Fix header tests (authored by Gilles).
Fix header tests
Wed, Feb 14, 10:23 AM
Gilles added a commit to T187159: Add prefixes to custom Thumbor headers: rTHMBREXTfc9cc9dc48ef: Fix header tests.
Wed, Feb 14, 10:23 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles committed rTHMBREXT60a20131f22c: Loosen SVG file type check (authored by Gilles).
Loosen SVG file type check
Wed, Feb 14, 10:16 AM
Gilles added a commit to T187088: Thumbor won't render some SVGs whose namespace is too far into the file: rTHMBREXT60a20131f22c: Loosen SVG file type check.
Wed, Feb 14, 10:16 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles closed D975: Loosen SVG file type check.
Wed, Feb 14, 10:16 AM
Gilles updated the task description for T187299: User-perceived page load performance study.
Wed, Feb 14, 10:14 AM · Performance-Team
Gilles updated the task description for T165272: Review research on performance perception.
Wed, Feb 14, 10:08 AM · Performance-Team
Gilles triaged T187299: User-perceived page load performance study as Normal priority.
Wed, Feb 14, 9:41 AM · Performance-Team

Tue, Feb 13

Gilles updated the diff for D975: Loosen SVG file type check.

Fix copyright year and variable name

Tue, Feb 13, 4:22 PM
Gilles committed rTHMBREXT26ebba5a4764: Add prefixes to custom Thumbor headers (authored by Gilles).
Add prefixes to custom Thumbor headers
Tue, Feb 13, 4:18 PM
Gilles committed rTHMBREXT64bc4a88f4ab: Add thumbor-request-id to all custom logger.error (authored by Gilles).
Add thumbor-request-id to all custom logger.error
Tue, Feb 13, 4:18 PM
Gilles added a commit to T179954: Thumbor errors should contain a trackable request id: rTHMBREXT64bc4a88f4ab: Add thumbor-request-id to all custom logger.error.
Tue, Feb 13, 4:18 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a commit to T187159: Add prefixes to custom Thumbor headers: rTHMBREXT26ebba5a4764: Add prefixes to custom Thumbor headers.
Tue, Feb 13, 4:18 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles closed D974: Add thumbor-request-id to all custom logger.error.
Tue, Feb 13, 4:18 PM
Gilles closed D973: Add prefixes to custom Thumbor headers.
Tue, Feb 13, 4:18 PM
Gilles added a comment to T187203: Modify upstream Thumbor to allow true async engines.

I've confirmed via testing on Vagrant that this phenomenon is real.

Tue, Feb 13, 4:15 PM · Performance-Team, Thumbor
Gilles closed T185466: Review Varnish slow log for upload.wikimedia.org as Resolved.

I think this concludes the review, I've filed a task for the real solution to that problem: https://phabricator.wikimedia.org/T187203

Tue, Feb 13, 2:50 PM · Patch-For-Review, Performance-Team
Gilles triaged T187203: Modify upstream Thumbor to allow true async engines as Low priority.
Tue, Feb 13, 2:49 PM · Performance-Team, Thumbor
Gilles created T187203: Modify upstream Thumbor to allow true async engines.
Tue, Feb 13, 2:49 PM · Performance-Team, Thumbor
Gilles added a comment to T185466: Review Varnish slow log for upload.wikimedia.org.

OK, it's definitely not the nginx timeouts. 404s and 200s are affected, now that I can see 200s. And I'm seeing now that it's consistently 59 seconds, not 60 seconds.

Tue, Feb 13, 2:46 PM · Patch-For-Review, Performance-Team
Gilles requested review of D975: Loosen SVG file type check.
Tue, Feb 13, 2:35 PM
Gilles added a project to T187088: Thumbor won't render some SVGs whose namespace is too far into the file: Patch-For-Review.
Tue, Feb 13, 2:35 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a revision to T187088: Thumbor won't render some SVGs whose namespace is too far into the file: D975: Loosen SVG file type check.
Tue, Feb 13, 2:34 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles requested review of D974: Add thumbor-request-id to all custom logger.error.
Tue, Feb 13, 1:42 PM
Gilles added a project to T179954: Thumbor errors should contain a trackable request id: Patch-For-Review.
Tue, Feb 13, 1:42 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a revision to T179954: Thumbor errors should contain a trackable request id: D974: Add thumbor-request-id to all custom logger.error.
Tue, Feb 13, 1:42 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a project to T187159: Add prefixes to custom Thumbor headers: Patch-For-Review.
Tue, Feb 13, 1:17 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles requested review of D973: Add prefixes to custom Thumbor headers.
Tue, Feb 13, 1:17 PM
Gilles added a revision to T187159: Add prefixes to custom Thumbor headers: D973: Add prefixes to custom Thumbor headers.
Tue, Feb 13, 1:16 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles triaged T187167: Give mw:thumbor read/write access to all private Swift containers as Normal priority.
Tue, Feb 13, 10:40 AM · Performance-Team (Radar), User-fgiunchedi, Thumbor
Gilles added a comment to T185466: Review Varnish slow log for upload.wikimedia.org.

And another one where it's a minute: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.02.13/logback?id=AWGOuxL5WMYqG9UiQ1bY&_g=(refreshInterval:('$$hashKey':'object:5953',display:'5%20seconds',pause:!f,section:1,value:5000),time:(from:now-15m,mode:quick,to:now))

Tue, Feb 13, 10:37 AM · Patch-For-Review, Performance-Team
Gilles added a comment to T185466: Review Varnish slow log for upload.wikimedia.org.

Now that 200s log the Thumbor-Request-Date, I've found an example of a 200 with 40s of delay between nginx and thumbor: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.02.13/logback?id=AWGOuA7cWMYqG9UiQxzA&_g=()

Tue, Feb 13, 10:36 AM · Patch-For-Review, Performance-Team
Gilles closed T186594: Log time when request makes it to Thumbor as Resolved.
Tue, Feb 13, 10:35 AM · Patch-For-Review, Thumbor, Performance-Team
Gilles closed T186594: Log time when request makes it to Thumbor, a subtask of T185466: Review Varnish slow log for upload.wikimedia.org, as Resolved.
Tue, Feb 13, 10:35 AM · Patch-For-Review, Performance-Team
Gilles closed T168929: Identify source of gadget/extension pointing to thumb.php on imagescalers as Declined.

This cleanup is no longer necessary, as thumb.php requests are now proxied to Thumbor on public wikis.

Tue, Feb 13, 10:11 AM · Chinese-Sites, Wikimedia-General-or-Unknown, Performance-Team, Thumbor
Gilles closed T168929: Identify source of gadget/extension pointing to thumb.php on imagescalers, a subtask of T121388: Service-based thumbnailing re-architecture in production with Thumbor, as Declined.
Tue, Feb 13, 10:11 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.

https://commons.wikimedia.org/w/thumb.php?f=Broom%20icon.svg&w=500
https://commons.wikimedia.beta.wmflabs.org/w/thumb.php?f=Victoria_memorial.jpg&w=300

Tue, Feb 13, 10:10 AM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.

Fixed on Beta, I has just modified the wrong PrivateSettings.php

Tue, Feb 13, 10:08 AM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles triaged T187159: Add prefixes to custom Thumbor headers as Low priority.
Tue, Feb 13, 8:33 AM · Patch-For-Review, Performance-Team, Thumbor
Gilles created T187159: Add prefixes to custom Thumbor headers.
Tue, Feb 13, 8:33 AM · Patch-For-Review, Performance-Team, Thumbor

Mon, Feb 12

Gilles added a comment to T169144: Serve thumb.php requests with Thumbor.

Deployment was successful on prod, seems to have broken thumb.php on Beta. I'll look into that tomorrow. Not high priority because thumb.php requests are highly unusual because they need to be manually crafted (and would thus be even more unusual on Beta).

Mon, Feb 12, 7:28 PM · Patch-For-Review, MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), Performance-Team, Thumbor
Gilles updated the task description for T165272: Review research on performance perception.
Mon, Feb 12, 6:37 PM · Performance-Team
Gilles added a comment to T165272: Review research on performance perception.

"Towards Better Measurement of Attention and Satisfaction in Mobile Search" Lagun, Hsieh, Webster, Navalpakkam 2014

Mon, Feb 12, 6:37 PM · Performance-Team
Gilles added a comment to T185466: Review Varnish slow log for upload.wikimedia.org.

I'm tempted to bump the only 2 nginx timeouts that currently default to 60s (we override proxy_read_timeout to 180s):

Mon, Feb 12, 5:14 PM · Patch-For-Review, Performance-Team
Gilles updated subscribers of T187088: Thumbor won't render some SVGs whose namespace is too far into the file.
Mon, Feb 12, 4:58 PM · Patch-For-Review, Performance-Team, Thumbor
Gilles reopened T186594: Log time when request makes it to Thumbor as "Open".

Actually doesn't show up for 200s yet, but I think that's because the swift proxy filtering kicks in only in this case (which is interesting in itself - error responses aren't subject to the header filtering).

Mon, Feb 12, 4:43 PM · Patch-For-Review, Thumbor, Performance-Team