Hi Mark! Could you summarize the back-and-forth? What were the alternatives considered?

In lieu of exporting a route map, MediaWiki could, as a first pass at the problem, emit a response header that signals to the CDN that a request contained garbage parameters. The CDN could use this information to throttle clients that issue too many such requests. This may be less desirable than filtering all such requests at the edge, but it is also simpler.

It's a bug in webhint, AFAICT. It thinks stale-while-revalidate should not hold a value, but that is wrong. This is the problematic code:

In T211661#9054485, @MatthewVernon wrote:

The other thing I can't quite leave alone is - why are we being asked for some thumbnails so often? Shouldn't the CDN be caching thumbs? If we served each thumb only once in that 24 hour period, that would have saved about 54 million requests to swift (which is 29% of the requests swift served), which is non-trivial...

Commonest-served thumbs on that day (with request counts):

8924 wikipedia-commons-local-thumb.8e/8/8e/Edit_remove.svg/15px-Edit_remove.svg.png
8053 wikipedia-commons-local-thumb.2c/2/2c/Broom_icon.svg/22px-Broom_icon.svg.png
6268 wikipedia-commons-local-thumb.de/d/de/Wynn.svg/25px-Wynn.svg.png
6264 wikipedia-commons-local-thumb.33/3/33/Crystal_Clear_action_viewmag.png/22px-Crystal_Clear_action_viewmag.png
6258 wikipedia-commons-local-thumb.1e/1/1e/Font_Awesome_5_solid_arrow-down.svg/19px-Font_Awesome_5_solid_arrow-down.svg.png
6256 wikipedia-commons-local-thumb.b2/b/b2/Font_Awesome_5_solid_arrow-up.svg/19px-Font_Awesome_5_solid_arrow-up.svg.png
5706 wikipedia-commons-local-thumb.b3/b/b3/Broom_icon_ref.svg/22px-Broom_icon_ref.svg.png
4990 wikipedia-commons-local-thumb.33/3/33/Crystal_Clear_action_viewmag.png/21px-Crystal_Clear_action_viewmag.png

I also don't know how well Swift would handle 15k QPS of object metadata updates (cf T211661#8377883)

Right. Now I remember. The initial expiration is indeed supposed to be set by Thumbor. The necessary functionality had some trouble landing in the Wikimedia Thumbor plugin repo, but it has since landed.

@MatthewVernon: my understanding is that rewrite.py is currently setting expiry headers for thumbnails on retrieval from Swift -- is that correct, and does that mean some thumbnails are already getting expired?

This is really confusing.

Vega ships an optional interpreter that can evaluate graph expressions by traversing an AST and performing each operation, rather than relying on runtime code generation. Per https://github.com/vega/vega/pull/3019#issuecomment-749107902, the interpreter mode is not the default because it is 10% slower. Seems like a negligible price to me. This seems like the only sensible option for keeping support for graph expressions but rooting out XSS vectors systematically.

@phuedx I don't know, sorry.

Does the edits graph in T327440#8542723 include bots? Bots may not be a large proportion of users but they do contribute a large proportion of edits.

+1 to @Tgr's proposal

It might be worth it to try and contact the library's co-maintainer. His contact info is at https://eatingco.de/about/.

ㅤ

@Jdforrester-WMF : the Beta Cluster instance of the function-evaluator now runs under GVisor. Some additional work will be required to make the production instance of the function-evaluator run under GVisor. There is documentation here: https://gvisor.dev/docs/user_guide/quick_start/kubernetes/.

I created a new task for the alerts, T321099. Let's continue there.

Wikifunctions on the Beta Cluster uses the *.wikimedia.beta.wmflabs.org wildcard cert, and the CertAlmostExpired alert was caused by automatic certificate renewal being broken on the Beta Cluster in general. T293585 is the issue; it looks like Valentin and Giuseppe fixed it.

@phuedx I'm not aware of anything actively using it, no, but I'm also out of the loop -- can you ask someone on the performance team to confirm?

Done by Ali: https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Semantics_of_Wikifunctions

Relevant: Provably-Safe Multilingual Software Sandboxing using WebAssembly

I've cherry-picked the two Puppet patches on the beta cluster. The mediawiki-function-evaluator service is now running under gVisor.

Never mind, I see that it is available for Bullseye -- sorry.

@Joe the Wikifunctions Beta Cluster instance is running Bullseye -- could you also pull it in there?

There are no outstanding issues that are specific to the Beta Cluster environment, AFAIK.

@cmassaro We have some logging now, and instructions on Wikitech on how to access the logs. I think there are more places where we can add additional logging to make debugging easier, but that is better dealt with on an ongoing basis than a dedicated task.

I suspect this is fallout from the URL query sorting change (cc @ori) not invalidating the cache of history pages properly.

In T132418#5508073, @Krinkle wrote:

Chrome is shipping this as of Chrome 75. Time to reconsider!

In T316886#8208837, @cmassaro wrote:

Can we see the function call that was sent? Even just copying the ZObject from expert mode in the UI will help.

That no longer looks like an error that would be specific to the Beta cluster environment. @AAssaf-WMF , can you see if you get the same error locally?

The API Sandbox request in the task description is still failing, but the underlying error is now different:

I think you can create a patch to remove it from package.json, and we'll see if all the integration tests pass. If anything breaks after merge we can always revert easily.

OK, it looks like the default User-Agent string sent by node-fetch is blocked by Varnish:
https://github.com/wikimedia/puppet/blob/9843300dba/modules/varnish/templates/wikimedia-frontend.vcl.erb#L716-L718
We need to set a custom user-agent string for the orchestrator.

Ok, I hacked in some debugging code to include the HTML body in the response, and it looks like the orchestrator is getting an error page with the message:

It seems that the orchestrator is getting an invalid response from the MediaWiki API:

When a page on a wiki is updated, MediaWiki sends purge requests to the CDN layer to invalidate objects in the cache. Currently, this is URL-based. So, for example, if I got edit the article on 'Science' on enwiki, MediaWiki will send purge requests to Varnish for the following URLs:

We're seeing errors again.

This is now rolled for text frontends.

This is now complete. Many thanks to @Vgutierrez for partnering with me to get this rolled out.

I tried setting EPP to 0 using x86_energy_perf_policy, thinking that bypassing the sysfs interface and writing directly to the MSR would make the setting sticky. Unfortunately this does not seem to be the case -- the EPP is gradually reset to 128, same as when you tried changing it via sysfs. At this point I also don't see value in further experimentation with the EPP knob and agree that performance is the way to go.

Actually, let me not step on your toes. But if you can tolerate a short extension of this task, I would very much like to see this setting tested. I think there is a good chance it will give the same or very similar performance increase with less waste of power. Just to be fully explicit, the setting is:

In T315398#8167048, @ori wrote:

So 'powersave' with EPP=0 gives a broader range of operating frequencies than 'performance'. We should see if in this mode the frequency scaling is still responsive enough for the workload.

Unfortunately continuation-local-storage and its more modern counterpart, AsyncLocalStorage come with a substantial performance cost, particularly for workloads with a lot of async/await calls. I don't think we can afford the performance penalty.

@Vlad.shapik thank you, but what about the other points I raised?

Congratulations, this is a huge win! I think we should dig deeper to see if we can get the same or similar performance benefit, but waste less power.

Thanks @roman-stolar. I think it's a mistake to combine (a) changes from (multiple?) upstream(s), (b) unmerged changes from Gerrit, and (c) your own work into a single commit, as in Icabc39dab. This destroys some useful history (for example, the authorship, review comments and discussion on Id6ec6d62c), and it makes future reconciliation with upstream code harder. It's also error-prone.

@Vlad.shapik, @roman-stolar, ping on the above :)
I'd also like to understand the deployment plan for this. Are you working with anyone on Wikimedia SRE to get this deployed? I strongly recommend deploying small, incremental updates rather than accumulating a lot of changes.

Based on https://www.kernel.org/doc/html/v5.6/admin-guide/pm/intel_pstate.html#operation-modes the scaling behavior will be different for systems depending on whether or not hardware-managed P-states (HWP) support is available and enabled. It looks like it is not available on 56 out of 265 app servers: P32411.