This appears to have been an issue with a previous version of librsvg, I've purged the caches for each of the affected images in this case and they now render correctly. If you notice any other similarly incorrect images, purging the cache will hopefully address it.

kubernetes* and mw* are ready

Confirming that this is an issue, seeing the same behaviour with a clean cache and locally with rsvg-convert. Either an issue with the file or a new issue with rsvg-convert.

Are there plans on when and how we need to move this forward to production?

It seems a bad frontend server was the source of these errors, and a rolling restart appears to have addressed this but we'll follow up during the week to see if there are any obvious causes.

The thumbor-side issues were a side-effect of the upgrade that our tests didn't catch due to differences in config between prod and test - I see the supplied image rendering now (I've purged the cache for that image to be sure). However, I am still seeing the Undefined error, which I suspect is either a long-tail side-effect of the Thumbor issue, or something unrelated.

I'm sure there'll be some tweaks further down the road, but this deployment has been created. Tracking further work in T356241

In T368405#9925476, @kostajh wrote:

In T368405#9925400, @Michael wrote:

In a different conversation, @kostajh pointed out that the requests to the AnalyticsQueryService, added in T235810 to show the site edits in the last day, might be a contributing factor. They are in fact still in use as a fallback in \GrowthExperiments\HomepageModules\SuggestedEdits::getMobileSummaryBody. And that else-branch also logs a warning if we do not get something useful back from the AnalyticsQueryService. And that warning is seeing a suspicious uptick as well:

So I think a good next place to look would be to ask AQS folks if anything there changed on June 18.

I'm sure the developers would be best positioned to say whether anything has changed, but as far as the AQS services themselves are concerned it doesn't seem like there have been any significant increases in latency: service-level view, REST gateway level view (easiest to read if you filter out the proton metrics)

In T364799#9924888, @SGupta-WMF wrote:

@VirginiaPoundstone I am not aware of any rate limits on AQS 2.0 service , tagging @hnowlan for confirmation.

Log level reduced to stop the bleeding:

I believe the straggling traffic here is a misnomer/a graph misunderstanding - the API gateway's envoy config refers to traffic to the mediawiki API as "mwapi_cluster" internally before and after the hostname was changed. I believe these requests are normal and are being routed to k8s already - drop mwapi_cluster from the graph and we're at zero! 🎉

In T361835#9712223, @SGupta-WMF wrote:

@WDoranWMF Yep , it makes sense . I confirmed with @mforns that API paths and we agreed on metrics/commons-analytics . Regarding the prefix , all the AQS services use /api/rest_v1/metrics , we could use that from consistency perspective . But still will like @VirginiaPoundstone to confirm .

I don't think this will be needed.

This change now uses librsvg's accept-language flag to obey`lang`.

In T295007#9829689, @Lucas_Werkmeister_WMDE wrote:

I’m not aware of any issues, but if this is working (great!), then we should probably enable it by default in MediaWiki, and everywhere in production? Right now AFAICT $wgEnableAsyncUploadsByURL still defaults to false in core and is only enabled in commonswiki, testwiki and the Beta Cluster in wmf-config.

Thin lines are now rendering in the test cases given

Considering this resolved as part of T355020. Please reopen if that's incorrect

In T246001#9874889, @hnowlan wrote:

Not seeing this behaviour as fixed onwiki in librsvg 2.54.7 in place

In T97233#9868270, @hnowlan wrote:

In T97233#9868186, @Colinstu wrote:

In T97233#9868140, @hnowlan wrote:

In T97233#9868011, @Colinstu wrote:

For existing images that have rendering issues, how does one kick-off rerendering existing images that had issues after this upgrade?

https://en.wikipedia.org/wiki/Help:Purge#Purge_request_to_server - tldr add ?action=purge when viewing the File:myfile.svg page.

Be sure to force refresh your browser after doing it to be sure changes have rendered.

Thank you. Gave that a shot and sadly do not see an improvement in the example I previously shared. (yes cleared browser cache and even tried another browser that hasn't previously pulled this page up before).
(Also does Wikipedia have a CDN? perhaps that's also caching it somehow/somewhere?)

I had actually already run a purge for your image, which improved some but not all of the text alignment. This must be another issue outstanding in our current version of librsvg, but I don't know enough to point to which specific issue unfortunately. In coming weeks we'll be upgrading to 2.54.7 which will offer further improvements.

In T246014#9871752, @Aklapper wrote:

@hnowlan: But that task is resolved already (please only write T336881 instead of full URLs to get corresponding link rendering) so why would this one remain open? Not sure I can follow.

Not seeing this behaviour as fixed onwiki in librsvg 2.54.7 in place

Is this still an issue? As far as I can tell we have all available Bengali fonts installed that are currently available in Debian (along with the noto fonts from T184664) but SiyamRupali is not one of them. If there's an example of a broken SVG it'd be helpful to debug this further

The generated image post-purge/post-upgrade appears to render correctly for me, fixed in the last upgrade?

This will be done with T355020

I agree this isn't necessarily a software issue. That said, since upgrades the 180px version of this image is no longer cut off.

https://gerrit.wikimedia.org/r/1039778 is now ready for review. In future we might need to revisit how we do our SSIM comparisons as regards reference thumbnails. As our tools change, the distances etc are diverging and there will come a point where tweaking the test values will become a problem.

Appears resolved.

Appears resolved by librsvg upgrade

Resolving for now, following up in related issues.

This appears fixed

Appears fixed

Has this issue been fixed by the upgrade and the improved text-anchor behaviours?

In T97233#9868186, @Colinstu wrote:

In T97233#9868140, @hnowlan wrote:

In T97233#9868011, @Colinstu wrote:

For existing images that have rendering issues, how does one kick-off rerendering existing images that had issues after this upgrade?

https://en.wikipedia.org/wiki/Help:Purge#Purge_request_to_server - tldr add ?action=purge when viewing the File:myfile.svg page.

Be sure to force refresh your browser after doing it to be sure changes have rendered.

Thank you. Gave that a shot and sadly do not see an improvement in the example I previously shared. (yes cleared browser cache and even tried another browser that hasn't previously pulled this page up before).
(Also does Wikipedia have a CDN? perhaps that's also caching it somehow/somewhere?)

I believe after purging SVGs are now obeying the font list. Not closing until this is confirmed

This appears to be resolved.

This specific issue appears resolved - I see some other unresolved issues in SVG_Test_TextAlign.svg but not pertinent to this issue.

In T97233#9868011, @Colinstu wrote:

For existing images that have rendering issues, how does one kick-off rerendering existing images that had issues after this upgrade?

This appears resolved by T265549

This appears to be fixed by T265549

Solved as of T265549

I believe this is solved by T265549

Since upgrading to 2.50.3 and doing some purges, I am seeing *some* improvement in the errors in this ticket but not in all cases.

We are using Thumbor on bullseye everywhere which means that SVGs will be rendered by 2.50.3. Keeping this task open for tracking issues for the moment.

In T364921#9866023, @Scott_French wrote:

The last log line is:

{"@timestamp":"2024-06-05T22:49:28Z","message":"Connecting to Cassandra database: aqs2001-a.codfw.wmnet,aqs2001-b.codfw.wmnet,aqs2002-a.codfw.wmnet,aqs2002-b.codfw.wmnet,aqs2003-a.codfw.wmnet,aqs2003-b.codfw.wmnet,aqs2004-a.codfw.wmnet,aqs2004-b.codfw.wmnet,aqs2005-a.codfw.wmnet,aqs2005-b.codfw.wmnet,aqs2006-a.codfw.wmnet,aqs2006-b.codfw.wmnet,aqs2007-a.codfw.wmnet,aqs2007-b.codfw.wmnet,aqs2008-a.codfw.wmnet,aqs2008-b.codfw.wmnet,aqs2009-a.codfw.wmnet,aqs2009-b.codfw.wmnet,aqs2010-a.codfw.wmnet,aqs2010-b.codfw.wmnet,aqs2011-a.codfw.wmnet,aqs2011-b.codfw.wmnet,aqs2012-a.codfw.wmnet,aqs2012-b.codfw.wmnet (port 9042)","log":{"level":"INFO"},"service":{"name":"data-gateway"}}

This suggests cassandra client session init is hanging for some reason.

I am now seeing results when using queries with urlencoded characters. Unfortunately we will need to add a manual hack if there are other non-alphanumeric chars in other parts of the URL in future, but for now I think this works:

It seems Envoy only normalises a subset of urlencoded characters:

hnowlan@plunkett ~/Code/deployment-charts (hnowlan/T365439-apigw_normalise_path_urls *) $ curl -s localhost:8087/core/v1/wikisource/a/%3A| grep original-path
    "x-envoy-original-path": "/core/v1/wikisource/a/%3A"
hnowlan@plunkett ~/Code/deployment-charts (hnowlan/T365439-apigw_normalise_path_urls *) $ curl -s localhost:8087/core/v1/wikisource/a/%31| grep original-path
    "x-envoy-original-path": "/core/v1/wikisource/a/1"

The normalisation change has unfortunately not fixed this issue - docs indicate that it should have but I suspect this is something to do with the use of regex matching as opposed to static matching. I'll try to come up with a workaround for the short term

certs updated in all DCs, alerts resolved. I sincerely hope we will have the mesh migration resolved so we can avoid having to update echostore's certificates in October, but in case something prevents that and for reference the process was:

• puppet cert revoke sessionstore.discovery.wmnet
• In the puppet repo on your local checkout ./utils/create_ecdsa_cert sessionstore.discovery.wmnet sessionstore.svc.eqiad.wmnet sessionstore.svc.codfw.wmnet
• On the puppetmaster, put the contents of /var/lib/puppet/server/ssl/ca/signed/sessionstore.discovery.wmnet.pem into certs.kask.cert in helmfile.d
• Add the contents of the new private key from ./modules/secret/secrets/ssl/sessionstore.discovery.wmnet.key to hieradata/role/common/deployment_server/kubernetes.yaml
• Validate the files and make sure everything looks okay using openssl ec/openssl x509, then git commit your changes in private
• Follow the Helm rollout process as normal, keeping an eye on the sessionstore graphs and the session loss graphs

Current situation - I have refreshed the .key file on the puppet master using a modified version of the create_ecdsa_cert script, and I have pushed the new key to the staging k8s secrets for sessionstore only. I've also updated the cert file in the helmfile configuration for sessionstore, but it hasn't picked it up because that part of the config wasn't checksummed to recreate the pods (fixed in this change). Tomorrow I will try to roll to codfw and, if successful, eqiad.

In T363996#9821312, @Eevans wrote:

In T363996#9816587, @elukey wrote:

In T363996#9794048, @hnowlan wrote:

Steps that I see:

• Renewing the existing cergen cert to give us breathing room just in case. We're looking at less than 2 weeks of headroom for a major change to one of our most critical services
• Add toggleable mesh support
• Get a baseline for performance in staging by running siege for a few hours
• Deploy mesh-enabled version to staging, compare performance before and after
• Roll forward if everything looks okay

Only concern around testing in staging is having reasonably representative infrastructure, given that it's both using cassandra-dev, and in codfw from pods in eqiad. That said, we're just looking for a baseline and not identical numbers.

+1! Not sure how to renew the existing cert since IIUC cergen wasn't used, but we can try to check in old tasks to see what it was done.

I'm out next week, and the cert is about to expire, so I'll try to reverse engineer this. If anyone has first-hand knowledge of how this was done, please let me know!

All codfw wikikube-ctrl nodes are operational

I suspect the fix for this is a relatively small change on the API gateway, but the change is a global one so I will need to take some time to test this, even if the impact is to make things standards-compliant. Hoping to get to it later this week