Here you can find the report for the new per-attribute stripping benchmarks
https://public.paws.wmcloud.org/User:JGiannelos_(WMF)/Parsoid%20benchmarks/Read%20views/Per%20attribute%20stripping/processingPerAttr.ipynb

Something like that can also work as a selector just to simplify things:

.PieChartTemplate div.notheme

Just a quick correction on the numbers: the current production container size is ~40M objects not ~12M (i was counting the wrong container).

Next steps to failover to codfw:

Hey @TheDJ i have some experience with using Swift from a dev perspective but for the actual swift config I would defer to the data persistence team.

Hey @fgiunchedi the size of the current active deployment is stabilized at ~12269804 objects for quite some time now (last 1 month). The theoretical upper limit can be way higher if we assume that all planet tiles are pregenerated and the count could be as high as 1431655765 (all tiles from zoom level 0 to zoom level 15). The first estimate is more realistic, we never end up generating all planet tiles.

@cooltey Just a clarification regarding the API.
When calling footer=true on the API endpoint the output is a demo page with a hardcoded value, but not what the apps consume. I think its worth fixing it too to display the right last edit timestamp but for the actual scope of the problem on the apps level what needs to be fixed is the way we render the last edit timestamp to string for the specific locale.

For now we can use the testcases for each template so no need to identify which template is used per test run

I managed to reproduce this locally. The problem is that parsoid falls back to wikitext for all content models other than javascript.
A quick solution (other than implementing a content handler on parsoid) could be to fallback to MW parser output for the supported content models.

Added parsoid since mobileapps output is the same with the parsoid output

Here is the report for the readability issues after testing all template testcase pages for en.wikipedia.org:

From a quick look on parsoid and mw:
https://ar.wikipedia.org/api/rest_v1/page/html/%D9%83%D9%84%D9%85%D8%A7%D8%AA_%D9%85%D8%AA%D9%82%D8%A7%D8%B7%D8%B9%D8%A9

Here are the results based on the last rt-testing run (as HTML export):

@vadim-kovalenko is there anything left to be done for this ticket or should we close it ?

This is already deployed in production

nit: I think there is a typo on the category, "infrastructure" instead of "infastructure"

I dont see any slow queries on the replicas after enabling the postgres config also kartotherian level errors are back to the regular rates:
https://logstash.wikimedia.org/goto/933d704ba3f70f30b6999dbc71fa6769

I think we should consider it done since we have both Vadims implementation for readability and pa11y which implements the w3c checks for a11y.

I think that given that we need to computed values for CSS from a browser i dont think that linter would be a good place to implement something at the moment. Going ahead and closing the ticket for now.

This view is very useful for debugging on the maps end:

Also maps1006 got killed by OOM at the same time.

2022-05-05 03:24:10 GMT LOG:  server process (PID 42352) was terminated by signal 9: Killed

Maybe its worth depooling maps1007 temporarily if its the only node that had issues to see if we still get errors.

Maybe the current issue with geoshapes is because of a corrupted index? Still TBD based on the output of slow query logs but in the past reindexing worked.

From postgres logs around the time errors started on maps1007:

2022-05-05 03:24:04 GMT LOG:  server process (PID 24334) was terminated by signal 9: Killed

At the moment:

• DB connections look healthy
• The load/IO issue on maps nodes is resolved
• Tiles DB query latency went back down
• Tiles request rate is is back to our regular traffic
• No errors related to map tiles/snapshots

It looks like manually GETing the tile fixed the DB query latency:

Lets see if this fixes things. One potential action item could be to fine tune the envoy timeout threshold.

Another thing that looks a bit off is that in the top 50 request on the error logs is that this tile show up repeatedly:
GET https://maps.wikimedia.org/osm-intl/6/33/21.png

The increase in HTTP traffic makes sense because of the way geoshapes retries the HTTP request on failure.