While the underlying pattern of "slowly increasing" still holds, the number of timeout events is now an order of magnitude lower than what is shown in the original task description. Also, given the current switch in datacenters, the production timeout events are currently happening on codfw instead of eqiad, except for the timeouts caused by the health check, which is still running in both datacenters.

Updated dashboard covering timeouts from both datacenters: https://logstash-next.wikimedia.org/goto/c303115e5870375b65bce1ba3f9244d8

Looking deeper into this data, a strange pattern was recognized. The events don't seem to be randomly distributed as one would expect, but there seem to be some patterns. In particular, it is notable that a lot of events are happening just in the minute after 22:00 UTC. This seems to be a recurring pattern on many, but not all, days.

It is unclear what causes this.

• Cache purges shouldn't happen based on the time of day
• Maybe some scripts are requesting a lot of data at 22:00 UTC which leads slower responses?
• Are we doing some daily dumps or something at 22:00 UTC?

In particular, it is notable that a lot of events are happening just in the minute after 22:00 UTC

Could be related to T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes which is currently happening at 22:00UTC
This should go away with T264881: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC being fixed
While investigating those tickets I looked a lot into other things happening around that time and at a bunch of metrics and didn't see anything else.
s8 load in general would make sense to affect termbox performance

@Michael do you see any further concrete actionables here, or things to be added to https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/SSR_Service ?

In T255450#6770588, @Addshore wrote:

@Michael do you see any further concrete actionables here, or things to be added to https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/SSR_Service ?

Not right now, I think it can be closed. I guess we'll have to see how good our documentation holds up when things actually start breaking and we need it in order to make sense of things. We can then use the learnings from that to improve it further.

Optionally, we could ask our technical writer to quickly glance over it. Maybe there is something like "typical errors that developers make when writing technical documentation"?