As written I don't think this task is actionable. "Fix all of the code so it stops logging errors" is not within the scope of the tiny handful of folks who look at infrastructure things in deployment-prep.

Yeah, that's why I was being vague and said "ideally". I don't think it needs to be a "fix everything" either, it's fine if there are some errors. It's more about the current state having occasional traces of useful data in an ocean of noise. I'm also curious to see what happens once the current issues with deployment-jobrunner05 are resolved, as some of these errors seem related.

In T389239#10648814, @Daimona wrote:

I'm also curious to see what happens once the current issues with deployment-jobrunner05 are resolved, as some of these errors seem related.

Indeed, many of those errors are gone. As for what remains:

• cxserver log messages don't use normalized_message, so they're a bit harder to filter. Filed T389273.
• CirrusSearch poolcounter stuff: can also be seen in production at a rate of ~10k errors/hour. These caused an UBN in production in 2023: T352444. Taking a closer look though, the log level is INFO, so despite the word "error" in the message, these might be harmless?!
• "Exec error in changeprop": T389252 tracks errors due to a missing parsoid service. I filed T389274 for those originating from wikifunctions.

And that should be everything for the >1000/h category.

The biggest normalized bucket right now is "Pool key 'CirrusSearch-Search:_elasticsearch' (CirrusSearch-Search): ⧼poolcounter-connection-error⧽" which is pretty obviously related to T380881: Re-create poolcounter instance in Beta Cluster (deployment-prep).

The next highest are "Failed to write logs to file" and "worker died, restarting" from CXServer.

ChangeProp is spamming "Commit failed" messages.