@aaron Please have a look at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569541/ and let us know if it reflects what we are going for, or if there is some nitpicking to be done. We can afterwards merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/574200/ to test routes for the /*/wm-wan/ keys.

In T240684#5925272, @aaron wrote:

Yeah, the eviction ("tko") delay should be low to avoid prolonging DB traffic spikes. Gutter tombstoning and FailoverWithExptimeRoute helps mitigate flapping from low TKO trigger thresholds.

done, sigh

This is stalled until we completely stop using redis and we have put the gutter pool in production

In T217924#5926912, @LarsWirzenius wrote:

I had a discussion with Tyler just now. We plan the following:

• add to docs/ in the source tree; this ends up in doc.wikimedia.org
• add to debian/changelog in the source tree
• update the wikitech Scap page
• notify ops list
• notify wikitech-l list
• have it mentioned in Scrum of Scrums

Great we merged this patch! Do we have a plan of how we will communicate this to the deployers when we release scap as well as how to test that it is all good in production? Thank you!

Thank you @herron and @fgiunchedi for your help!

In T245841#5919699, @Joe wrote:

What would having all scap proxies also be mcrouter proxies change in terms of the scenario you described above?

Hey Will,

(Reopening to discuss this a bit more)

@aaron @Krinkle Please let us know if the configuration and the information on this task are enough to proceed with testing the gutter proxies for /*/wm-wan/ or if there is anything that needs further investigation/explanation.

After a lot of fiddling with @herron, we are finally at this https://phabricator.wikimedia.org/P10513 !

If there are no objections, I would like to proceed with this

Let me know if we come across any issues:)

With a little bit more fiddling, I managed to run puppet on ssh deployment-mediawiki-09.deployment-prep.eqiad.wmflabs! @herron does that unblock you?

I have uploaded a patch which I manually tried on beta, this seems to work, but sadly, puppet breaks a bit further down the road

Package has been build and pushed to wikimedia repo, I'll roll it to production on Monday

@herron any ideas how to proceed here? Is there someone who can help? Apparently this patch could potentially break beta.

It appears that on beta the variable $server_role = $::_role.split('/')[-1 is not evaluated properly, while in production, it looks just fine https://puppet-compiler.wmflabs.org/compiler1001/20912/mwdebug1001.eqiad.wmnet/, I believe this has something to do with this server's role in beta?

To proceed with testing, we will puppetise the following configuration, and roll it to a couple of canary servers, and block traffic towards mc* like we did earlier. Our goal is to test memcached 1.5.x on debian Buster, since that is what the gutter pool is running

In T240684#5893489, @elukey wrote:

Very nice summary, thanks!
A couple of questions:

FailoverWithExptimeRoute, where we define how long the values keys updated (eg set/add) during the failover will live on the gutter. All keys set during a failover with the following config, had a TTL of 10s.

What happens if we try to set a key with TTL of say 5s? Will it get TTL 10s or 5s? I would hope for 5s but I suspect it might not be the case. I am asking since setting a high enough TTL like 10/20/30 mins could be good if keys set with lower TTLs are not affected.

Weights of mw1349-mw1355 were switched to 30

Test if failover works and when to failover

Thank you all!

Oh great, thanks! From https://logstash-beta.wmflabs.org/, it looks that logs are flowing, so maybe we can try the change there after all

thank you daniel!

@herron I fiddled a bit on beta, it appears that for some reason, nothing is being streamed there since today, I am not sure if I broke it myself while I was trying to make it work :/ FWIW, I have restored the original config on deployment-logstash03

I have uploaded a patch that could possibly work, my issue generally is that I can't find a sane and safe way to test if those logstash filters will do what we need. @herron any ideas are welcome

In T227734#5879320, @Tgr wrote:

• getid3 (a TimedMediaHandler dependency I think?) uses it
• BitmapHandler in core uses it, but probably only as a fallback
• the IPTC parser in core uses it, which AIUI might be triggered for certain JPEG files

@RobH, those hosts are delivered and in production per T241795

@elukey thank you for unblocking this !!!

@CGlenn it appears you are already in the ldap wmf group as 'keepit-ssh', which is the username (shell name) in wikitech as well. Can you check please if your access is in order?

will fix :)

On mwdebug1001 we have deployed a config similar to T240684#5826966, where we failover in case of a TKO to the gutter pool servers (mc-gp*). We again have set --timeouts-until-tko=3.

@kzimmerman oh my bad, sorry!

done

@CGlenn done:)