I have enabled TLS on staging for now, which will use some default certs

Unless @JMeybohm was not the culprit, we can mark this as resolved

Ι think there is a correlation between the latency spikes and TCP retransmits

In T271967#7112446, @jbond wrote:

i wonder if we have considered just having the TLS port every where accept localhost?

regardless i guess we need a transition config but still curious if this if that is the long trm goal or not?

No!

Continued in T282148

@akosiaris @Ottomata should we resolve this in favour of T282148 ?

@dancy it appears that rolling out scap coincided with the greek easter this time , sorry!

@dancy thank you very much, everything worked well!
I have installed scap 3.17.1-1 on our api canaries, I will proceed with a full rollout this week.

In T279100#7019419, @akosiaris wrote:

So, the crux of the issue is at those 2 functions below

def pool(self, pooled):
    """Pool a list of services"""
    try:
        for obj in pooled:
            obj.update({"pooled": "yes"})
        self._verify_status(True)
        return True 
    except (BackendError, PoolStatusError) as e:
        logger.error("Error depooling the servers: %s", e)
        return False

def _verify_status(self, want_pooled):
    if want_pooled:
        desired_status = "enabled/up/pooled"
    else:
        desired_status = "disabled/*/not pooled"
    for baseurl in self.lvs_uris:
        parsed = parse.urlparse(baseurl)
        url = "{baseurl}/{fqdn}".format(baseurl=baseurl, fqdn=self.fqdn)
        # This will raise a PoolStatusError
        logger.debug("Now verifying %s", url) 
        self._fetch_retry(url, want_pooled, parsed, desired_status)

The pool() function will get a list of services to pool, in the specific case the jeena raised, it will only pool the service that was pooled already before the script started (that is the jobrunner service) and leave the videoscaler service depooled (correctly). However it calls the _verify_status() function that has no such knowledge of previous state and relies only on the lvs uris passed on the command line.

Let me see if I can fix that easily.

It has been deployed to production, I forgot to close that task, sorry!

The reasons have been identified in https://phabricator.wikimedia.org/T253673#6569013

@LarsWirzenius as discussed in T265501, using docker to build scap, is not an option. In T277793#6957246 I wrote that "we will be happy with a step by step documented process of how to build the debian package on our build host". Please update the "Building" section in https://wikitech.wikimedia.org/wiki/Scap3#Production_Upgrade, and I will be happy to build and roll it out.

In T280582#7015884, @Krinkle wrote:

Note that deployment of T113916 was halted because the Redis capacity was actually considered too small. (That task is about migrating module dep store from a MW core DB table written to during GET requests, to instead use Main Stash)

I do believe however that after MainStash DB (x2, T113916) is fully online and configured as backend for MainStash in MW, that indeed this cluster can be downsized, even much further to only 3 or 4 servers perhaps? ChronologyProtector would likely be its only consumer, perhaps with one or two other consumers besides it that store small and short-lived dc-local values at an every lower frequency.

In T279100#6997273, @akosiaris wrote:

I 've gone a bit overboard and created https://gerrit.wikimedia.org/r/679258 that uses YAML map merges to implement the patches above also in the structures. That will probably help avoid some unfortunate typo or mistake where the actual state in conftool differs from what we 've documented. What do you think?