Change 542183 had a related patch set uploaded (by Krinkle; owner: Jforrester):
[operations/mediawiki-config@master] Drop nutcracker indirection for HHVM servers, just point to localhost

https://gerrit.wikimedia.org/r/542183

Change 542183 merged by jenkins-bot:
[operations/mediawiki-config@master] Drop nutcracker indirection for HHVM servers, just point to localhost

https://gerrit.wikimedia.org/r/542183

What's the ask for CPT here?

As far as I remember, the php memcached extension didn't allow using a unix socket to connect to memcached. According to https://www.php.net/manual/en/memcached.addserver.php this is possible since version 2.0 of the memcached extension. We're using 3.0.1 in production now, and we might think of moving mcrouter to listen on a unix socket locally as well, but it would need some testing.

(Also, we don't use nutcracker anymore for managing memcached connections at all in production anymore, as far as I remember).

@Joe @Krinkle What are the next steps for this? Is it blocking other work?

In T235216#5577250, @daniel wrote:

What's the ask for CPT here?

I thought some of this abstraction was in core and would've asked whether there are downsides/benefits to using sockets vs ports from what that abstraction gives us. However I see now that this abstraction goes all the way into php-memcached so from MW it is handled transparently.

The ask that remains then is I suppose better aimed at ServiceOps, which is:

In terms of system performance and other trade offs at the server level, would there be a benefit to keep doing what we did on HHVM (use sockets). Or should we continue with the php5 approach that has (unintentionally) become our php7 mode now?

In T235216#5600189, @WDoranWMF wrote:

What are the next steps for this? Is it blocking other work?

No next steps for CPT (yet). I'll re-tag if that changes.

In terms of blocking, it's not immediately critical. I've observed an unintentional change in how we connect to Memcached as side effect from HHVM>PHP7 migration. I'd like to make sure we will not be caught by a surprise down the line if this has some mid-long term impact (which might take a while to surface).

And even if it has no negative impact, we would want to determine if this is a performance opportunity. If so, it would be an easy win perf team could drive.

@Krinkle Thanks for taking the time to update, we'll retriage if it ends up needing us and gets retagged.

I'm pretty sure using unix sockets would improve performance, it did for sure when we were on HHVM. It's pretty easy to test this effect as we now collect timing data per-server, so we could change the configuration conditionally on one api and one appserver, and check if there is a positive performance impact pretty quickly.

Mcrouter can't be configured to listen both on a unix socket and on a TCP port. This means, apart from how cumbersome the change is going to be if we want to do it, that we'd need to change our architecture and have separate "mcrouter proxies" for cross-DC replication.

Given that's a lot of work and a change in architecture, we won't do it without a proven gain of functionality. We'll experiment a bit with using unix sockets, and see what performance impact, if any, that grants us. Unless the gain is enormous, we don't think it would be worth the hassle.

I'm confused. If such change requires the kind of migration you describe, then what did we do during the PHP 7 migration this year?

In T235216#5729967, @Krinkle wrote:

I'm confused. If such change requires the kind of migration you describe, then what did we do during the PHP 7 migration this year?

We did nothing. Mcrouter was always accessed via TCP from HHVM as well.

Update from a chat with @jijiki: This isn't easy to do right now, unless mcrouter can listen to a socket and TCP port at the same time (which maybe it can, but is additional complexity we may want to avoid). The reason we'd need both is that we do our cross-dc WAN cache purges by talking to another mcrouter between the local mcrouter and the remote Memc cluster.

MW → local mcrouter → local Memc cluster
MW → local mcrouter → mcrouter purge proxy → remote Memc cluster

However, switching to a unix socket for Mcrouter should be relatively easy once we have phased out purge proxies in favour of talking directly cross-dc for Memc purges. This work is tracked at T271967, and is unblocked by being able to use TLS there.