This is spammy spammy spammy:

for rbf1001 we've had 44200 error logs in the past 15 minutes
rbf1002 (I presume, it's 10.64.0.163) is less but still ~60 in the last 15 minutes

gave aaron a call and left a vm

called ori who knew understood the workload and he said he was minutes away from the office and will get into it

thanks @ori

for posterity

fluorine:~# grep '10.64.0.162: Connection timed out'  /a/mw-log/hhvm.log | wc -l
2241856

I disabled puppet with a reference to this issue, turned off AOF and restarted REDIS. It seems to have helped in one case and not much in the other.

Failures in the last 10K

root@fluorine:~# tail -n 10000 /a/mw-log/hhvm.log | grep 'Connection timed out' | grep 0.162 | wc -l
9968
root@fluorine:~# tail -n 10000 /a/mw-log/hhvm.log | grep 'Connection timed out' | grep 0.163 | wc -l
19

It seems teh master/slave status is defunct, and if I understand correctly @aaron thinks:

that would make some sense, the code would get stuck in an update loop on 1/2 the filter checks

I'm not sure this could have ever been working as the same redis role is applied in both cases, so they both are set as ignorant masters?

node /^rbf100[1-2]\.eqiad\.wmnet/ {
    role db::redis
    include admin

    # Background save may fail under low memory condition unless
    # vm.overcommit_memory is 1.
    sysctl::parameters { 'vm.overcommit_memory':
        values => { 'vm.overcommit_memory' => 1, },
    }
}

@aaron seems to have turned this off for now. No connected clients and 0 error logs. Good odds the lack of master/slave status combined with AOF on the "slave" will clear this up, we'll need to coordinate further.

I was told aaron went to lunch so I guess this is on hold. This needs an incident report per @greg and should not be re-enabled until an ops person with access is around to coordinate to monitor and debug.

side note, I think the replication config was moved to hiera and maybe this just isn't working?

https://phabricator.wikimedia.org/rOPUP457d58535e9b3e49e0eb7a91c42b76316f84c44f

The currently enforced slave config from puppet for rbf1002:

-slaveof 10.64.0.162 6379
+# slaveof <masterip> <masterport>

@Joe ideas?

@chasemp what exactly doesn't work in the hiera change? I see the config on rbf1002 server seems to be correct at the moment.

In T92591#1121388, @Joe wrote:

@chasemp what exactly doesn't work in the hiera change? I see the config on rbf1002 server seems to be correct at the moment.

Caught up via IRC but for posterity the change in question was hard coded from Friday testing.

relevant https://phabricator.wikimedia.org/T90923

I see appendonly is manually disabled on the host and puppet is disabled again. I am pretty sure I fixed the replication glitch in puppet... @chasemp can you update the status of this?

In T92591#1152373, @Joe wrote:

I see appendonly is manually disabled on the host and puppet is disabled again. I am pretty sure I fixed the replication glitch in puppet... @chasemp can you update the status of this?

I think it was never enabled and run post your fixes. I'm not sure what @aaron wants to do here, I had let it sit as I it was my understanding he is moving away from the master/slave model entirely? Really not sure, but this is assigned to him so I'll let him chime in.

In the mean time, I enabled and ran puppet and things seem normal.

Two thoughts about rbf1002 changes from puppet:

This will break if DNS ever has issues (previously I have used IP's for redis replication), just a note:

-slaveof 10.64.0.162 6379
+# slaveof <masterip> <masterport>
+slaveof rbf1001 6379

If this is going to be a slave it seems wasteful to do persistence on it, and the value of it should probably be looked at for rdf1001 also IMO:

-appendonly no
+appendonly yes