Double checked across all nodes, this has been applied successfully. Resolving. Thanks!

show system1/log1 etc has 2 telling entries

I 'll lower priority for this. We may have the solution. https://grafana.wikimedia.org/d/000000607/cluster-overview?panelId=87&fullscreen&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=ores&var-instance=All&from=now-7d&to=now lacks the pattern on the 30th and the 1st, whereas it is clearly visible on all the days before that. I 'll leave it be for a couple more days though so we can monitor.

In T256629#6264055, @WDoranWMF wrote:

@akosiaris Thanks for putting this together. We're very excited to have SLOs around all our services. For the API Gateway work, Hugh, Giuseppe, Wolfgang and I discussed having a basic set of SLOs in place for Gateway and ideally for one of the services that back it under T254916.

The initial areas we were going to look at would be throughput, latency, error budget and availability. I know the dimensions we consider will vary depending on the component or service we're covering but are there general guidelines, process or methodology we could already be reviewing?

My bet is that when you shutdown the workers, the PyObject c structure of each python object is 'touched' forcing the 'copy part' of the COW behaviour.

Everything seems fine. I am gonna resolve this. @Pchelolo thanks again!

Both paths outlined in the description have been followed. We now have 4 different sessionstore dedicated nodes in each DC, with capacity of 6 kask pods each for a total of 24pods. That is 6 times larger than the capacity we had when the incident was triggered and 3 times larger than the capacity we now have allocated. So we have room to react in a sudden increase, hopefully in the future even automatically.

VMs created, installed and ready. Resolving

In T244530#6258712, @Dzahn wrote:

kubestagetcd1004 is still down. not sure if desired. ACKed in Icinga.

It seems like the restbase deploy of 821e96b fixed the issue. I 'll lower priority but leave this open for a few hours and if everything is fine, close as Resolved

@Jclark-ctr Excellent. I started the process of emptying ganeti1006 (and filling ganeti1005), that should take quite a while, but we should be on time for next Thursday. Many thanks!

Couple of more benefits of k8s I forgot to mention yesterday

@Jclark-ctr: ganeti1005 is ready. Fully depooled, downtimed and powered off.

@Pchelolo I think that yesterday's deploy of restbase has caused the issue described in this task. Restbase also alerts with

The start of the increases coincides with a deploy of restbase

codfw wikifeeds is by the way exhibiting similar behavior. https://grafana.wikimedia.org/d/35vIuGpZk/wikifeeds?panelId=12&fullscreen&orgId=1&from=now-24h&to=now&var-dc=codfw%20prometheus%2Fk8s&var-service=wikifeeds

Turnillo points out that these requests come disproportionately with this User Agent

In T256256#6252638, @JMeybohm wrote:

With kube-state-metrics (sorry for me repeating this over and over 😂 ) there is kube_pod_container_status_restarts_total and kube_pod_container_status_last_terminated_reason which can be used to detect OOM on containers.

An interesting thing to note here is that some services have quite often pod restarts. e.g.

In T255975#6247460, @JMeybohm wrote:

Thanks for writing this up @akosiaris! I think it would be nice to have the follow up tasks linked here. Like the removal of the service-runner and splitting up changeprop into multiple deployments (one per topic?).

There are no oresrdb nodes at all anymore. ORES redis has been moved to the redis::misc cluster. This is probably no longer relevant. I 'll remove the vm-requests tag, feel free to readd.

Currently, sessionstore sets a limit of 400Mi and 2.5 CPUs[1]. Memory wise, the nodes have 4GB RAM and 6 CPUs. The easy win here is to increase the amount of the vCPUs from 6 to 15. That should allow for a 200% increase in the amount of pods the node can run (from 2 to 6). The amount of memory available supports already 10 pods in theory, in practice since the node also consumes some memory it's more like ~8, which makes the 2 resources pretty close to each other.

@Jclark-ctr: OK, how about 1 host per week? no need for specific timeframes. I 'll have the host depooled, emptied, powered off and downtimed in icinga and ready for the memory upgrade. All you 'll need is to add the memory and power up.

With the last 2 changes merged, the work on this has been completed. Many thanks @apakhomov !

In T253843#6244473, @JMeybohm wrote:

I need to make decisions regarding TLS and storage:

Do we want to use envoy here (ChartMuseum is able to TLS termination as well)? I think it might be desirable as it prevents us from writing/maintaining TLS stuff in the ChartMuseum class/profile and the overhead is probably irrelevant here.

In T255975#6244382, @hnowlan wrote:

This looks great, thanks @akosiaris! The only thing I'd note (not sure if it even warrants inclusion in the body of the report but it's an interesting correlation) is how closely CPU throttling was linked with memory maxing out in the container. The troughing seen is of course the pod being OOM killed but the peaks correspond quite closely with memory usage peaking.

@JMeybohm @Pchelolo @hnowlan: This is an account of the investigation we went through for the changeprop memory/cpu limiting.
It's part of the sessionstore incident[1] actionables, although it's after all largely unrelated. Please proof read and correct any errors I might have made.

In T254556#6243442, @Marostegui wrote:

For what is worth this is probably going to be scheduled for next quarter (so July-September).

In T105378#6233389, @tstarling wrote:

The log messages represent failure of all servers in the pool, although the error is not user-visible. PoolCounter_ConnectionManager::open() fails silently after two attempts to connect to a server. Once the loop over all hosts in PoolCounter_ConnectionManager::get() fails to get a valid connection, the last error is wrapped in a Status and returned to PoolCounterWork::execute(), which logs the error. It's those log messages that we see. The impact is that PoolCounterWork::execute() runs the work locally, as if the pool population was zero -- a fair assumption unless the cause of the connection error is a very heavy system-wide overload.

In T251626#6231628, @Papaul wrote:

@akosiaris the racking/setup ticket needs to be closed and open another ticket for the service.

Thanks

Great! Thanks Papaul

Service owner steps done, DC ops work can begin.

Service ops owner steps done, machines are ready to be handled by dc ops.

In T255179#6228123, @RoySmith wrote:

As an external observer, I'm fearful of "log everyone out". This will cause a spike in traffic to the authentication infrastructure as everybody logs back in again. Which, of course, was the root cause of this problem to begin with. Is there some way to do a rolling logout, killing say, 1% of the sessions per hour?

In T105378#6230593, @tstarling wrote:

The count over 24 hours was 1689 connection timeout errors. I think that crosses my threshold of concern. Another tweak may be needed.

In T255179#6218789, @Tgr wrote:

Should something be done about users who thought they have logged out but actually didn't?

In T224041#6225405, @jeena wrote:

I created a helm test and got the integration and functional tests running in minikube. Do we have DNS within the ci cluster?

In T255410#6224239, @Michael wrote:

In T185664#6221695, @DannyS712 wrote:

Is there a default result for an extension if no team agrees to be responsible for it? Undeployment?

kubernetes2007 has been reimage successfully, it seems like kubernetes2008 to kubernetes2014 require networking configuration on the switch side.

Yes, absolutely agreed. The trigger was indeed insufficient capacity in sessionstore to handle a, at least, 33% (~15k to ~20k if not more) sudden increase in requests. We 've gone ahead and added capacity to the service and will follow up with adding more capacity to the entire cluster as well as the dedicated sessionstore nodes. So, I 'll be bold and resolve this, feel free to reopen though.