Mentioned in SAL (#wikimedia-cloud) [2024-01-02T10:42:51Z] <dcaro> flushed the redis db on tools-harbor-1 (T354176)

I have stopped harbor while the VM was starting:

root@tools-harbor-1:/srv/ops/harbor# docker-compose down

Then started up redis only:

root@tools-harbor-1:/srv/ops/harbor# docker-compose up -d redis
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating redis      ... done

And started a shell to connect to the db while it's loading the keys on startup:

root@tools-harbor-1:/srv/ops/harbor# docker exec -ti redis bash
redis [ ~ ]$ redis-cli
127.0.0.1:6379> info
...
# Keyspace
db0:keys=86724,expires=84749,avg_ttl=0
db1:keys=65,expires=65,avg_ttl=0
db2:keys=282164,expires=276869,avg_ttl=0

The db2 is the one getting out of hand:

# Keyspace                                                                                                                                                                                                                                                                                                                                                                                    
db0:keys=86724,expires=84749,avg_ttl=0                                                                                                                                                                                                                                                                                                                                                        
db1:keys=65,expires=65,avg_ttl=0                                                                                                                                                                                                                                                                                                                                                              
db2:keys=555330,expires=544869,avg_ttl=0

So I truncated all the dbs (we lose the cleanup runs logs):

127.0.0.1:6379> FLUSHALL async

And that freed a lot of memory:

root@tools-harbor-1:/srv/ops/harbor# free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       865Mi        13Gi       1.0Mi       1.6Gi        14Gi
Swap:             0B          0B          0B

We might want to reconsider T344433: [harbor] See if we can replace the per-project cleanup policy with a harbor-wide one

Some docs about memory management: https://redis.io/docs/management/optimization/memory-optimization/

If maxmemory is not set Redis will keep allocating memory as it sees fit and thus it can (gradually) eat up all your free memory. Therefore it is generally advisable to configure some limits. You may also want to set maxmemory-policy to noeviction (which is not the default value in some older versions of Redis).
It makes Redis return an out-of-memory error for write commands if and when it reaches the limit - which in turn may result in errors in the application but will not render the whole machine dead because of memory starvation.

Mentioned in SAL (#wikimedia-cloud) [2024-01-02T11:06:06Z] <dcaro> restart toolsdb database to flush connections (T354176)

Added a 1h log deletion policy to harbor:

Pasting a quick link to the free memory Grafana chart: https://grafana.wmcloud.org/goto/bBFK8SKIk?orgId=1

There's still a downward trend that is likely to result in OOM after a few days :/

Looking 90 days far, the rate at which the memory was being used was ~0.089 GB/day, currently is around 0.065 GB/day, this will take ~198 days to exhaust (a bit more than 3 months).

Let's wait a few days see if it stabilizes, but it should have done it already I think (as I set the cleanup to keep only 1h of logs).

In T354176#9432056, @dcaro wrote:

Looking 90 days far, the rate at which the memory was being used was ~0.089 GB/day, currently is around 0.065 GB/day, this will take ~198 days to exhaust (a bit more than 3 months).

Let's wait a few days see if it stabilizes, but it should have done it already I think (as I set the cleanup to keep only 1h of logs).

Wait, that makes no sense, the current rate is per-hour xd

that is, 0.065 GB/hour, that is 1.56GB/day, that makes it a bit more than 8 days to exhaust the current memory, we can wait a few days but yes, let's keep an eye.

There has been a substantial increase in the number of pulls on harbor side https://grafana.wmcloud.org/d/m9V1RQs4k/harbor-overview?orgId=1&from=now-30d&to=now

But I think it should not have been enough to see such increase of memory usage.

Memory usage has started flattening out \o/, crossing fingers:

Still going down slowly, but now it also goes up from time to time:

In T354176#9441291, @taavi wrote:

Is there a reason we could not configure the maximum memory limit for Redis?

We probably could, we have not because it would not improve much the situation and it requires complicating the configuration (even if just a bit, though would have to be investigated).

Today the minimum free memory hit 12.1 GB, seems to still be going down very slowly, will keep looking.

Is there a reason we could not configure the maximum memory limit for Redis?

According to the docs, it will lead to a Redis error instead of an OOM, which would likely cause Harbor to crash anyway. I'm not sure that type of Harbor crash would trigger an alert, while the OOM does, so that's one reason I'm not sure if setting the limit in the Redis config is a net gain.

Having a quick look at the graph (there was a hiccup on the last outage though), it seems that it takes ~6 days/GB, and there's 11.8GB left, let's revisit this in three weeks.

Still seems to be going down, but slower and slower:

This might be related to T356037: [harbor] cleanup execution + task tables , flushed redis again because of that one.

After the cleanup, the redis memory usage is stable with >14G of ram free... so closing this :)