Thanks @Arnoldokoth for enabling the cleanup job again. The inode metrics look much better now and stabilized at around 2.5% usage. Also thanks for the new alert, which should warn us before we see the no space left on device errors.

There is an open alert since 11 days Linting problems found for GitlabPackagePullerFailedOnRun:

I don't think a check will help. We can either

The Inode usage grew from 2% to 10% already in the past day: https://grafana.wikimedia.org/d/000000371/vrts?orgId=1&from=now-2d&to=now&timezone=utc&var-node=vrts1003&viewPanel=panel-30. So I like the idea of enabling the cleanup job again.

In T406762#11409848, @hashar wrote:

This task was for gerrit2003 which had a 55GB backup because, as part of a migration, the git repositories have been entirely rewritten. That specific cause has been resolved for now (but will trigger again next time the repos are wiped and regenerated via parent T387833).

@jcrespo for gerrit1003, that is the primary server and that is certainly a different issue. Can you please file a new task for that? If you have examples of files being touched that would help. Thanks!

This has happened again in T411452

In T409832#11413329, @MoritzMuehlenhoff wrote:

In T409832#11404160, @ABran-WMF wrote:

@MoritzMuehlenhoff as @Dzahn mentioned, you might be interested to review what's been done on the gitlab-package-puller script. I'd be happy to have a chat about it if needed, let me know if you see any missing things or any room for improvement

The changes to the puller script and the alerting look good to me, I'm wondering however if this shouldn't rather alert the "sre" team instead of just collaboration? If the alerts are flagged in #wikimedia-operations this would also allow to see the people who uploaded a broken package that something failed?

In T410756#11397377, @Dzahn wrote:

Ah, I had moved this host back into "insetup" role back in October: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198128

That removed the docker.io package but it did not delete zuul services.

Not exactly sure why the alert started only now.

But first fix is to manually delete the service files:

root@zuul2001:/lib/systemd/system# rm zuul-
zuul-nodepool.service   zuul-scheduler.service  zuul-web.service

After discussing with @MoritzMuehlenhoff I removed the duplicate package from /incoming on apt-staging2001. We also talked about cleaning the /incoming folder regularly with a job to prevent issues like this in the future.

Thank you! I thank that can be resolved, the logo looks good to me in a fresh browser. My current session uses the old logo somehow.

Thanks @Aklapper for submitting the change. I tweaked the colors a bit more, see my comment in the change.

Yes that was one of the files I was looking at. However the wmfmariadbpy package creates some more .deb files, see:

Happened because of the upgrade T407943

All instances were updated successfully. I'll resolve the task.

This happened because of the reboot in T407110. It resolved automatically.

Test hosts done, I'll proceed with the replicas soon.

I'll bump the apt package to 18.3* and start upgrading the test hosts.

In T407557#11284684, @Ladsgroup wrote:

Assuming gitlab ssh config needs changing, this needs attention of collaboration-services team. Production seems to be handled already.

My new FIDO ssh key was added and works and the old ssh key was removed. I'll resolve the task.

The gitlab-ce and gitlab-runner packages were removed from the Update section in the apt repo config. So the old buster and bullseye packages should no longer be updated. We agreed to keep the old packages around to make sure reprepro is not complaining.

In T403946#11260702, @jcrespo wrote:

Is it ok to deploy now?

Great, then I'll resolve this task. I opened T406824: Evaluate generic backup tooling for object storage buckets as a follow up to track the object storage backup work in a more generic way.

Today is another scheduled maintenance for Gerrit: T387833. So this task is blocked again.

Related to the upgrade in T406801.

Deployment-wise everything is done, gitlab artifacts and packages use the object storage backend now.

reggie is quite a critical component of the non-production/dev CI stack. So maybe it's possible to make it a bit more resilient or HA (which is probably tricky because of the persistent volume).

The reggie-01 pod was moved to another node and the pvc was still stuck on the old node. I guess that happened during automatic scale down of the cluster.

The puppet issue is fixed, I'll resolve the task.

The change above increased the paging delay to 15 minutes. So short traffic bursts should not page immediately. For paging Phabricator has to be slow (latency 6s) for at least 15 minutes.

In my opinion no downtime is needed for gitlab-runners.

I'll bump the alert_after threshold from 5m to 15m (also to reduce alerting noise for the new 24/7 shift).

I tried to extract the error from the failed cookbook execution from the cumin log:

This was a small traffic burst which automatically resolved. I'll resolve the task, we can do more investigation if that happens again.

In T403946#11241419, @jcrespo wrote:

@Jelto @Dzahn The backup seems to be progressing correctly. I will revert the test and we can try to deploy the new configuration on Monday.

Thank you @dancy , I'll take a look!

I don't have the full context here, but if the new gitlab-packages fileset (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1192535) causes any trouble, this can be disabled for a few days/weeks from our side. The gitlab fileset is much more important and should be much smaller now, around 50-100GBs instead of 400GB+ like in the past.

Today the metamonitoring pa.ged on-call SREs. When checking the metamonitoring_public_endpoint.service on alert1002 I see a python stack trace during the time of the the alert:

@BTullis @bking there is an undeployed change for flink in the dse-k8s-eqiad cluster which made it a bit tricky to deploy the new wikikube-eqiad CIDRs (to update network policies).

The update is blocked by T406094 currently

In T378922#11228699, @jcrespo wrote:

Do we need daily full backups for objects? Assuming only a few object change per day, cannot we do incrementals? We currently don't have the space to backup daily 500 GB until we dedicate the storage for gitlab.

Sorry, I am not sure if you refer to gitlab package or object backup. The packages should be full, but I was hoping to do incrementals for objects.

All GitLab packages are migrated to APUS object storage. The additional sync from bucket to the backup directory /srv/gitlab-backup also works and performance of the APUS cluster is great. A small fix was needed (see patch above) to make sure s3cmd does not pull all packages on every run. But after fixing the trailing slashes in the folder path the sync works without generating too much traffic.

This happened because of T378922, resolved after deploying a fix

Sync from object storage to a local folder works with the new ceph::client::sync_local module. I tested this on GitLab replica gitlab2002: