PDF diagram that shows Separation into untrusted and trusted environments that has been talked about in SvcOps before

Yes, we will take a look to see how the new database can be put on all appservers.

In T288375 we are discussing how this extension would have access to the maxmind databases when we migrate MediaWiki to Kubernetes. It would be helpful to know how important the "freshness" of the database is, i.e. how quickly after maxmind publishes an update do we need that update in production? Maxmind updates the database mostly in a weekly rhythm, but for certain elements quicker (and slower). See: https://support.maxmind.com/geoip-faq/databases-and-database-updates/how-often-are-the-geoip2-and-geoip-legacy-databases-updated/. One of the options is too include the maxmind database in the release, which happens mostly daily, but in extreme cases (no deployment weeks for example, christmas holidays, etc) there can be intervals of up to 10 days without a release. Is that acceptable to the users of IP Info? The migration to Kubernetes is currently in the works, a first version can be accessed via the Wikimedia Debug extension for Firefox or Chrome and then selecting "k8s-experimental", we are projecting a full migration for early 2022.

Regarding 1 - a maxmind db change requires a new MediaWiki image and deployment. I believe the current frequency of MediaWiki deployments (multiple times a week) is good enough for the maxmind databases. The automated update of databases to the build system and the inclusion in the next image should be good enough.

Papaul, Effie is on vacation for 2 more weeks. Just FYI, so don't expect any action here soon.

I have added a bit of text to explain services

Is this the dashboard? https://grafana.wikimedia.org/d/U7JT--knk/joe-k8s-mwdebug?viewPanel=70&orgId=1&from=1625227688488&to=1625246654342

Thanks everybody for the feedback on the communications for the DC switchover process. We will spend some time this quarter (Q1) in working through the typical timelines for the process and define the groups involved and the best mechanism to make sure that everybody is aware of the status and progress. Please take a look at https://wikitech.wikimedia.org/wiki/Switch_Datacenter and https://wikitech.wikimedia.org/wiki/Switch_Datacenter/Coordination to see if what we have captured there makes sense.

Notes by Quiddity:

Let's use the restricted group

3 days duration confirmed as sufficient.

We are waiting for Thumbor to migrate to k8s before retiring these servers. New servers that were originally been purchased for thumbor have been repurposed for mediawiki. I will check with the dev team on where they are with the k8s migration.

We are waiting for Thumbor to migrate to k8s before retiring these servers. New servers that were originally been purchased for thumbor have been repurposed for mediawiki. I will check with the dev team on where they are with the k8s migration.

gitlab is configured to store backups in /srv/gitlab-backup

I think that is something that Jelto (our new SRE - starts June 7) can handle, i.e. add a second disk with the right dimensions.

approved

On my test machine I see the following:

I think we should enable the following exporters:

Approved.

The SLO project addresses some points: https://wikitech.wikimedia.org/wiki/SLO - we will continue to add applications on wikitech as they get defined together with engineering and product. The SLO project has been going on in FY20/21 and will continue in FY21/22.

Latest state that I am aware of:

For Prometheus/Grafana:

To provide as little obstacles for developers as possible access through port 22 is the preferred option.

Let's close and reopen if a second server becomes necessary.

gitlab1001 has the standard exim mail package installed, so local e-mail works already. What does Gitlab need?

https://docs.gitlab.com/omnibus/settings/nginx.html#update-the-ssl-certificates has the command to execute when a certificate is updated: gitlab-ctl hup nginx

Let's go with the simpler solution and use the CNAME.

gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well?

I created https://phabricator.wikimedia.org/T276673 to track the decison on how to manage HTTPS certificates.

Approved.

Please recreate the 2 VMs in the VLAN that allows for direct external IP addresses.

A second sshd daemon for port 29418 concentrates all configuration for gitlab/ssh access in a separate configuration file and does not touch the "normal" config file. That seems to me a more robust way of configuring, minimizing impact on sshd on port 22. sshd on port 29418 can then simply be limited to user git.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/667731

Yes, we can do all servers in A3 at once.