Page MenuHomePhabricator

Jelto (jwodstrcil)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Jun 7 2021, 7:25 AM (14 w, 5 d)
Availability
Available
LDAP User
Jelto
MediaWiki User
JWodstrcil (WMF) [ Global Accounts ]

Recent Activity

Yesterday

Jelto updated the task description for T291322: Create GitLab mirror.
Sat, Sep 18, 10:04 AM · GitLab
Jelto created T291322: Create GitLab mirror.
Sat, Sep 18, 10:02 AM · GitLab

Fri, Sep 17

Jelto updated the task description for T251305: Migrate to helm v3.
Fri, Sep 17, 10:01 AM · Patch-For-Review, Kubernetes, serviceops, SRE
Jelto updated the task description for T251305: Migrate to helm v3.
Fri, Sep 17, 10:00 AM · Patch-For-Review, Kubernetes, serviceops, SRE
Jelto added a comment to T251305: Migrate to helm v3.

My current plan for cluster-wise migration and deploying services with helm3 is:

Fri, Sep 17, 10:00 AM · Patch-For-Review, Kubernetes, serviceops, SRE

Wed, Sep 15

jcrespo awarded T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200) a Like token.
Wed, Sep 15, 7:09 AM · Patch-For-Review, Release-Engineering-Team (Radar), User-brennen, GitLab
Jelto closed T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200) as Resolved.

The warning was removed in the GitLab upgrade to 14.x (https://phabricator.wikimedia.org/T289802.) The grep was removed in https://gerrit.wikimedia.org/r/719930. So I'm closing this issue.
Feel free to open again.

Wed, Sep 15, 6:42 AM · Patch-For-Review, Release-Engineering-Team (Radar), User-brennen, GitLab
Jelto closed T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200), a subtask of T132324: Tracking and Reducing cron-spam to root@ , as Resolved.
Wed, Sep 15, 6:41 AM · Tracking-Neverending, SRE

Tue, Sep 14

Jelto added a comment to T290963: Drop the use of nonexisting groups in kubernetes infrastructure_users.

Thanks for looking this up!

Tue, Sep 14, 12:55 PM · Prod-Kubernetes, Kubernetes
Jelto updated subscribers of T285867: GitLab replica in codfw.

@Arnoldokoth some thoughts on the wrong settings on the replica after restore:

Tue, Sep 14, 12:06 PM · Patch-For-Review, serviceops, GitLab

Wed, Sep 8

Jelto added a comment to T251305: Migrate to helm v3.

Some additional RBAC requirements:

Wed, Sep 8, 3:56 PM · Patch-For-Review, Kubernetes, serviceops, SRE
Jelto added a comment to T284628: Kubernetes timeing out before pulling the mediawiki-multiversion image .

@dancy Thanks for finding this issue!

Wed, Sep 8, 3:20 PM · Kubernetes, serviceops, MW-on-K8s
Jelto added a comment to T289802: GitLab major version upgrade: 14.x.

I updated gitlab-ce to 14.2.3-ce.0 on apt1001

Wed, Sep 8, 2:52 PM · Release-Engineering-Team (Yak Shaving 🐃🪒), serviceops, User-brennen, GitLab
Jelto added a comment to T289802: GitLab major version upgrade: 14.x.

I updated gitlab-runner to 14.2.0 and gitlab-ce to 14.1.5-ce.0 on apt1001.

Wed, Sep 8, 1:34 PM · Release-Engineering-Team (Yak Shaving 🐃🪒), serviceops, User-brennen, GitLab
Jelto added a comment to T289802: GitLab major version upgrade: 14.x.

@Jelto I'll coordinate with you tomorrow, but if you want to go ahead with the upgrade on gitlab2001 before I'm online, feel free.

Wed, Sep 8, 10:25 AM · Release-Engineering-Team (Yak Shaving 🐃🪒), serviceops, User-brennen, GitLab

Mon, Sep 6

Jelto added a comment to T251305: Migrate to helm v3.

helm test annotations changed a bit:

Note that until Helm v3, the job definition needed to contain one of these helm test hook annotations: helm.sh/hook: test-success or helm.sh/hook: test-failure. helm.sh/hook: test-success is still accepted as a backwards-compatible alternative to helm.sh/hook: test.

We might also want to add "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded as of https://helm.sh/docs/topics/charts_hooks/#hook-deletion-policies

Mon, Sep 6, 1:09 PM · Patch-For-Review, Kubernetes, serviceops, SRE
Jelto updated subscribers of T286958: Document long-term requirements for GitLab job runners.

I started to collect some thoughts for the long-term GitLab Runner setups here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Future_Gitlab_Runner_setup_(T286958)

Mon, Sep 6, 8:03 AM · Release-Engineering-Team (Seen), User-brennen, GitLab

Fri, Sep 3

Jelto added a comment to T251305: Migrate to helm v3.

Adding a comment in here since I am trying to figure out a similar thing (although I have way less context) for what we'll probably call ml-services dir under helmfile.d (see T286791). I would love to start with helm3 straight away, and we could think about using the ml-serve-eqiad cluster as testbed in case it fits (we are testing an MVP, nothing in production yet). Let me know!

Fri, Sep 3, 12:30 PM · Patch-For-Review, Kubernetes, serviceops, SRE

Tue, Aug 31

Jelto edited P17115 kubestage1001 helm, tiller and IO debugging.
Tue, Aug 31, 12:48 PM · Kubernetes
Jelto edited P17115 kubestage1001 helm, tiller and IO debugging.
Tue, Aug 31, 12:47 PM · Kubernetes
Jelto created P17115 kubestage1001 helm, tiller and IO debugging.
Tue, Aug 31, 12:44 PM · Kubernetes

Mon, Aug 30

Jelto triaged T290020: Evaluate and enable audit logging for kubeapi-server as Low priority.
Mon, Aug 30, 4:39 PM · Prod-Kubernetes, serviceops, Kubernetes
Jelto created T290020: Evaluate and enable audit logging for kubeapi-server.
Mon, Aug 30, 4:39 PM · Prod-Kubernetes, serviceops, Kubernetes

Sat, Aug 28

Jelto closed T289454: GitLab Puma reduced availability due to automated restart as Resolved.
Sat, Aug 28, 6:13 PM · serviceops, GitLab (Initialization)
Jelto added a comment to T289454: GitLab Puma reduced availability due to automated restart.

I tested the behavior of gitlab2001 during a rolling restart of puma workers:

Sat, Aug 28, 6:13 PM · serviceops, GitLab (Initialization)
Jelto created P17100 GitLab availability log 202108281122.
Sat, Aug 28, 6:10 PM

Thu, Aug 26

Jelto added a comment to T251305: Migrate to helm v3.

There is a ClusterRole named deploy already for the aggregation of view and pods/portForward permissions. So I would prefer using the names <service-name> and <service-name>-rw for the two users. If we use the deploy in the name of the new user this may be confused with the read-only user and the deploy ClusterRole.

Thu, Aug 26, 1:23 PM · Patch-For-Review, Kubernetes, serviceops, SRE
Jelto added a comment to T251305: Migrate to helm v3.

And some more input regarding RBAC and the replacement of Tiller service account:

Thu, Aug 26, 8:21 AM · Patch-For-Review, Kubernetes, serviceops, SRE

Wed, Aug 25

Jelto added a comment to T251305: Migrate to helm v3.

That is pretty cool, thanks! Did you actually deploy something using helm3 with tillerNamespace: still set? Is it just ignored in that case (helmfile can be really picky :))?

Wed, Aug 25, 7:43 AM · Patch-For-Review, Kubernetes, serviceops, SRE

Tue, Aug 24

Jelto added a comment to T251305: Migrate to helm v3.

The helm binary in helmfile can be set using the --helm-binary option or by setting helmBinary in the helmfile.yaml.
It can be set globally (like in admin-ng). But it is also possible to to change the helm binary depending on the environment.

Tue, Aug 24, 2:01 PM · Patch-For-Review, Kubernetes, serviceops, SRE

Mon, Aug 23

Jelto added a comment to T289454: GitLab Puma reduced availability due to automated restart.

gitlab1001 just had another rolling restart of puma workers:

Mon, Aug 23, 2:32 PM · serviceops, GitLab (Initialization)
Jelto created P17063 GitLab availability log 20210803140550.
Mon, Aug 23, 2:25 PM
Jelto created P17062 GitLab check availability script.
Mon, Aug 23, 2:23 PM
Jelto updated the task description for T289454: GitLab Puma reduced availability due to automated restart.
Mon, Aug 23, 8:13 AM · serviceops, GitLab (Initialization)
Jelto added a comment to T275170: Define monitoring for gitlab.

I created a dedicated task for the reduced availability for the puma worker/exporter: https://phabricator.wikimedia.org/T289454

Mon, Aug 23, 8:05 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability
Jelto triaged T289454: GitLab Puma reduced availability due to automated restart as Low priority.
Mon, Aug 23, 8:04 AM · serviceops, GitLab (Initialization)
Jelto created T289454: GitLab Puma reduced availability due to automated restart.
Mon, Aug 23, 8:04 AM · serviceops, GitLab (Initialization)
Jelto updated the task description for T251305: Migrate to helm v3.
Mon, Aug 23, 7:12 AM · Patch-For-Review, Kubernetes, serviceops, SRE

Sun, Aug 22

Jelto added a comment to T287117: Establish a routine GitLab deployment / update window.

I like the idea of having a semi-formal update window for GitLab as well.

Sun, Aug 22, 9:41 AM · GitLab, User-brennen, Release-Engineering-Team (Next)

Aug 19 2021

Jelto updated subscribers of T275170: Define monitoring for gitlab.

I would like to either finish this task or add additional requirements. Currently we are collecting metrics of all GitLab components on gitlab1001 and gitlab2001. We have Grafana dashboards and basic alerts in icinga.

Aug 19 2021, 9:31 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Aug 18 2021

Jelto closed Unknown Object (Task), a subtask of T215411: (OoW) thumbor1004 memory errors, as Invalid.
Aug 18 2021, 2:49 PM · User-jijiki, Thumbor, ops-eqiad, serviceops, SRE
Dzahn awarded T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) a Like token.
Aug 18 2021, 2:47 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops
Jelto reopened Unknown Object (Task), a subtask of T215411: (OoW) thumbor1004 memory errors, as Open.
Aug 18 2021, 2:42 PM · User-jijiki, Thumbor, ops-eqiad, serviceops, SRE
Jelto added a comment to T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).

From my side everything is done. Thanks everyone. I'm going to close this ticket. Feel free to open it in case I missed anything.

Aug 18 2021, 2:42 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops
Jelto closed T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) as Resolved.
Aug 18 2021, 2:42 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops

Aug 17 2021

Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Aug 17 2021, 8:39 AM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Aug 17 2021, 7:40 AM · Patch-For-Review, SRE, serviceops

Aug 13 2021

Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Aug 13 2021, 5:14 PM · Patch-For-Review, SRE, serviceops
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Aug 13 2021, 1:13 PM · Patch-For-Review, SRE, serviceops

Aug 12 2021

Jelto added a comment to T275170: Define monitoring for gitlab.

The puma workers get killed either if they hit memory limit (which is puma['per_worker_max_memory_mb'] = 1024 ~1gb by default) or automatically after 12 hours. I can't see one of the workers hitting the limit in the past. They are getting killed because the uptime is above 12 hours and during the restart the service availability is reduced. My assumption is that some parts of the puma webserver/GitLab have memory leaks. This is the reason for the filling memory and why GitLab uses tools like puma_worker_killer. I try to debug and troubleshoot this problem on gitlab2001 a little bit further. I would like to have a smooth restart of the puma workers without reduced availability.

Aug 12 2021, 11:07 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Aug 11 2021

Jelto added a comment to T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200).

Should be fixed with https://gerrit.wikimedia.org/r/710676. No new error message from root@gitlab1001

Aug 11 2021, 6:57 AM · Patch-For-Review, Release-Engineering-Team (Radar), User-brennen, GitLab

Aug 6 2021

Dzahn awarded T285867: GitLab replica in codfw a Grey Medal token.
Aug 6 2021, 4:13 PM · Patch-For-Review, serviceops, GitLab
Jelto claimed T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200).
Aug 6 2021, 10:24 AM · Patch-For-Review, Release-Engineering-Team (Radar), User-brennen, GitLab

Aug 5 2021

Jelto added a comment to T285867: GitLab replica in codfw.

I restored the backup of gitlab1001 to gitlab2001 using the restore instructions of S&F.
I enhanced the guide and moved it to wikitech: https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore#Restore

Aug 5 2021, 5:58 PM · Patch-For-Review, serviceops, GitLab

Aug 4 2021

Jelto renamed T285870: request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001 from request service IP / DNS name for gitlab-failover, apply puppet role on gitlab2001 to request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001.
Aug 4 2021, 11:03 AM · Infrastructure-Foundations, serviceops, GitLab
Jelto closed T285870: request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001, a subtask of T285867: GitLab replica in codfw, as Resolved.
Aug 4 2021, 11:03 AM · Patch-For-Review, serviceops, GitLab

Aug 2 2021

Jelto added a comment to T275170: Define monitoring for gitlab.

GitLab rails service has reduced availability roughly every ~24 hours (plus some offsets). See this Grafana dashboard.

Aug 2 2021, 1:54 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Aug 2 2021, 7:53 AM · Patch-For-Review, SRE, serviceops

Jul 30 2021

Jelto added a comment to T275170: Define monitoring for gitlab.

Basic Icinga alerts for the public https and SSH endpoints of GitLab are in place now: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=gitlab.wikimedia.org

Jul 30 2021, 10:59 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Jul 27 2021

Jelto added a comment to T275170: Define monitoring for gitlab.

The scrape configuration for GitLab is in place and Prometheus collects metrics.

Jul 27 2021, 3:34 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Jul 23 2021

Jelto added a comment to T285867: GitLab replica in codfw.

I modified and ran the install script using ansibles --check (dry-run) flag against gitlab2001. Looks good, there are two errors due to check mode usage.

Jul 23 2021, 1:57 PM · Patch-For-Review, serviceops, GitLab
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Jul 23 2021, 12:54 PM · Patch-For-Review, SRE, serviceops
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Jul 23 2021, 12:14 PM · Patch-For-Review, SRE, serviceops
Jelto committed rLPRIc545963eb6fc: add mcrouter certs for mw1422.eqiad.wmnet to mw1446.eqiad.wmnet (authored by Jelto).
add mcrouter certs for mw1422.eqiad.wmnet to mw1446.eqiad.wmnet
Jul 23 2021, 11:30 AM

Jul 21 2021

Jelto updated subscribers of T233196: Migrate thumbor to Kubernetes .
Jul 21 2021, 9:02 AM · SRE, serviceops, Thumbor
Jelto added a comment to T233196: Migrate thumbor to Kubernetes .

During the refresh of old mw app servers in eqiad we noticed that thumbor machines thumbor1001 and thumbor1002 are renamed/reimaged mw hosts. As mentioned in T280203 and T233196 these machines are end of life and have to be refreshed.

Jul 21 2021, 9:01 AM · SRE, serviceops, Thumbor
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Jul 21 2021, 8:22 AM · Patch-For-Review, SRE, serviceops
Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Jul 21 2021, 8:21 AM · Patch-For-Review, SRE, serviceops

Jul 15 2021

Jelto claimed T285870: request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001.
Jul 15 2021, 2:15 PM · Infrastructure-Foundations, serviceops, GitLab
Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Jul 15 2021, 1:59 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops
Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Jul 15 2021, 1:58 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops

Jul 14 2021

Jelto claimed T275170: Define monitoring for gitlab.
Jul 14 2021, 11:22 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Jul 13 2021

Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Jul 13 2021, 12:38 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops
Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Jul 13 2021, 11:39 AM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops

Jul 2 2021

Jelto committed rLPRI1c7918f3d590: add mcrouter certs for mw1415.eqiad.wmnet to mw1421.eqiad.wmnet (authored by Jelto).
add mcrouter certs for mw1415.eqiad.wmnet to mw1421.eqiad.wmnet
Jul 2 2021, 11:44 AM
Jelto committed rLPRI7a7bfe9ae6a7: add mcrouter certs for mw1414.eqiad.wmnet (authored by Jelto).
add mcrouter certs for mw1414.eqiad.wmnet
Jul 2 2021, 9:09 AM

Jul 1 2021

Jelto updated the task description for T279309: bring 43 new mediawiki appserver in eqiad into production.
Jul 1 2021, 2:00 PM · Patch-For-Review, SRE, serviceops
Jelto updated the task description for T280203: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301).
Jul 1 2021, 1:57 PM · ops-eqiad, decommission-hardware, Patch-For-Review, SRE, serviceops

Jun 30 2021

Jelto added a project to T285870: request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001: Infrastructure-Foundations.
Jun 30 2021, 3:00 PM · Infrastructure-Foundations, serviceops, GitLab
Jelto closed T285456: codfw: 1 of VMs requested for gitlab, a subtask of T285867: GitLab replica in codfw, as Resolved.
Jun 30 2021, 2:43 PM · Patch-For-Review, serviceops, GitLab
Jelto closed T285456: codfw: 1 of VMs requested for gitlab as Resolved.
Jun 30 2021, 2:43 PM · GitLab, serviceops, SRE, vm-requests
Jelto added a parent task for T285456: codfw: 1 of VMs requested for gitlab: T285867: GitLab replica in codfw.
Jun 30 2021, 2:42 PM · GitLab, serviceops, SRE, vm-requests
Jelto added a subtask for T285867: GitLab replica in codfw: T285456: codfw: 1 of VMs requested for gitlab.
Jun 30 2021, 2:42 PM · Patch-For-Review, serviceops, GitLab
Jelto created T285867: GitLab replica in codfw.
Jun 30 2021, 2:41 PM · Patch-For-Review, serviceops, GitLab

Jun 28 2021

Jelto added a comment to T284431: Refresh all images in production-images.

I merged and rolled out the additional parameter for intra-service dependencies in systemd::timer::job and the change for a weekly rebuild of production-images.

Jun 28 2021, 12:50 PM · docker-pkg, serviceops, SRE
Jelto claimed T284431: Refresh all images in production-images.
Jun 28 2021, 12:37 PM · docker-pkg, serviceops, SRE

Jun 25 2021

Jelto updated the task description for T285456: codfw: 1 of VMs requested for gitlab.
Jun 25 2021, 1:27 PM · GitLab, serviceops, SRE, vm-requests

Jun 22 2021

Jelto added a comment to T274463: Backups for GitLab.

I can explain this, I have created a gitlab module which is not used in production, but is running in a cloud project (ping me if you want access). The modules is a first pass at converting the ansible code into puppet code. I think its about 80-90% complete. I'm not sure what the future plans are here in relation to if we are integrating gitlab further into puppet or moving it out of puppet and into a k8s cluster, so for now i have mostly stalled on the production aspects of the code (for my needs i just needs a basic instance up to test api's); however happy to revive things or go through the code if its useful

Jun 22 2021, 12:37 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 16 2021

Jelto added a comment to T274463: Backups for GitLab.

@Dzahn I think https://gerrit.wikimedia.org/r/c/operations/puppet/+/697850 is not merged and deployed, so the fileset for GitLab doesn't exist.

Jun 16 2021, 11:00 AM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 15 2021

Jelto added a comment to T284431: Refresh all images in production-images.

So we have two options for a rebuild:

Jun 15 2021, 8:27 AM · docker-pkg, serviceops, SRE

Jun 14 2021

Jelto added a comment to T284431: Refresh all images in production-images.

@Joe I think a periodic job rebuilding the images is implemented already.
See modules/docker/manifests/baseimages.pp#65

Jun 14 2021, 11:00 AM · docker-pkg, serviceops, SRE

Jun 11 2021

Jelto added a comment to T275170: Define monitoring for gitlab.

I checked the current exporter configuration.
Host/node metrics are available by node_exporter. The node_exporter in GitLab is disabled because we are using the stand-alone node_exporter. The host metrics are available in Grafana already. So host/node metrics should be fine.

Jun 11 2021, 2:43 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Jun 10 2021

Jelto created T284733: New VictorOps user request.
Jun 10 2021, 11:30 AM · observability

Jun 9 2021

Jelto added a comment to T274463: Backups for GitLab.

@Dzahn and I checked the backups on gitlab1001.

Jun 9 2021, 5:02 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 8 2021

Jelto created T284543: Please add Jelto to security@wikimedia.org mailing list.
Jun 8 2021, 8:50 AM · SecTeam-Processed, Security-Team