Page MenuHomePhabricator

Sergey.Trofimovsky.SF (Sergey Trofimovsky)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Feb 26 2021, 6:58 PM (39 w, 6 d)
Availability
Available
LDAP User
Sergey Trofimovsky
MediaWiki User
Unknown

Recent Activity

Jul 20 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

Following on a previous discussion, noting some concerns that current implementation (copying over the latest backup to a separate latest file) may cause. It's not a major problem, but definitely something that can be optimized.

Jul 20 2021, 6:39 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 9 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

A bit of a context: the plan was to hold 7 days worth of backups locally, but in order to test local retention and its interaction with Bacula this number was to be be reduced to 3 days for the duration of the early deployment stage. @wkandek correct me if I'm wrong, please.

Jun 9 2021, 5:33 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 4 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

It may contain sensitive data, particularly /etc/gitlab/gitlab-secrets.json. If you guys believe a single backup set is safe enough for both general backup and configuration/secrets, then it's fine with us.

Jun 4 2021, 8:49 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

On a different note, I'd like to remind about a separate backup set for Gitlab configuration backup, which we discussed earlier (if memory doesn't fail me) : /etc/gitlab/config_backup

Jun 4 2021, 5:22 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.

Also, guys, can you verify that the manual UI configurations were done after running Ansible?

Jun 4 2021, 5:13 PM · Patch-For-Review, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.
Jun 4 2021, 5:00 PM · Patch-For-Review, GitLab (Initialization)

Jun 1 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

Yes, this is reasonable. These two variables need to be updated accordingly (and Ansible redeployed):

Jun 1 2021, 7:17 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

May 28 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

We can create a second virtual hard disk and mount it into the existing file system of the VM. It requires a restart of the VM.

How much space would be needed though? We need to check resources on the ganeti cluster if it's a lot.

May 28 2021, 12:36 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.

Thanks! I confirm we can now log in to Gitlab.

May 28 2021, 12:32 AM · Patch-For-Review, GitLab (Initialization)

May 27 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

Looking at gitlab1001, I only see a root volume:

May 27 2021, 9:52 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

Is this something you guys will be changing (and deploying)? Just to verify that we are on the same page here.

May 27 2021, 5:51 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

I believe this is going to be finished today, you should not get more of these, sorry about that.

May 27 2021, 4:24 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

The installation has not been completed yet, the server has been brought down intentionally, backup failures are expected at the time.

May 27 2021, 3:24 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

May 26 2021

Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.

Considering that installation is in an unfinished state now (especially for the default password reset part, and then UI configurations) the safe way would be to shut Gitlab down with gitlab-ctl until it can be finished. However, password can be reset with CLI (sudo gitlab-rake "gitlab:password:reset, https://docs.gitlab.com/ee/security/reset_user_password.html) and installation left alone until tomorrow.

May 26 2021, 5:25 PM · Patch-For-Review, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.

Yes, it seems exactly like it, everything else seems fine so far.

May 26 2021, 4:51 PM · Patch-For-Review, GitLab (Initialization)

May 25 2021

Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

On a closer inspection, this setting is not directly monitoring related in the context of this ticket. It configures Prometheus server for Gitlab UI Prometheus integration: https://docs.gitlab.com/ee/user/project/integrations/prometheus.html

May 25 2021, 5:15 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 21 2021

Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

This is configurable now: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/+/8c0007586e4b25da076a8ce624411050bd9abe9b%5E%21/#F0

May 21 2021, 12:04 AM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 19 2021

Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

From now on this is going to be default:

# Monitoring configuration
gitlab_prometheus_enable: "false"
gitlab_grafana_enable: "false"
gitlab_alertmanager_enable: "false"
gitlab_gitlab_exporter_enable: "false"
gitlab_node_exporter_enable: "false"
gitlab_postgres_exporter_enable: "false"
gitlab_redis_exporter_enable: "false"
May 19 2021, 4:14 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 18 2021

Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

Along with the other settings for an eternal prometheus server

May 18 2021, 5:14 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 17 2021

Sergey.Trofimovsky.SF added a comment to T282496: Add LICENSE/COPYING to operations/gitlab-ansible.

Here's the LICENSE file finally in the root of the repo, please review: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/+/refs/heads/master/LICENSE

May 17 2021, 4:04 PM · Release-Engineering-Team (Seen), User-brennen, GitLab (Initialization), Software-Licensing

May 14 2021

Sergey.Trofimovsky.SF added a comment to T275170: Define monitoring for gitlab.

As soon as we have a running production Gitlab instance I will post an update here with the list of Prometheus exporter endpoints to pull.

May 14 2021, 6:09 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 13 2021

Sergey.Trofimovsky.SF added a comment to T282496: Add LICENSE/COPYING to operations/gitlab-ansible.

This is weird, I vividly remember we added a LICENSE file to the repo. The original Gitlab-suggested playbook we started with was licensed under MIT. We'll verify and re-add the license back to the repo.

May 13 2021, 9:11 PM · Release-Engineering-Team (Seen), User-brennen, GitLab (Initialization), Software-Licensing

May 10 2021

Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.

More good news: inspired by this issue (https://gitlab.com/gitlab-org/gitlab/-/issues/24510), @Sfigor was able to make LogOut work on the Gitlab side.

May 10 2021, 9:57 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen
Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.

Requesting some more information on the current LDAP schema, which attributes can and should be used for key mapping (https://github.com/tduehr/omniauth-cas3), specifically these:

May 10 2021, 6:44 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen
Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.

Anyway, given that the decision to stick with SSO has been made, we're going with CAS, having following limitations in mind:

  • No SSH keys import/sync at the moment (can probably be implemented at later stages with fetch_raw_info)
  • No group membership (turns out that was never a requirement)
  • No git remote passwords from SSO
  • Logout current state
May 10 2021, 6:18 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen
Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.
May 10 2021, 6:18 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

These were all addressed (please review). Please note, that some options are overridden in hostvars for production.

May 10 2021, 4:54 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

May 8 2021

Sergey.Trofimovsky.SF added a comment to T274462: Logging for GitLab.

Just to confirm that we are on the same page here: Logstash agents are installed and configured by Puppet, we're only proving the list of logs to be ingested and configuring output format if needed, right?

May 8 2021, 7:25 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T274462: Logging for GitLab.
May 8 2021, 7:22 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)

May 7 2021

Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.

For the moment, I'll just put it here for history, documentation to the 3rd party OmniAuth CAS Strategy module: https://github.com/tduehr/omniauth-cas3, which brings light to configuration options and features.

May 7 2021, 9:09 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen
Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

One thing that should be made explicit is that we had into account the amount of disk and resources needed to store backups in our long term storage (bacula). Any resources needed to generate the tarballs and store them for a short term until they are backed up by bacula (in a similar way we do for databases, e.g., where we store a couple of recent exports in case something goes wrong with the exporting process) it will have to be accounted on your side or budget - (e.g. extra disk space on the local gitlab hosts for one or several exports) and added on your own into the annual plan- basically the pure service needs outside of remote backups.

We wanted to make this explicit to prevent confusion.

Ack, we'll make sure we have this covered on our side.

May 7 2021, 6:32 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T274462: Logging for GitLab.

If by shipped you guys mean ingested by Logstash into an ElasticSearch instance, then yes, this was the plan.

May 7 2021, 6:07 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)

May 3 2021

Sergey.Trofimovsky.SF added a comment to T276459: Outgoing mail for Gitlab.

Gitlab outbound mail configuration is done:

May 3 2021, 6:08 PM · Mail, GitLab (Initialization)

Apr 30 2021

Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Here it is, requesting settings review:

Apr 30 2021, 9:56 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

Apr 14 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

Using Gerrit backups as a baseline makes sense. What components are currently included in the hourly Gerrit backups? What is the retention policy for build artifacts, build data (logs etc)? Any information would be valuable here. Thank you.

Apr 14 2021, 12:55 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

The plan overall is to utilize Gitlab's built in backup that (to some point) takes care of the consistency of backups, including database (PostgreSQL) and repositories. It backs up components separately, as large (tar.gz) files:

Apr 14 2021, 12:49 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Apr 13 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

It is possible to have multiple backup sets with different components included. For example, a weekly full backup of all components, and a daily backup that does not include artifacts and CI builds, preserving backup space and reducing backup runtime. Let us know if this is a strategy we should proceed with.

Apr 13 2021, 1:28 AM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Apr 12 2021

Sergey.Trofimovsky.SF added a comment to T274463: Backups for GitLab.

As per Gitlab suggestion (https://docs.gitlab.com/omnibus/settings/backups.html):

It is recommended to keep a copy of /etc/gitlab, or at least of /etc/gitlab/gitlab-secrets.json, in a safe place. If you ever need to restore a GitLab application backup you need to also restore gitlab-secrets.json. If you do not, GitLab users who are using two-factor authentication will lose access to your GitLab server and 'secure variables' stored in GitLab CI will be lost.
It is not recommended to store your configuration backup in the same place as your application data backup, see below.
Apr 12 2021, 10:36 PM · GitLab (Infrastructure), serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Apr 8 2021

Sergey.Trofimovsky.SF added a comment to T279545: Gitlab Installation Procedure.

@jbond Also, any feedback is welcome and expected, please let us know. Thanks!

Apr 8 2021, 8:55 PM · Patch-For-Review, GitLab (Initialization)

Apr 7 2021

Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.
Apr 7 2021, 5:41 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

Apr 6 2021

Dzahn awarded T276148: SSH Access of Git data in GitLab a Like token.
Apr 6 2021, 9:28 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Sounds like a plan to me then.

Apr 6 2021, 8:18 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

boldly assigning this to @Sergey.Trofimovsky.SF now.

Apr 6 2021, 8:02 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

Mar 23 2021

Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Should it be gitlab.wikimedia.org for both, https and ssh? (so both the webserver and second sshd would listen on that new additonal IP).

Or should it be a different name, like git-ssh.wikimedia.org is for Phabricator?

Mar 23 2021, 10:27 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

Mar 19 2021

Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.

Something missing from the docs?

ahh yes, i have placed the ldap cn=admin password in idp01.sso.eqiad1.wikimedia.cloud:/root/ldap

Mar 19 2021, 6:40 AM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen

Mar 11 2021

Sergey.Trofimovsky.SF added a comment to T276673: Certificate Management for GitLab.

Agreed

Mar 11 2021, 12:09 AM · GitLab (Initialization)

Mar 10 2021

Sergey.Trofimovsky.SF added a comment to T276673: Certificate Management for GitLab.

The service has still to be notified somehow, to reload certificates at least. It can be implemented as simple as a hook (a predefined script for example) that can be called on a change. We will manage the script within the installation.

Mar 10 2021, 10:37 PM · GitLab (Initialization)

Mar 9 2021

Sergey.Trofimovsky.SF added a comment to T274461: Define auth strategy for GitLab.
Mar 9 2021, 8:22 PM · GitLab (Auth & Access), Release-Engineering-Team (Doing), SRE, User-brennen

Mar 8 2021

Sergey.Trofimovsky.SF added a comment to T276170: DNS for GitLab.

Yeah, good question - sorry I conflated machine-specific with a -test / -beta hostname in my response.

I think gitlab.wikimedia.org is good, on my current understanding that this box will be the MVP production host without being recreated in the meanwhile.

Mar 8 2021, 11:17 PM · SRE, Traffic, DNS, serviceops, Release-Engineering-Team-TODO, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276170: DNS for GitLab.
Mar 8 2021, 9:58 PM · SRE, Traffic, DNS, serviceops, Release-Engineering-Team-TODO, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276144: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed).

@Sergey.Trofimovsky.SF Do you agree we should keep the firewall closed until you say it's time to open up 80/443 ?

Mar 8 2021, 6:03 AM · SRE, Traffic, Patch-For-Review, User-brennen, GitLab (Initialization), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3))

Mar 5 2021

Sergey.Trofimovsky.SF added a comment to T276144: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed).

From our perspective Gitlab-managed auto-renewed Let's Encrypt certificates are the most straightforward and preferable solution, and this is what we're starting with unless you guys find otherwise.

Mar 5 2021, 10:42 PM · SRE, Traffic, Patch-For-Review, User-brennen, GitLab (Initialization), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3))

Mar 3 2021

Sergey.Trofimovsky.SF added a comment to T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.

Thanks, let's keep it this way!

Mar 3 2021, 8:09 PM · SRE, SRE-Access-Requests
Sergey.Trofimovsky.SF added a comment to T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.

Sergeys everywhere!

Mar 3 2021, 7:54 PM · SRE, SRE-Access-Requests

Mar 2 2021

Sergey.Trofimovsky.SF updated the task description for T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.
Mar 2 2021, 11:40 PM · SRE, SRE-Access-Requests
Sergey.Trofimovsky.SF added a comment to T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.

@jbond It's an outcome of me trying to separate personal and S&F accounts here, sorry about that. I updated the ticket with the correct _shell username_ (strofimovsky01), hope this helps.

Mar 2 2021, 11:39 PM · SRE, SRE-Access-Requests
Sergey.Trofimovsky.SF updated the task description for T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.
Mar 2 2021, 11:37 PM · SRE, SRE-Access-Requests
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Allow me to stress that the SSH port for Gitlab is a long term choice. Whatever is decided here, will have to be carried for a long time, it will stay in numerous remote origins forever. My +1 to a standard port here.

Mar 2 2021, 5:46 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Gitlab takes a bit of an opposite approach with this. Gitlab server manages its own user key database, as well as they can sync in user keys from LDAP. They do indeed support AuthorizedKeysCommand, but in their own way, to quickly look up a key in a local database, to alleviate issues with huge authorized_keys files. I think this is even a default for some configuration as of lately.

Mar 2 2021, 12:03 AM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)

Mar 1 2021

Sergey.Trofimovsky.SF added a comment to T275564: Provide Speed & Function rough numbers for our current Gerrit web traffic.

Thank you, guys! That should be enough to start working on load tests, I'm sure we'll come up with more follow up questions on the way.

Mar 1 2021, 11:37 PM · User-brennen, Gerrit, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T276148: SSH Access of Git data in GitLab.

Given the ratio between number of people that will be using git SSH access, and people that will log in to manage the system, I would spend an extra minute here to try and stick with the default SSH port for git. Here's a few options I believe may balance out convenience and security concerns:

Mar 1 2021, 11:13 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
Sergey.Trofimovsky.SF added a comment to T275722: Requesting access to gitlab1001 / gitlab1002 for Sergey Trofimovsky from Speed & Function.

This is the confirmation that the L3 document is signed. You signed this document on Fri, Feb 26, 7:18 PM.

Mar 1 2021, 5:44 PM · SRE, SRE-Access-Requests

Feb 26 2021

Sergey.Trofimovsky.SF added a comment to T274460: Shell access for VMs for Speed & Function contractors.

Trying to un-mess things with registering on the S&F email now, here's my info:

Feb 26 2021, 7:15 PM · GitLab (Initialization)