John Bond, one of our SREs is interested in reviewing the procedure for the installation of GitLab. He has prior GitLab CE/Ee experience and will be able to see if there any obvious areas that GitLab might interfere with our base puppet install. Is there a repo/document that he can review?
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T274458 Remove Speed & Function blockers for GitLab work | |||
| Resolved | Sergey.Trofimovsky.SF | T279545 Gitlab Installation Procedure |
Event Timeline
Comment Actions
As the Repository can be accessed by the emails, I've sent the invitation to @jbond, so he should be able to get the installation repository.
Comment Actions
@Eugene.chernov thanks, i have recived the invite to review the repository however i am unable to login to the gitlab portal as i don't know my password and there dosn't seem to be a password reset function on https://gitlab.gluzdov.com/users/sign_in
Comment Actions
hi @jbond , you should've received the email with the further instructions. Please let me know if you haven't
Comment Actions
Also, any feedback is welcome and expected, please let us know. Thanks!
Thanks, i took a quick look through the main gitlab_server role and from what i can see its mostly just the stock ansible role from gitlab so so not a whole lot of feedback just yet, i think the ssh config will be the thing im most intrested in.
That said i did make a few notes. currently the ansible job downloads and runs a bash script as the ansible user (guessing root). All the script dose is configure the gitlab Debian repository so one can install the gitlab-omniauth packages.
For the productions hosts i would just configure this repo via puppet and then we should prevent theses bits running in production.
The other issue i noticed is you install the postfix as a gitlab dependency. Our current puppet policy installs and configures exim as the default MTA and installing postfix could cause issues. All the other dependencies listed in this block are installed by default as such i think you could just drop the task to install dependencies. further gnupg is also installed so you don't need to run that bit on production either
Comment Actions
Change 679328 had a related patch set uploaded (by Jbond; author: John Bond):
[operations/puppet@production] C:aptrepo: add gitlab repo mirror
Comment Actions
Change 679328 merged by Jbond:
[operations/puppet@production] C:aptrepo: add gitlab repo mirror
Comment Actions
Change 684418 had a related patch set uploaded (by Jbond; author: John Bond):
[operations/puppet@production] P:gitlab: install gitlab-ce
Comment Actions
Change 684418 merged by Jbond:
[operations/puppet@production] P:gitlab: install gitlab-ce
Comment Actions
First attempt at running install-gitlab-server.sh from gitlab-ansible:
PLAY [Install Gitlab Server] **********************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Include OS-specific variables] **********************************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Check if GitLab configuration file already exists] **************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Check if GitLab is already installed] ***************************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Gather package facts] *******************************************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab dependencies] ************************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab dependencies (Debian)] ***************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Check GitLab dependencies] **************************************************************************************
skipping: [gitlab-server-prod] => (item=openssh-server)
skipping: [gitlab-server-prod] => (item=curl)
skipping: [gitlab-server-prod] => (item=openssl)
skipping: [gitlab-server-prod] => (item=tzdata)
TASK [gitlab_server : Check GitLab dependencies (Debian)] *****************************************************************************
skipping: [gitlab-server-prod] => (item=gnupg)
TASK [gitlab_server : Download GitLab repository installation script] *****************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab repository] **************************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Define the Gitlab package name] *********************************************************************************
ok: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab] *************************************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Reconfigure GitLab (first run)] *********************************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Create GitLab SSL configuration folder] *************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Create self-signed certificate] *********************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Copy GitLab configuration file] *********************************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Create GitLab crontab] ******************************************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab sshd - create config file] ***********************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab sshd - create moduli file] ***********************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab sshd - create host keys] *************************************************************************
changed: [gitlab-server-prod] => (item=/etc/ssh-gitlab/ssh_host_rsa_key)
changed: [gitlab-server-prod] => (item=/etc/ssh-gitlab/ssh_host_ecdsa_key)
changed: [gitlab-server-prod] => (item=/etc/ssh-gitlab/ssh_host_ed25519_key)
TASK [gitlab_server : Install GitLab sshd - create service] ***************************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Install GitLab sshd - enable service] ***************************************************************************
changed: [gitlab-server-prod]
TASK [gitlab_server : Remove GitLab sshd - stop service] ******************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Remove GitLab sshd - remove config file] ************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Remove GitLab sshd - remove config directory] *******************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Remove GitLab sshd - remove service] ****************************************************************************
skipping: [gitlab-server-prod]
TASK [gitlab_server : Remove GitLab sshd - reload systemd] ****************************************************************************
skipping: [gitlab-server-prod]
RUNNING HANDLER [gitlab_server : restart gitlab] **************************************************************************************
changed: [gitlab-server-prod]
PLAY RECAP ****************************************************************************************************************************
gitlab-server-prod : ok=16 changed=10 unreachable=0 failed=0 skipped=14 rescued=0 ignored=0Looks like it succeeded, there's a running GitLab at https://gitlab.wikimedia.org - prompts for admin pass, as typical with omnibus install, but gives a CAS error on attempting to set:
I'm assuming this just means the application needs registered with CAS...
Comment Actions
Considering that installation is in an unfinished state now (especially for the default password reset part, and then UI configurations) the safe way would be to shut Gitlab down with gitlab-ctl until it can be finished. However, password can be reset with CLI (sudo gitlab-rake "gitlab:password:reset, https://docs.gitlab.com/ee/security/reset_user_password.html) and installation left alone until tomorrow.
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2021-05-26T17:37:42Z] <brennen> gitlab1001: reset admin password and ran gitlab-ctl stop (T279545)
Comment Actions
Change 696015 had a related patch set uploaded (by Jbond; author: John Bond):
[operations/puppet@production] idp: add gitlab to production idp
Comment Actions
Change 696015 merged by Jbond:
[operations/puppet@production] idp: add gitlab to production idp
Comment Actions
Change 696024 had a related patch set uploaded (by Jbond; author: John Bond):
[operations/puppet@production] P:gitlab: open SSH port to the world
Comment Actions
First attempt at running install-gitlab-server.sh from gitlab-ansible:
It would be useful if you could run this and then straight after run puppet agent -t puppet should hopefully run without and corrections or changes.
I'm assuming this just means the application needs registered with CAS...
Done for now i have enabled access for users in the ops, wmf and nda ldap group
the safe way would be to shut Gitlab down
In a similar vein to this the ssh port is currently still firewalled, however the CR is drafted and ready to go
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2021-05-27T16:05:17Z] <brennen> gitlab1001: re-running ansible and puppet per T279545
Comment Actions
It would be useful if you could run this and then straight after run puppet agent -t puppet should hopefully run without and corrections or changes.
Noting that after a puppet agent -t run, re-running the Ansible does give:
TASK [gitlab_server : Install GitLab sshd - enable service] **************************************************************************************************changed: [gitlab-server-prod]
Not clear that it's actually changing any state, though.
Comment Actions
One other auth hurdle here: S&F folks need to be able to login. Think it'd be appropriate to add them to NDA, if we don't want to open up access to anyone with a wikitech account just yet, or alternatively define an LDAP group specifically for them?
We're planning to run a load test against this instance, and then will reset contents of the GitLab instance since that creates a slew of projects.
Planning to turn off nginx over the long weekend, and otherwise leave GitLab up.
Comment Actions
all the S&F people are covered by NDA so happy to add them to the NDA ldap group, can you list all the id's
Comment Actions
all the S&F people are covered by NDA so happy to add them to the NDA ldap group, can you list all the id's
Based on who's in gitlab-roots, I think:
- eugene-chernov
- strofimovsky01
- il
Ought to cover it.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2021-05-27T20:37:43Z] <jbond> add eugene-chernov, strofimovsky01, il to ldap nda #T279545
Comment Actions
have added these to the nda group as @KFrancis already confirmed NDA status for all of theses in the access requests so no issue (of course if im corrected access will be removed and we will correct)
Comment Actions
Change 696024 merged by Jbond:
[operations/puppet@production] P:gitlab: open SSH port to the world
Comment Actions
In a similar vein to this the ssh port is currently still firewalled, however the CR is drafted and ready to go
I have now opened the ssh port, however ssh is still not working as the git user is disabled
Jun 4 09:15:06 gitlab1001 sshd[20266]: User git not allowed because account is locked
$ getent shadow git git:!:18773::::::
Comment Actions
I have now opened the ssh port, however ssh is still not working as the git user is disabled
Thanks, John, we'll verify and push the changes. It seems like one of the differences between the dev instance and production that we didn't have a chance to verify until actual release.
Comment Actions
Also, guys, can you verify that the manual UI configurations were done after running Ansible?
Gitlab keeps warning about open registration still being enabled, which was #1 on the list.
Should we reactive a discussion about a possible automation for this type of configurations with Gitlab API?
Comment Actions
Also, guys, can you verify that the manual UI configurations were done after running Ansible?
My bad. Ran through the list - handful of things were in place, but I hadn't done the overall set.
Should we reactive a discussion about a possible automation for this type of configurations with Gitlab API?
A very good idea, but I think we can figure that one out later - the settings API looks easy enough to work with. We probably don't want to bake that into the Puppet review loop, since settings admin will require some pretty fast turnaround and experimentation, but we can probably set up a wikimedia-gitlab-settings repo or some such with a script and use that for any changes. I wrote up T284336 for that.
For the record here: Planning to run K6 tests and then reset instance data.
Comment Actions
GitLab is installed, we have a recurring procedure for it. Calling this one finished, although T283076 is the obvious next thing.