Page MenuHomePhabricator
Create Task
Maniphest T332208

[tbs] Harbor broken in toolsbeta because of missing db password
Closed, ResolvedPublic
Actions

Description

While working on T316323 I broke the toolsbeta Harbor deployment at https://harbor.toolsbeta.wmflabs.org

Harbor is currently failing to connect to its dedicated Trove database

What I think has happened is that while I was doing some tests in the Toolsbeta instance (toolsbeta-harbor-1.toolsbeta.eqiad1.wikimedia.cloud) I ran /srv/ops/harbor/prepare and that overwrote the config files in /srv/ops/harbor/common/config/db which contained the correct password, with the value coming from /etc/puppet/private in toolsbeta-puppetmaster-04 (profile::toolforge::harbor::db_harbor_pwd) which is currently dummypass but does not seem to be the current password for the harbor user in that database.

My assumption is that the password was set manually in /srv/ops/harbor/harbor.yml, and then copied to /srv/ops/harbor/common/config/db the last time that /srv/ops/harbor/prepare was run. Puppet than overwrote the value in /srv/ops/harbor/harbor.yml with the one coming from /etc/puppet/private, but since prepare was not run until today, the value in /srv/ops/harbor/common/config/db was not modified until today.

Unless that password was also stored somewhere else or someone can remember it, it must be reset in the Trove db (which is also not easy because I don't know any credentials for that db).

The trove DB is named ttg4ncgzifw.svc.trove.eqiad1.wikimedia.cloud and was created in T316232.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLucasWerkmeisterT320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedmatmarexT319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
 OpendcaroT194332 [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
 ResolveddcaroT267374 [tbs.beta] Create a toolforge build service beta release
 ResolvedRaymond_NdibeT324827 tbs: user-story 3 - I want to cancel the build using the toolforge build cancel feature
 ResolvedRaymond_NdibeT324828 tbs: user-story 3 - cancel cli should give the right error when the service is down
 ResolvedRaymond_NdibeT331409 [tbs.cli] The cancel command should return a message saying that the build is already finished when trying to cancel a finished build
 ResolvedRaymond_NdibeT331410 [tbs.cli] The cancel command should show a useful message when the connection to tekton works but returns http error (requests.exceptions.HTTPError)
 ResolvedNoneT332294 [tbs][cli] build cancel --help lists the -v option twice
 ResolveddcaroT324833 tbs: user-story 5 - I want to check the status of a given build using the toolforge build show feature.
 ResolveddcaroT324830 tbs: user-story 4 - I want to check the status of the last build using the toolforge build show feature
 ResolvedRaymond_NdibeT324831 tbs: user-story 4 - show cli should give the right error when the service is down
 ResolveddcaroT324823 tbs: user-story 2 - Feature: I can start a new build from a git url
 ResolvedRaymond_NdibeT324832 tbs: user-story 4 - show cli should use the latest build if no build id is passed
 ResolveddcaroT324834 tbs: user-story 6 - I want to start a webservice with the image I built earlier.
 ResolvedRaymond_NdibeT316327 [tbs] Deploy on tools
 ResolvedRaymond_NdibeT332347 Duplicate the maintain-harbor deployment for "tools"
 ResolvedfnegriT316323 [tbs.harbor.tools] Replicate toolsbeta harbor deployment
 ResolvedfnegriT332208 [tbs] Harbor broken in toolsbeta because of missing db password

Event Timeline

fnegri created this task.Mar 15 2023, 6:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2023, 6:23 PM
fnegri changed the task status from Open to In Progress.Mar 15 2023, 6:24 PM
fnegri triaged this task as High priority.
fnegri moved this task from Next Up to In Progress on the Toolforge Build Service (Focus Week!) board.
fnegri updated the task description. (Show Details)Mar 15 2023, 6:33 PM
dcaro added a comment.Mar 16 2023, 1:52 PM

This has been manually reset (using the root account given by trove through the ui in horizon database tab-> instanes -> manage root access)

fnegri closed this task as Resolved.Mar 16 2023, 5:05 PM

To clarify, the "root account" has been used only to access the db with psql and modify the password of the harbor user:

harbor=# ALTER USER harbor WITH PASSWORD 'xx';

The new password has been stored in the private Git repo: /etc/puppet/private/hieradata/labs.yaml in toolsbeta-puppetmaster-04 (profile::toolforge::harbor::db_harbor_pwd).

https://harbor.toolsbeta.wmflabs.org is up and running again.

Raymond_Ndibe moved this task from In Progress to Done on the Toolforge Build Service (Focus Week!) board.Mar 16 2023, 7:21 PM
dcaro moved this task from Backlog to Done on the cloud-services-team (FY2022/2023-Q3) board.Apr 11 2023, 9:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits