gerritro user getting access denied from gerrit1002
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Marostegui
	Jan 27 2020, 11:09 PM

Description

While investigating issues with gerrit we found this on the logs:

er.java:61)\n\tat Main.main(Main.java:24)\nCaused by: java.sql.SQLException: Access denied for user 'gerritro'@'10.64.32.156' (using password: YES)\n\tat com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)\n\tat com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)\n\tat com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)\n\tat com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919)\n\tat com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694)\n\tat com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1244)\n\tat com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397)\n\tat com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)\n\tat com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)\n\tat com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)\n\tat com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n\tat com.mysql.jdbc.Util.handleNewInstance(Util.java:411)\n\tat com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)\n\tat com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)\n\tat com.google.gwtorm.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:104)\n\tat com.google.gwtorm.jdbc.Database.<init>(Database.java:74)\n\t... 34 more\n\n3 errors\n\tat com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)\n\tat com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)\n\tat com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)\n\tat com.google.inject.Guice.createInjector(Guice.java:87)\n\tat com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:188)\n\t... 11 more","exception_class":"com.google.gerrit.common.Die","exception_message":"Unable to determine SqlDialect\n  caused by java.sql.SQLException: Access denied for user 'gerritro'@'10.64.32.156' (using password: YES)"},"source_host":"gerrit1002","method":"run","level":"ERROR","message":"Unable to start daemon","mdc":{},"@timestamp":"2020-01-27T22:57:52.011Z","file":"Daemon.java","line_number":"294","thread_name":"main","@version":1,"logger_name":"com.google.gerrit.pgm.Daemon","class":"com.google.gerrit.pgm.Daemon"}

The gerritro is a test user created at T239151 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563302/

My guess is that it is using the wrong password on the gerrit side?
I have been able to use the password on the pw repo and it works:

root@cumin1001:~# mysql --skip-ssl -ugerritro -p -hdbproxy1007
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 88465738
Server version: 10.1.39-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

gerritro@dbproxy1007[(none)]> use reviewdb
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
gerritro@dbproxy1007[reviewdb]> show tables;
+-----------------------------+
| Tables_in_reviewdb          |
+-----------------------------+
| account_external_ids        |
| account_group_by_id         |
| account_group_by_id_aud     |
| account_group_id            |
| account_group_members       |
| account_group_members_audit |
| account_group_names         |
| account_groups              |
| account_id                  |
| accounts                    |
| change_id                   |
| change_messages             |
| changes                     |
| patch_comments              |
| patch_set_approvals         |
| patch_sets                  |
| schema_version              |
| system_config               |
+-----------------------------+
18 rows in set (0.00 sec)

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+5 -1	gerrit: make db_pass configurable from (private) Hiera
	operations/puppet	production	+3 -1	gerrit: for db_user, remove default and replace hiera() with lookup()

Customize query in gerrit

Related Objects

Mentioned In: T254516: Get a writable reviewdb clone to test Gerrit upgrade with
T239151: Gerrit VM to test data migration
Mentioned Here: T239151: Gerrit VM to test data migration

Event Timeline

• Marostegui created this task.Jan 27 2020, 11:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2020, 11:09 PM

• Marostegui renamed this task from gerritro user getting access denied from dbproxy1007 to gerritro user getting access denied from gerrit1002.Jan 27 2020, 11:10 PM

• Marostegui updated the task description. (Show Details)

I think this needs updating https://github.com/wikimedia/puppet/blob/production/modules/gerrit/manifests/jetty.pp#L44 (unless the pass is the same for the *ro user).

The password for the ro user is different from that one (on the pw repo, which is the one I used).
I tried the one on https://github.com/wikimedia/puppet/blob/production/modules/gerrit/manifests/jetty.pp#L44 and that is for the gerrit user, which doesn't work for gerritro user (which is expected).
So I assume we need to change, somewhere, the fact that gerritro needs to use its password, and not gerrit's user one.

Couldn't we add the pass here https://github.com/wikimedia/labs-private/tree/master/hieradata/hosts for gerrit1002?

Dzahn claimed this task.Jan 28 2020, 1:52 AM

This user was not created in the linked ticket, it was pre-existing and we are just trying to use it.

In T243800#5835375, @Dzahn wrote:

This user was not created in the linked ticket, it was pre-existing and we are just trying to use it.

can you double check the password it uses?

In T243800#5835217, @Paladox wrote:

Couldn't we add the pass here https://github.com/wikimedia/labs-private/tree/master/hieradata/hosts for gerrit1002?

That won't work since this is a production ganeti VM and labs-private is only used on wmcs cloud instances.

Private config including the db_pass is not in (private) Hiera unfortunately. It is in the passwords module and it's not a class parameter either yet.

We should move that to Hiera and then Paladox' suggestion to override it in private/hieradata/hosts can work. Requires a change to puppet code though and touches prod.

Will upload a fix for that later.

In T243800#5835376, @Marostegui wrote:

can you double check the password it uses?

Turns out the password is hardcoded to the one for the rw user. ^

Dzahn triaged this task as Medium priority.Jan 28 2020, 2:39 AM

Change 570976 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: make db_pass configurable from (private) Hiera

https://gerrit.wikimedia.org/r/570976

gerritbot added a project: Patch-For-Review.Feb 8 2020, 1:28 AM

Change 571567 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: for db_user, remove default and replace hiera() with lookup()

https://gerrit.wikimedia.org/r/571567

Change 571567 merged by Dzahn:
[operations/puppet@production] gerrit: for db_user, remove default and replace hiera() with lookup()

https://gerrit.wikimedia.org/r/571567

Change 570976 merged by Dzahn:
[operations/puppet@production] gerrit: make db_pass configurable from (private) Hiera

https://gerrit.wikimedia.org/r/570976

@Marostegui I made some changes to make the db_user and db_pass configurable for gerrit. Thing is just i don't know the clear text version of the hashed password for 'gerritro'. I took a look at the relevant m2-master behind dbproxies, db1132 and i see

MariaDB [mysql]> select host from user where user='gerritro';
+--------------+
| host         |
+--------------+
| %            |
| 10.192.16.9  |
| 10.64.0.135  |
| 10.64.0.166  |
| 10.64.32.156 |
+--------------+

Those are dbproxy1002, dbproxy1007, dbproxy2002 and dbproxy1013.

Could you alter the password hash to something different and let me know it?

Or is it ok if i make these myself on db1132 directly?

Also i suggest we remove the "@%" user while at it and keep just the entries for coming from dbproxies.

In T243800#5874562, @Dzahn wrote:

@Marostegui I made some changes to make the db_user and db_pass configurable for gerrit. Thing is just i don't know the clear text version of the hashed password for 'gerritro'. I took a look at the relevant m2-master behind dbproxies, db1132 and i see

The gerritro user password is on the pwstore repo, in the gerritfile.

root@cumin1001:~# mysql --skip-ssl -u gerritro -p -hdbproxy1007
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 94850485
Server version: 10.1.39-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

gerritro@dbproxy1007[(none)]> Ctrl-C -- exit!

Prio High now because work on the upgrade will start again.

After talking with Manuel i realized finally the ro password was in pwstore, not in the passwords module in the private puppet repo.

The one from the pwstore was imported from /srv/passwords on iron in 2015 ... and it works.

I then took that password and added it to /srv/private/hieradata/hosts/gerrit1002.yaml in _private_ Hiera on the puppetmaster.

Thanks to our previous changes to the gerrit class this meant running puppet on gerrit1002 made it change the db_pass used by gerrit to the readonly password while gerrit1001 stayed unchanged (hieradata/role/common/gerrit.yaml overrides hieradata/hosts/gerrit1002.yaml and has the regular rw password).

After that puppet run i could confirm the ro password is now in /var/lib/gerrit2/review_site/etc/secure.config on gerrit1002 (cc: @QChris )

" Access denied for user 'gerritro'@'10.64.32.156'" should be GONE now.

Mentioned in SAL (#wikimedia-operations) [2020-05-28T09:35:22Z] <mutante> restarting gerrit on gerrit1002 after fixing db_pass to the readonly one (T243800)

Dzahn closed this task as Resolved.May 28 2020, 9:52 AM

Dzahn mentioned this in T239151: Gerrit VM to test data migration.

Dzahn mentioned this in T254516: Get a writable reviewdb clone to test Gerrit upgrade with.Jun 8 2020, 9:42 AM

gerritro user getting access denied from gerrit1002Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

gerritro user getting access denied from gerrit1002
Closed, ResolvedPublic
Actions