Maniphest T202549

cloudvps: eqiad1: move nova db to m5-master
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Aug 22 2018, 3:37 PM

Description

Another step forward to having eqiad1 in production is to have the nova and nova-api databases in a production db server (m5-master).
Since the nova names are already in use by the old main deployment, we should pick another name, like nova-eqiad1/nova-api-eqiad1 or something like that.

Steps:

leave room for more nova connections
review connectivity (should simply work after previous database migrations)
m5-master DB GRANTS https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455114/
prepare puppet patch to refresh nova DB references for eqiad1 (server + db name)
db backup-copy-import
sync nova DB pass for main/eqiad1 (labs/private, etc)
merge puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454774/
Review m5 database backups so everything important is backed up

This task is for tracking such work.

For the record:

2 new DBs:

nova_eqiad1
nova_api_eqiad1

From:

cloudcontrol1003.wikimedia.org / 208.80.154.23
cloudcontrol1004.wikimedia.org / 208.80.154.132

I think the GRANTs would look like:

GRANT ALL PRIVILEGES ON nova_eqiad1.* TO 'nova'@'208.80.154.23' IDENTIFIED BY '<%= @nova_dbpass %>';
GRANT ALL PRIVILEGES ON nova_api_eqiad1.* TO 'nova'@'208.80.154.23' IDENTIFIED BY '<%= @nova_dbpass %>';
GRANT ALL PRIVILEGES ON nova_eqiad1.* TO 'nova'@'208.80.154.132' IDENTIFIED BY '<%= @nova_dbpass %>';
GRANT ALL PRIVILEGES ON nova_api_eqiad1.* TO 'nova'@'208.80.154.132' IDENTIFIED BY '<%= @nova_dbpass %>';

Password is hieradata/eqiad/profile/openstack/eqiad1/nova.yaml profile::openstack::eqiad1::nova::db_pass in the private repo.
(we can use the same as in hieradata/eqiad/profile/openstack/main/nova.yaml profile::openstack::main::nova::db_pass)

Details

Subject	Repo	Branch	Lines +/-
cloudvps: eqiad1: move nova DBs to m5-master	operations/puppet	production	+17 -4
production-m5.sql.erb: Add GRANTS for nova user	operations/puppet	production	+6 -0
nova: scale back the database and worker usage for cloudcontrol1003	operations/puppet	production	+3 -3

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Open	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T196662 Set up LVS in beta like prod
Resolved	bd808	T166396 Program 1 Outcome 4: VPS hosting
Resolved	None	T167293 Nova-network to Neutron migration
Resolved	aborrero	T202549 cloudvps: eqiad1: move nova db to m5-master

Event Timeline

aborrero triaged this task as Medium priority.Aug 22 2018, 3:37 PM

aborrero created this task.

aborrero edited projects, added DBA; removed Epic.Aug 23 2018, 9:08 AM

For the record, these are the local databases in cloudcontrol1003.wikimedia.org before this task:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| nova               |
| nova_api           |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)

So we can just drop the mysql packages while at it.

Some questions to try to get a picture of what are the implications:

How big are those databases?
How many concurrent connections they have? Are those throttled? (reminder: T188589)
Can those applications support downtime? As you'll probably need to stop them to avoid writes while the migration happens.

Change 454774 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: eqiad1: move nova DBs to m5-master

https://gerrit.wikimedia.org/r/454774

gerritbot added a project: Patch-For-Review.Aug 23 2018, 9:27 AM

In T202549#4526012, @Marostegui wrote:

Some questions to try to get a picture of what are the implications:

How big are those databases?
How many concurrent connections they have? Are those throttled? (reminder: T188589)
Can those applications support downtime? As you'll probably need to stop them to avoid writes while the migration happens.

The DBs are the nova_eqiad1 and nova_api_eqiad1 ones which have been bootstrapped but have no actual usage right now (i.e, they aren't used by anything in production).
Regarding the concurrent connections, this setup is similar in that regard to the other main deployment. The fixes that @Bstorm applied to other day to connection pooling and usage should be valid to this DB as well.

So, yes, this support downtime (I can simply stop the nova services in the eqiad1 deployment while moving the DB).

In T202549#4526124, @aborrero wrote:

The DBs are the nova_eqiad1 and nova_api_eqiad1 ones which have been bootstrapped but have no actual usage right now (i.e, they aren't used by anything in production).

Ah, good, so we don't have to actually migrate any data, right?
Do you have some estimations about how much they can grow?

Regarding the concurrent connections, this setup is similar in that regard to the other main deployment. The fixes that @Bstorm applied to other day to connection pooling and usage should be valid to this DB as well.

So how many connections are we talking about in total here?
I am asking because right now we have 315 connections in total on m5 master (nova still using 137) and the max_connections is set to 500, so if we add new stuff we need to calculate if we have to increase that 500.

So, yes, this support downtime (I can simply stop the nova services in the eqiad1 deployment while moving the DB).

Good!

In T202549#4526186, @Marostegui wrote:

In T202549#4526124, @aborrero wrote:

The DBs are the nova_eqiad1 and nova_api_eqiad1 ones which have been bootstrapped but have no actual usage right now (i.e, they aren't used by anything in production).

Ah, good, so we don't have to actually migrate any data, right?
Do you have some estimations about how much they can grow?

Well, there is some data, but just basic. A couple of services registered and a couple of test VMs, but nothing more right now.

Check these mysqldump files:

-rw-r--r-- 1 aborrero wikidev 212K Aug 23 09:54 nova_api.db
-rw-r--r-- 1 aborrero wikidev 478K Aug 23 09:54 nova.db

They can grow more or less the same as the current nova/nova-api databases in m5-master.

Regarding the concurrent connections, this setup is similar in that regard to the other main deployment. The fixes that @Bstorm applied to other day to connection pooling and usage should be valid to this DB as well.

So how many connections are we talking about in total here?
I am asking because right now we have 315 connections in total on m5 master (nova still using 137) and the max_connections is set to 500, so if we add new stuff we need to calculate if we have to increase that 500.

We should perhaps do both: you increase limits in your side and we limit usage in ours.

In T202549#4526214, @aborrero wrote:
Well, there is some data, but just basic. A couple of services registered and a couple of test VMs, but nothing more right now.

Check these mysqldump files:
-rw-r--r-- 1 aborrero wikidev 212K Aug 23 09:54 nova_api.db
-rw-r--r-- 1 aborrero wikidev 478K Aug 23 09:54 nova.db
They can grow more or less the same as the current nova/nova-api databases in m5-master.

Cool - just for the record:

12G	nova
258M	nova_api

Regarding the concurrent connections, this setup is similar in that regard to the other main deployment. The fixes that @Bstorm applied to other day to connection pooling and usage should be valid to this DB as well.

So how many connections are we talking about in total here?
I am asking because right now we have 315 connections in total on m5 master (nova still using 137) and the max_connections is set to 500, so if we add new stuff we need to calculate if we have to increase that 500.

We should perhaps do both: you increase limits in your side and we limit usage in ours.

That is probably likely to happen but I don't want to increase the number of connections just to an arbitrary number as there are more services on that host, and if for whatever reason we overload the server to a point where it becomes unresponsinve (frozen) because of the amount of load, that'll be worse than throwing "too many connections" but not losing the server.

It would be good if you could work out the math to see how many connections will be allowed in total from these services + the other nova ones we currently have on m5 so we have a better picture.

Thanks!

Backup configuration for m5 should be reviewed too.

In T202549#4526260, @jcrespo wrote:

Backup configuration for m5 should be reviewed too.

What do you mean? Are we doing backups of this server, right?

Do you have some information -or docs- for me to read on how we deal with DB backups (and restores) and, specifically, what we need to do in case of a DB failure and restore required?

In T202549#4526283, @aborrero wrote:

In T202549#4526260, @jcrespo wrote:

Backup configuration for m5 should be reviewed too.

What do you mean? Are we doing backups of this server, right?

He probably means we have to add them to: https://github.com/wikimedia/puppet/blob/production/modules/role/templates/mariadb/backups/dumps-misc.sh.erb#L24-L25

Do you have some information -or docs- for me to read on how we deal with DB backups (and restores) and, specifically, what we need to do in case of a DB failure and restore required?

We have this for now: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Current_status

Forget that last part of dumps-misc that is not used anymore :-)

In T202549#4526217, @Marostegui wrote:

It would be good if you could work out the math to see how many connections will be allowed in total from these services + the other nova ones we currently have on m5 so we have a better picture.

Checking the local DB, this is what we have with our current config:

mysql> select user, host, count(*) FROM information_schema.processlist GROUP BY USER, HOST;
+------+--------------------------------------+----------+
| user | host                                 | count(*) |
+------+--------------------------------------+----------+
| nova | cloudcontrol1003.wikimedia.org:35962 |        1 |
| nova | cloudcontrol1003.wikimedia.org:35964 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36724 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36726 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36848 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36850 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36860 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36868 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36872 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36914 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36916 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36926 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36948 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36956 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36960 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37008 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37014 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37016 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37018 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37022 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37038 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37046 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37048 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37094 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37096 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37110 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37150 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37152 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37158 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37192 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37252 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37266 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37268 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37288 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37290 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37326 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37366 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37410 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37430 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37436 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37438 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37440 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37442 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37460 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37474 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37478 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37480 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37490 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37500 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37502 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37518 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37530 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37542 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37548 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37614 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37616 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37656 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37666 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37668 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37676 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37678 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37680 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37682 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37684 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37686 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37688 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37692 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37694 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37696 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37698 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37716 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37752 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37796 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37798 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37818 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37872 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37878 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37894 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37906 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37908 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37940 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37942 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37954 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37956 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37960 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37976 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37986 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37996 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37998 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38054 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38074 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38086 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38088 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38266 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38274 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38276 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38278 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38280 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38332 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38334 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38336 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38338 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38344 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38350 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38378 |        1 |
| root | localhost                            |        1 |
+------+--------------------------------------+----------+
106 rows in set (0.00 sec)

So, about ~100 connections.

In T202549#4526309, @aborrero wrote:

In T202549#4526217, @Marostegui wrote:

It would be good if you could work out the math to see how many connections will be allowed in total from these services + the other nova ones we currently have on m5 so we have a better picture.

Checking the local DB, this is what we have with our current config:

mysql> select user, host, count(*) FROM information_schema.processlist GROUP BY USER, HOST;
+------+--------------------------------------+----------+
| user | host                                 | count(*) |
+------+--------------------------------------+----------+
| nova | cloudcontrol1003.wikimedia.org:35962 |        1 |
| nova | cloudcontrol1003.wikimedia.org:35964 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36724 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36726 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36848 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36850 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36860 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36868 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36872 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36914 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36916 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36926 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36948 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36956 |        1 |
| nova | cloudcontrol1003.wikimedia.org:36960 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37008 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37014 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37016 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37018 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37022 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37038 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37046 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37048 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37094 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37096 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37110 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37150 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37152 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37158 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37192 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37252 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37266 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37268 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37288 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37290 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37326 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37366 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37410 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37430 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37436 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37438 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37440 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37442 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37460 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37474 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37478 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37480 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37490 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37500 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37502 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37518 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37530 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37542 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37548 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37614 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37616 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37656 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37666 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37668 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37676 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37678 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37680 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37682 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37684 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37686 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37688 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37692 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37694 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37696 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37698 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37716 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37752 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37796 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37798 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37818 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37872 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37878 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37894 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37906 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37908 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37940 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37942 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37954 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37956 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37960 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37976 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37986 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37996 |        1 |
| nova | cloudcontrol1003.wikimedia.org:37998 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38054 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38074 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38086 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38088 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38266 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38274 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38276 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38278 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38280 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38332 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38334 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38336 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38338 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38344 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38350 |        1 |
| nova | cloudcontrol1003.wikimedia.org:38378 |        1 |
| root | localhost                            |        1 |
+------+--------------------------------------+----------+
106 rows in set (0.00 sec)

So, about ~100 connections.

Is that throttled already?

In T202549#4526312, @Marostegui wrote:

Is that throttled already?

Apparently. @Bstorm applied same config to both deployments (main, eqiad1).

Yes, but there is stuff that has not been properly throttled yet: T188589#4523613

aborrero updated the task description. (Show Details)Aug 23 2018, 4:33 PM

There are definitely more workers on there, and larger pools than are on the labcontrol1001/labnet1001 hosts right now. Since this environment is queried by most things almost as often as the old one, despite it having almost no actual VMs under its control, it should be expected to have a similar performance profile with a very DB small size.

Right now, it has 40 total workers and pool sizes of 10 each. That means the 100+ connections can easily be scaled back (since if this came under heavy enough load, it would create far more connections, and likely doesn't need what it has). I can set it to only have 20 workers with pools of 4, making it slightly smaller than the current arrangement that connects to m5.

Change 454897 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] nova: scale back the database and worker usage for cloudcontrol1003

https://gerrit.wikimedia.org/r/454897

If we feel good merging that, and the local db connections seem appropriate after it is in place (they really shouldn't go to 100 right now), it should be less destructive to put on m5.

Change 454897 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] nova: scale back the database and worker usage for cloudcontrol1003

https://gerrit.wikimedia.org/r/454897

After I merged the patch written by @Bstorm this is what I get in cloudcontrol1003.wikimedia.org local DB:

mysql> select user, host, db, count(*) FROM information_schema.processlist GROUP BY USER, HOST;
+------+--------------------------------------+------+----------+
| user | host                                 | db   | count(*) |
+------+--------------------------------------+------+----------+
| nova | cloudcontrol1003.wikimedia.org:39616 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39618 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39620 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39622 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39624 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39626 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39628 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39630 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39692 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39694 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39698 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39700 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39702 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39704 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39706 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39708 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39710 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39712 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39714 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39716 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39718 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39720 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39722 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39724 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39726 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39728 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39730 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39732 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39734 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39736 | nova |        1 |
| nova | cloudcontrol1003.wikimedia.org:39808 | nova |        1 |
| root | localhost                            | NULL |        1 |
+------+--------------------------------------+------+----------+
32 rows in set (0.01 sec)

So, this moved us from ~100 connections to ~30 connections.

Are we ready to move to m5-master these DBs?

Thanks for the update! That is a really good improvement.
I would love to know if we are done with T188589 or what is pending there and avoid mixing things so we can isolate problems (if they arise)

In T202549#4529173, @Marostegui wrote:

I would love to know if we are done with T188589 or what is pending there and avoid mixing things so we can isolate problems (if they arise)

We should be done, I consider the other ticket a different issue, more of a long term thing.

In T202549#4529192, @aborrero wrote:

In T202549#4529173, @Marostegui wrote:

I would love to know if we are done with T188589 or what is pending there and avoid mixing things so we can isolate problems (if they arise)

We should be done, I consider the other ticket a different issue, more of a long term thing.

It is not entirely clear to me (as I don't really have the whole picture of your services) whether we have applied all the throttling measures or not. If everything is done from your side, I would appreciate to close the task and not mix it with future plans.
From my side, MySQL is now fine and using 272 connections out of 500 so we could close that task.

That task was created for an specific problem, we can discuss the other long-term stuff in a different one, to avoid losing/mixing things on a long task which can be hard to follow.

My reasoning of not proceeding with this task until T188589 was completely solved is because I don't want to be touching two very related things at the same time, because in case of problems we can get confused of what caused them and what we'd need to revert :-)

Marostegui moved this task from Triage to In progress on the DBA board.Aug 24 2018, 7:50 AM

So, what about the GRANTS
What is it needed? And from which IPs?

In T202549#4529302, @Marostegui wrote:

So, what about the GRANTS
What is it needed? And from which IPs?

2 new DBs:

nova-eqiad1
nova-api-eqiad1

From:

cloudcontrol1003.wikimedia.org
cloudcontrol1004.wikimedia.org

I think the GRANTs would look like:

GRANT ALL PRIVILEGES ON nova-eqiad1.* TO 'nova'@'cloudcontrol1003.wikimedia.org' IDENTIFIED BY 'xxxxxxxxxx';
GRANT ALL PRIVILEGES ON nova-api-eqiad1.* TO 'nova'@'cloudcontrol1003.wikimedia.org' IDENTIFIED BY 'xxxxxxxxxx';
GRANT ALL PRIVILEGES ON nova-eqiad1.* TO 'nova'@'cloudcontrol1004.wikimedia.org' IDENTIFIED BY 'xxxxxxxxxx';
GRANT ALL PRIVILEGES ON nova-api-eqiad1.* TO 'nova'@'cloudcontrol1004.wikimedia.org' IDENTIFIED BY 'xxxxxxxxxx';

Password is hieradata/eqiad/profile/openstack/eqiad1/nova.yaml profile::openstack::eqiad1::nova::db_pass in the private repo.
(we can use the same as in hieradata/eqiad/profile/openstack/main/nova.yaml profile::openstack::main::nova::db_pass)

For the future, this information should be on the request summary on top- as it is vital, not on a hidden comment that will be easily lost. Clear request will be processed faster (as we can receive many of this).

Change 455114 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m5.sql.erb: Add GRANTS for nova user

https://gerrit.wikimedia.org/r/455114

aborrero updated the task description. (Show Details)Aug 24 2018, 11:09 AM

aborrero updated the task description. (Show Details)

Change 455114 merged by Marostegui:
[operations/puppet@production] production-m5.sql.erb: Add GRANTS for nova user

https://gerrit.wikimedia.org/r/455114

aborrero updated the task description. (Show Details)Aug 27 2018, 11:49 AM

Marostegui updated the task description. (Show Details)Aug 27 2018, 11:56 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-27T11:59:01Z] <marostegui> Create empty databases nova_api_eqiad1 and nova_eqiad1 on m5 master (db1073) - T202549

Mentioned in SAL (#wikimedia-operations) [2018-08-27T12:00:10Z] <marostegui> Create empty databases nova_api_eqiad1 and nova_eqiad1 on m5 master (db1073) - T202549

I have created the empty databases:

nova_api_eqiad1
nova_eqiad1

Also applied the GRANTS

GRANT USAGE ON *.* TO 'nova'@'208.80.154.23' IDENTIFIED BY PASSWORD 'xx' WITH MAX_USER_CONNECTIONS 100 
GRANT ALL PRIVILEGES ON `nova_api_eqiad1`.* TO 'nova'@'208.80.154.23'                                                                         
GRANT ALL PRIVILEGES ON `nova_eqiad1`.* TO 'nova'@'208.80.154.23'

GRANT USAGE ON *.* TO 'nova'@'208.80.154.132' IDENTIFIED BY PASSWORD 'xx' WITH MAX_USER_CONNECTIONS 100 
GRANT ALL PRIVILEGES ON `nova_api_eqiad1`.* TO 'nova'@'208.80.154.132'                                                                         
GRANT ALL PRIVILEGES ON `nova_eqiad1`.* TO 'nova'@'208.80.154.132'

aborrero mentioned this in T202889: cloudvps: dedicated openstack database.Aug 27 2018, 12:34 PM

The two new DB will imported tomorrow.

"Review m5 database backups" Usually means giving extra grants to the backup user, review the list and check those get backed up for the first time.

Mentioned in SAL (#wikimedia-operations) [2018-08-28T10:51:13Z] <arturo> T202549 downtime cloudcontrol1003.wikimedia.org in icinga for 2h

aborrero updated the task description. (Show Details)Aug 28 2018, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-28T10:52:21Z] <marostegui> Import nova_api_eqiad1 and nova_eqiad1 into m5 master - T202549

aborrero updated the task description. (Show Details)Aug 28 2018, 10:53 AM

Change 454774 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: eqiad1: move nova DBs to m5-master

https://gerrit.wikimedia.org/r/454774

aborrero updated the task description. (Show Details)Aug 28 2018, 10:57 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-28T11:00:46Z] <arturo> T202549 delete mysql-server from cloudcontrol100[3,4.wikimedia.org

Grants added on m5 for the backups:

| GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `nova_api_eqiad1`.* TO 'dump'@'10.192.0.25'                                             |
| GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `nova_eqiad1`.* TO 'dump'@'10.192.0.25'

| GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `nova_api_eqiad1`.* TO 'dump'@'10.64.48.17'                                             |
| GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `nova_eqiad1`.* TO 'dump'@'10.64.48.17'

aborrero updated the task description. (Show Details)Aug 28 2018, 11:02 AM

Mentioned in SAL (#wikimedia-cloud) [2018-08-28T11:04:26Z] <arturo> T202549 eqiad1 databases are all now running in m5-master. Mysql has been cleaned from cloudcontrol100[3,4]

All is done. Thanks @Bstorm, @Marostegui and @jcrespo !!

The backups will run today, so we will check tomorrow if they were generated correctly

Backups for nova_api_eqiad1 and nova_eqiad1 were generated correctly

nova_api_eqiad1

-rw-r--r-- 1 dump dump  492 Aug 28 18:05 nova_api_eqiad1.build_requests-schema.sql.gz
-rw-r--r-- 1 dump dump  920 Aug 28 18:05 nova_api_eqiad1.build_requests.sql.gz
-rw-r--r-- 1 dump dump  308 Aug 28 18:05 nova_api_eqiad1.cell_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  335 Aug 28 18:05 nova_api_eqiad1.flavor_extra_specs-schema.sql.gz
-rw-r--r-- 1 dump dump  312 Aug 28 18:05 nova_api_eqiad1.flavor_projects-schema.sql.gz
-rw-r--r-- 1 dump dump  358 Aug 28 18:05 nova_api_eqiad1.flavors-schema.sql.gz
-rw-r--r-- 1 dump dump  318 Aug 28 18:05 nova_api_eqiad1.host_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  352 Aug 28 18:05 nova_api_eqiad1.instance_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  214 Aug 28 18:05 nova_api_eqiad1.migrate_version-schema.sql.gz
-rw-r--r-- 1 dump dump  217 Aug 28 18:05 nova_api_eqiad1.migrate_version.sql.gz
-rw-r--r-- 1 dump dump  290 Aug 28 18:05 nova_api_eqiad1.request_specs-schema.sql.gz
-rw-r--r-- 1 dump dump 8.4K Aug 28 18:05 nova_api_eqiad1.request_specs.sql.gz
-rw-r--r-- 1 dump dump   96 Aug 28 18:04 nova_api_eqiad1-schema-create.sql.gz

And nova_eqiad1

-rw-r--r-- 1 dump dump  345 Aug 28 18:05 nova_eqiad1.agent_builds-schema.sql.gz
-rw-r--r-- 1 dump dump  336 Aug 28 18:05 nova_eqiad1.aggregate_hosts-schema.sql.gz
-rw-r--r-- 1 dump dump  349 Aug 28 18:05 nova_eqiad1.aggregate_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  272 Aug 28 18:05 nova_eqiad1.aggregates-schema.sql.gz
-rw-r--r-- 1 dump dump  290 Aug 28 18:05 nova_eqiad1.allocations-schema.sql.gz
-rw-r--r-- 1 dump dump  491 Aug 28 18:05 nova_eqiad1.block_device_mapping-schema.sql.gz
-rw-r--r-- 1 dump dump 5.0K Aug 28 18:05 nova_eqiad1.block_device_mapping.sql.gz
-rw-r--r-- 1 dump dump  340 Aug 28 18:05 nova_eqiad1.bw_usage_cache-schema.sql.gz
-rw-r--r-- 1 dump dump  330 Aug 28 18:05 nova_eqiad1.cells-schema.sql.gz
-rw-r--r-- 1 dump dump  300 Aug 28 18:05 nova_eqiad1.certificates-schema.sql.gz
-rw-r--r-- 1 dump dump  541 Aug 28 18:05 nova_eqiad1.compute_nodes-schema.sql.gz
-rw-r--r-- 1 dump dump 2.1K Aug 28 18:05 nova_eqiad1.compute_nodes.sql.gz
-rw-r--r-- 1 dump dump  341 Aug 28 18:05 nova_eqiad1.console_pools-schema.sql.gz
-rw-r--r-- 1 dump dump  363 Aug 28 18:05 nova_eqiad1.consoles-schema.sql.gz
-rw-r--r-- 1 dump dump  292 Aug 28 18:05 nova_eqiad1.dns_domains-schema.sql.gz
-rw-r--r-- 1 dump dump  498 Aug 28 18:05 nova_eqiad1.fixed_ips-schema.sql.gz
-rw-r--r-- 1 dump dump  390 Aug 28 18:05 nova_eqiad1.floating_ips-schema.sql.gz
-rw-r--r-- 1 dump dump  358 Aug 28 18:05 nova_eqiad1.instance_actions_events-schema.sql.gz
-rw-r--r-- 1 dump dump 4.5K Aug 28 18:05 nova_eqiad1.instance_actions_events.sql.gz
-rw-r--r-- 1 dump dump  378 Aug 28 18:05 nova_eqiad1.instance_actions-schema.sql.gz
-rw-r--r-- 1 dump dump  12K Aug 28 18:05 nova_eqiad1.instance_actions.sql.gz
-rw-r--r-- 1 dump dump  362 Aug 28 18:05 nova_eqiad1.instance_extra-schema.sql.gz
-rw-r--r-- 1 dump dump 5.9K Aug 28 18:05 nova_eqiad1.instance_extra.sql.gz
-rw-r--r-- 1 dump dump  372 Aug 28 18:05 nova_eqiad1.instance_faults-schema.sql.gz
-rw-r--r-- 1 dump dump 2.6K Aug 28 18:05 nova_eqiad1.instance_faults.sql.gz
-rw-r--r-- 1 dump dump  317 Aug 28 18:05 nova_eqiad1.instance_group_member-schema.sql.gz
-rw-r--r-- 1 dump dump  315 Aug 28 18:05 nova_eqiad1.instance_group_policy-schema.sql.gz
-rw-r--r-- 1 dump dump  300 Aug 28 18:05 nova_eqiad1.instance_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  273 Aug 28 18:05 nova_eqiad1.instance_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump 3.7K Aug 28 18:05 nova_eqiad1.instance_id_mappings.sql.gz
-rw-r--r-- 1 dump dump  342 Aug 28 18:05 nova_eqiad1.instance_info_caches-schema.sql.gz
-rw-r--r-- 1 dump dump 5.4K Aug 28 18:05 nova_eqiad1.instance_info_caches.sql.gz
-rw-r--r-- 1 dump dump  323 Aug 28 18:05 nova_eqiad1.instance_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  787 Aug 28 18:05 nova_eqiad1.instances-schema.sql.gz
-rw-r--r-- 1 dump dump 8.3K Aug 28 18:05 nova_eqiad1.instances.sql.gz
-rw-r--r-- 1 dump dump  326 Aug 28 18:05 nova_eqiad1.instance_system_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  12K Aug 28 18:05 nova_eqiad1.instance_system_metadata.sql.gz
-rw-r--r-- 1 dump dump  377 Aug 28 18:05 nova_eqiad1.instance_type_extra_specs-schema.sql.gz
-rw-r--r-- 1 dump dump  343 Aug 28 18:05 nova_eqiad1.instance_type_projects-schema.sql.gz
-rw-r--r-- 1 dump dump  387 Aug 28 18:05 nova_eqiad1.instance_types-schema.sql.gz
-rw-r--r-- 1 dump dump  258 Aug 28 18:05 nova_eqiad1.instance_types.sql.gz
-rw-r--r-- 1 dump dump  327 Aug 28 18:05 nova_eqiad1.inventories-schema.sql.gz
-rw-r--r-- 1 dump dump  345 Aug 28 18:05 nova_eqiad1.key_pairs-schema.sql.gz
-rw-r--r-- 1 dump dump  222 Aug 28 18:05 nova_eqiad1.migrate_version-schema.sql.gz
-rw-r--r-- 1 dump dump  208 Aug 28 18:05 nova_eqiad1.migrate_version.sql.gz
-rw-r--r-- 1 dump dump  516 Aug 28 18:05 nova_eqiad1.migrations-schema.sql.gz
-rw-r--r-- 1 dump dump  555 Aug 28 18:05 nova_eqiad1.networks-schema.sql.gz
-rw-r--r-- 1 dump dump  478 Aug 28 18:05 nova_eqiad1.pci_devices-schema.sql.gz
-rw-r--r-- 1 dump dump  342 Aug 28 18:05 nova_eqiad1.project_user_quotas-schema.sql.gz
-rw-r--r-- 1 dump dump  277 Aug 28 18:05 nova_eqiad1.provider_fw_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  286 Aug 28 18:05 nova_eqiad1.quota_classes-schema.sql.gz
-rw-r--r-- 1 dump dump  307 Aug 28 18:05 nova_eqiad1.quotas-schema.sql.gz
-rw-r--r-- 1 dump dump  194 Aug 28 18:05 nova_eqiad1.quotas.sql.gz
-rw-r--r-- 1 dump dump  330 Aug 28 18:05 nova_eqiad1.quota_usages-schema.sql.gz
-rw-r--r-- 1 dump dump  493 Aug 28 18:05 nova_eqiad1.quota_usages.sql.gz
-rw-r--r-- 1 dump dump  403 Aug 28 18:05 nova_eqiad1.reservations-schema.sql.gz
-rw-r--r-- 1 dump dump  24K Aug 28 18:05 nova_eqiad1.reservations.sql.gz
-rw-r--r-- 1 dump dump  235 Aug 28 18:05 nova_eqiad1.resource_provider_aggregates-schema.sql.gz
-rw-r--r-- 1 dump dump  308 Aug 28 18:05 nova_eqiad1.resource_providers-schema.sql.gz
-rw-r--r-- 1 dump dump  248 Aug 28 18:05 nova_eqiad1.s3_images-schema.sql.gz
-rw-r--r-- 1 dump dump  416 Aug 28 18:05 nova_eqiad1.s3_images.sql.gz
-rw-r--r-- 1 dump dump   92 Aug 28 18:04 nova_eqiad1-schema-create.sql.gz
-rw-r--r-- 1 dump dump  281 Aug 28 18:05 nova_eqiad1.security_group_default_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  353 Aug 28 18:05 nova_eqiad1.security_group_instance_association-schema.sql.gz
-rw-r--r-- 1 dump dump  351 Aug 28 18:05 nova_eqiad1.security_group_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  315 Aug 28 18:05 nova_eqiad1.security_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  289 Aug 28 18:05 nova_eqiad1.security_groups.sql.gz
-rw-r--r-- 1 dump dump  373 Aug 28 18:05 nova_eqiad1.services-schema.sql.gz
-rw-r--r-- 1 dump dump  477 Aug 28 18:05 nova_eqiad1.services.sql.gz
-rw-r--r-- 1 dump dump  292 Aug 28 18:05 nova_eqiad1.shadow_agent_builds-schema.sql.gz
-rw-r--r-- 1 dump dump  263 Aug 28 18:05 nova_eqiad1.shadow_aggregate_hosts-schema.sql.gz
-rw-r--r-- 1 dump dump  270 Aug 28 18:05 nova_eqiad1.shadow_aggregate_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  262 Aug 28 18:05 nova_eqiad1.shadow_aggregates-schema.sql.gz
-rw-r--r-- 1 dump dump  401 Aug 28 18:05 nova_eqiad1.shadow_block_device_mapping-schema.sql.gz
-rw-r--r-- 1 dump dump  322 Aug 28 18:05 nova_eqiad1.shadow_bw_usage_cache-schema.sql.gz
-rw-r--r-- 1 dump dump  309 Aug 28 18:05 nova_eqiad1.shadow_cells-schema.sql.gz
-rw-r--r-- 1 dump dump  273 Aug 28 18:05 nova_eqiad1.shadow_certificates-schema.sql.gz
-rw-r--r-- 1 dump dump  507 Aug 28 18:05 nova_eqiad1.shadow_compute_nodes-schema.sql.gz
-rw-r--r-- 1 dump dump  305 Aug 28 18:05 nova_eqiad1.shadow_console_pools-schema.sql.gz
-rw-r--r-- 1 dump dump  290 Aug 28 18:05 nova_eqiad1.shadow_consoles-schema.sql.gz
-rw-r--r-- 1 dump dump  269 Aug 28 18:05 nova_eqiad1.shadow_dns_domains-schema.sql.gz
-rw-r--r-- 1 dump dump  328 Aug 28 18:05 nova_eqiad1.shadow_fixed_ips-schema.sql.gz
-rw-r--r-- 1 dump dump  317 Aug 28 18:05 nova_eqiad1.shadow_floating_ips-schema.sql.gz
-rw-r--r-- 1 dump dump  310 Aug 28 18:05 nova_eqiad1.shadow_instance_actions_events-schema.sql.gz
-rw-r--r-- 1 dump dump  316 Aug 28 18:05 nova_eqiad1.shadow_instance_actions-schema.sql.gz
-rw-r--r-- 1 dump dump  323 Aug 28 18:05 nova_eqiad1.shadow_instance_extra-schema.sql.gz
-rw-r--r-- 1 dump dump  299 Aug 28 18:05 nova_eqiad1.shadow_instance_faults-schema.sql.gz
-rw-r--r-- 1 dump dump  265 Aug 28 18:05 nova_eqiad1.shadow_instance_group_member-schema.sql.gz
-rw-r--r-- 1 dump dump  262 Aug 28 18:05 nova_eqiad1.shadow_instance_group_policy-schema.sql.gz
-rw-r--r-- 1 dump dump  276 Aug 28 18:05 nova_eqiad1.shadow_instance_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  258 Aug 28 18:05 nova_eqiad1.shadow_instance_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  279 Aug 28 18:05 nova_eqiad1.shadow_instance_info_caches-schema.sql.gz
-rw-r--r-- 1 dump dump  277 Aug 28 18:05 nova_eqiad1.shadow_instance_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  644 Aug 28 18:05 nova_eqiad1.shadow_instances-schema.sql.gz
-rw-r--r-- 1 dump dump  279 Aug 28 18:05 nova_eqiad1.shadow_instance_system_metadata-schema.sql.gz
-rw-r--r-- 1 dump dump  278 Aug 28 18:05 nova_eqiad1.shadow_instance_type_extra_specs-schema.sql.gz
-rw-r--r-- 1 dump dump  268 Aug 28 18:05 nova_eqiad1.shadow_instance_type_projects-schema.sql.gz
-rw-r--r-- 1 dump dump  347 Aug 28 18:05 nova_eqiad1.shadow_instance_types-schema.sql.gz
-rw-r--r-- 1 dump dump  314 Aug 28 18:05 nova_eqiad1.shadow_key_pairs-schema.sql.gz
-rw-r--r-- 1 dump dump  229 Aug 28 18:05 nova_eqiad1.shadow_migrate_version-schema.sql.gz
-rw-r--r-- 1 dump dump  417 Aug 28 18:05 nova_eqiad1.shadow_migrations-schema.sql.gz
-rw-r--r-- 1 dump dump  470 Aug 28 18:05 nova_eqiad1.shadow_networks-schema.sql.gz
-rw-r--r-- 1 dump dump  372 Aug 28 18:05 nova_eqiad1.shadow_pci_devices-schema.sql.gz
-rw-r--r-- 1 dump dump  283 Aug 28 18:05 nova_eqiad1.shadow_project_user_quotas-schema.sql.gz
-rw-r--r-- 1 dump dump  282 Aug 28 18:05 nova_eqiad1.shadow_provider_fw_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  275 Aug 28 18:05 nova_eqiad1.shadow_quota_classes-schema.sql.gz
-rw-r--r-- 1 dump dump  276 Aug 28 18:05 nova_eqiad1.shadow_quotas-schema.sql.gz
-rw-r--r-- 1 dump dump  298 Aug 28 18:05 nova_eqiad1.shadow_quota_usages-schema.sql.gz
-rw-r--r-- 1 dump dump  301 Aug 28 18:05 nova_eqiad1.shadow_reservations-schema.sql.gz
-rw-r--r-- 1 dump dump  251 Aug 28 18:05 nova_eqiad1.shadow_s3_images-schema.sql.gz
-rw-r--r-- 1 dump dump  286 Aug 28 18:05 nova_eqiad1.shadow_security_group_default_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  278 Aug 28 18:05 nova_eqiad1.shadow_security_group_instance_association-schema.sql.gz
-rw-r--r-- 1 dump dump  298 Aug 28 18:05 nova_eqiad1.shadow_security_group_rules-schema.sql.gz
-rw-r--r-- 1 dump dump  282 Aug 28 18:05 nova_eqiad1.shadow_security_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  327 Aug 28 18:05 nova_eqiad1.shadow_services-schema.sql.gz
-rw-r--r-- 1 dump dump  258 Aug 28 18:05 nova_eqiad1.shadow_snapshot_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  311 Aug 28 18:05 nova_eqiad1.shadow_snapshots-schema.sql.gz
-rw-r--r-- 1 dump dump  308 Aug 28 18:05 nova_eqiad1.shadow_task_log-schema.sql.gz
-rw-r--r-- 1 dump dump  289 Aug 28 18:05 nova_eqiad1.shadow_virtual_interfaces-schema.sql.gz
-rw-r--r-- 1 dump dump  258 Aug 28 18:05 nova_eqiad1.shadow_volume_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  370 Aug 28 18:05 nova_eqiad1.shadow_volume_usage_cache-schema.sql.gz
-rw-r--r-- 1 dump dump  253 Aug 28 18:05 nova_eqiad1.snapshot_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  306 Aug 28 18:05 nova_eqiad1.snapshots-schema.sql.gz
-rw-r--r-- 1 dump dump  209 Aug 28 18:05 nova_eqiad1.tags-schema.sql.gz
-rw-r--r-- 1 dump dump  369 Aug 28 18:05 nova_eqiad1.task_log-schema.sql.gz
-rw-r--r-- 1 dump dump  378 Aug 28 18:05 nova_eqiad1.virtual_interfaces-schema.sql.gz
-rw-r--r-- 1 dump dump  251 Aug 28 18:05 nova_eqiad1.volume_id_mappings-schema.sql.gz
-rw-r--r-- 1 dump dump  369 Aug 28 18:05 nova_eqiad1.volume_usage_cache-schema.sql.gz

jcrespo awarded a token.Aug 29 2018, 7:38 AM

cloudvps: eqiad1: move nova db to m5-masterClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

cloudvps: eqiad1: move nova db to m5-master
Closed, ResolvedPublic
Actions

Related Objects
Search...