Set up roles for new wiki replicas layout
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Bstorm
	Aug 19 2020, 6:39 PM

Description

We likely need to base it on role(mariadb::dbstore_multiinstance), and when adding the specific requirements from role(labs::db::wikireplica_web), role(labs::db::wikireplica_analytics) and role(labs::db::wikireplica_analytics::dedicated), this is a good opportunity to remove yet another use of the word "labs".

Details

Subject	Repo	Branch	Lines +/-
wikireplicas: set up site.pp and hosts hiera for new servers	operations/puppet	production	+34 -1
wikireplicas: create multiinstance roles and profiles	operations/puppet	production	+253 -0
wikireplicas: test removing deprecated passwords module from role	operations/puppet	production	+0 -3
wikireplicas: fix path typo for the heartbeat-views file	operations/puppet	production	+1 -1
wikireplicas: refactor to eliminate confusing "labsdb" naming	operations/puppet	production	+239 -319
wikireplicas: add new location for these fake passwords	labs/private	master	+4 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Marostegui	T233766 labsdb1011 mariadb crashed
			Restricted Task
			Restricted Task
Open		None	T215858 Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema
Open		None	T280152 Mitigate breaking changes from the new Wiki Replicas architecture
			Unknown Object (Task)
Resolved		RobH	T260441 (Need By: ASAP) rack/setup/install clouddb10[13-20]
Resolved		Bstorm	T260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
Resolved		Bstorm	T260843 Set up roles for new wiki replicas layout
Resolved		Marostegui	T265135 wikireplicas: Define MW sections per host
Resolved		Marostegui	T267090 Productionize clouddb10[13-20]
Resolved		Marostegui	T268312 Deploy labsdbuser and views to new clouddb hosts
Resolved		Bstorm	T269200 Create end-user accounts on the new clouddb hosts
Resolved		Bstorm	T269620 maintain-dbusers doesn't close connections right on harvest-replicas
Resolved		MoritzMuehlenhoff	T268725 Include mail on standard_packages.pp
Resolved		Marostegui	T268742 Test upgrading sanitarium hosts to Buster + 10.4
Resolved		Marostegui	T272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
Resolved		• Cmjohnson	T272125 Memory errors on clouddb1019
Resolved		dcaro	T272127 2021-01-15: PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL
Resolved		Marostegui	T280492 Upgrade all sanitarium masters to 10.4 and Buster
Resolved	Request	wiki_willy	T281794 decommission db1082.eqiad.wmnet
Resolved	Request	• Cmjohnson	T281959 decommission db1074.eqiad.wmnet
Resolved	Request	• Cmjohnson	T282079 decommission db1079.eqiad.wmnet
Resolved	Request	• Cmjohnson	T282093 decommission db1087.eqiad.wmnet
Resolved	Request	• Cmjohnson	T282096 decommission db1085.eqiad.wmnet
Declined		None	T267376 Set up IP addresses for the new wiki replicas setup
Resolved		Bstorm	T272720 Allocate service IPs for new wikireplicas setup
Resolved		Kormat	T268098 Make innodb_change_buffering configurable

Event Timeline

Bstorm created this task.Aug 19 2020, 6:39 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2020, 6:39 PM

Bstorm triaged this task as Medium priority.Aug 19 2020, 6:39 PM

Bstorm added a parent task: T260389: Redesign and rebuild the wikireplicas service using a multi-instance architecture.

Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

Bstorm moved this task from Backlog to Wiki replicas on the Data-Services board.

I'm thinking something like role(mariadb::wikireplica_multiinstance::web) and role(mariadb::wikireplica_multiinstance::analytics) to be more consistent with other DB roles, and I could try to move the WMCS/wiki-replicas specific stuff over to profiles under profile::wmcs::wikireplicas. That would start getting things out of role::labs and profile::labs and put those namespaces to bed.

I could start getting those profiles ready if that works for you @Marostegui .

Obviously the existing views scripts will need changes to work with multi-instance, but I'll get there.

Change 621618 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikireplicas: refactor to eliminate confusing "labsdb" naming

https://gerrit.wikimedia.org/r/621618

gerritbot added a project: Patch-For-Review.Aug 21 2020, 12:44 AM

Change 621777 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/private@master] wikireplicas: add new location for these fake passwords

https://gerrit.wikimedia.org/r/621777

Change 621777 merged by Bstorm:
[labs/private@master] wikireplicas: add new location for these fake passwords

https://gerrit.wikimedia.org/r/621777

Bstorm mentioned this in rLPRI22912b2d603a: wikireplicas: add new location for these fake passwords.Aug 21 2020, 6:48 PM

In T260843#6397956, @Bstorm wrote:

I could start getting those profiles ready if that works for you @Marostegui .

Absolutely! Thank you very much

Change 621618 merged by Bstorm:
[operations/puppet@production] wikireplicas: refactor to eliminate confusing "labsdb" naming

https://gerrit.wikimedia.org/r/621618

Change 622387 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikireplicas: fix path typo for the heartbeat-views file

https://gerrit.wikimedia.org/r/622387

Change 622387 merged by Bstorm:
[operations/puppet@production] wikireplicas: fix path typo for the heartbeat-views file

https://gerrit.wikimedia.org/r/622387

Ok, that sorts out the name spacing. Now to add something to incorporate a multi-instance option.

Change 622444 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikireplicas: create multiinstance roles and profiles

https://gerrit.wikimedia.org/r/622444

Change 623623 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikireplicas: test removing deprecated passwords module from role

https://gerrit.wikimedia.org/r/623623

Change 623623 merged by Bstorm:
[operations/puppet@production] wikireplicas: test removing deprecated passwords module from role

https://gerrit.wikimedia.org/r/623623

jbond subscribed.Sep 4 2020, 2:04 PM

Change 622444 merged by Bstorm:
[operations/puppet@production] wikireplicas: create multiinstance roles and profiles

https://gerrit.wikimedia.org/r/622444

@Bstorm how's this work progressing? Should I assign for now the mariadb::dbstore_multiinstance for now (when the hosts get installed) so we can get the data copied and mysql up while you keep refining the final role?

In T260843#6532209, @Marostegui wrote:

@Bstorm how's this work progressing? Should I assign for now the mariadb::dbstore_multiinstance for now (when the hosts get installed) so we can get the data copied and mysql up while you keep refining the final role?

I'm distracted with the work on Quarry at the moment to get it ready, but I feel good about where I left wmcs::db::wikireplicas::web_multiinstance and wmcs::db::wikireplicas::analytics_multiinstance so go ahead and try using them if you can. I'll help fix it if it's not actually good 😁

If we end up with a proxy somewhere that we can put 8 IP addresses on, I can really try out the suggested proxy role when there's things to connect to. The proxy setup is the part that I'm a bit less sure of. I will be circling back to view scripts shortly, but that doesn't block any of your stuff I don't think @Marostegui

In T260843#6532903, @Bstorm wrote:

In T260843#6532209, @Marostegui wrote:

@Bstorm how's this work progressing? Should I assign for now the mariadb::dbstore_multiinstance for now (when the hosts get installed) so we can get the data copied and mysql up while you keep refining the final role?

I'm distracted with the work on Quarry at the moment to get it ready, but I feel good about where I left wmcs::db::wikireplicas::web_multiinstance and wmcs::db::wikireplicas::analytics_multiinstance so go ahead and try using them if you can. I'll help fix it if it's not actually good 😁

Will do thanks. We still need to decide on T265135: wikireplicas: Define MW sections per host

If we end up with a proxy somewhere that we can put 8 IP addresses on, I can really try out the suggested proxy role when there's things to connect to. The proxy setup is the part that I'm a bit less sure of. I will be circling back to view scripts shortly, but that doesn't block any of your stuff I don't think @Marostegui

Nope, that won't be blocking me. Thank you!

Nintendofan885 subscribed.Oct 26 2020, 12:15 PM

@Bstorm new clouddb hosts are racked and online. I have checked the HW and it looks good, they are ready to get the data (T267090)
Can you set them up on puppet (you might want to also check the data we're going to put on them before: T265135#6598952 ).
Once you are happy with puppet running there, I can start the data transfer.

Note: it is ok to set up one or two host at the time, I don't need all of them with puppet to start transferring data :-)

Marostegui mentioned this in T265135: wikireplicas: Define MW sections per host.Nov 3 2020, 6:41 AM

Marostegui closed subtask T265135: wikireplicas: Define MW sections per host as Resolved.Nov 4 2020, 4:05 PM

Marostegui mentioned this in T267090: Productionize clouddb10[13-20].Nov 6 2020, 6:57 AM

Change 639815 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wikireplicas: set up site.pp and hosts hiera for new servers

https://gerrit.wikimedia.org/r/639815

Change 639815 merged by Marostegui:
[operations/puppet@production] wikireplicas: set up site.pp and hosts hiera for new servers

https://gerrit.wikimedia.org/r/639815