Page MenuHomePhabricator

Support multi-instance database hosts on mediawiki-config
Closed, ResolvedPublic

Description

We are going to introduce the concept of multi-instance hosts in core.
Basically a host that will have two (or more) instances running on different ports.
This is needed for the goal to set up s8 and also to make sure we get the most out of our new and powerful hosts.
By doing this we'll be able to serve (with redundancy) recentchanges service for all the core shards with 8 hosts.
For example:

db2084 will be replicating s4 on port 3314 and s5 on port 3315

We would like know how to add it to mediawiki-config, and to confirm that mediawiki-config supports "HOST:PORT" syntax.

I have created a gerrit patch with that specific host db2084 so someone can actually review how it should be done to make sure this doesn't break the site: https://gerrit.wikimedia.org/r/385152

I have added a bunch of reviewers that I think that might help, but feel free to remove yourself if you are not sure you can.
Same with the project tags I have added.

Thanks.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 19 2017, 8:44 AM

Change 385152 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] Draft: Setting a multi-instance host

https://gerrit.wikimedia.org/r/385152

Gilles moved this task from Inbox to Radar on the Performance-Team board.Oct 23 2017, 8:25 PM
Gilles edited projects, added Performance-Team (Radar); removed Performance-Team.

Anyone would be able to review the patch so we can move forward with splitting wikidata from s5 into its own shard (s8)?.
It would be good to start having a clear picture of how codfw will look like in regard with the server allocation

Thanks!

Change 386810 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Pool db2084 as multi-instance host

https://gerrit.wikimedia.org/r/386810

Marostegui added a comment.EditedOct 27 2017, 11:46 AM

I would like to deploy: https://gerrit.wikimedia.org/r/#/c/386810/ next week, can I get someone else apart from @Anomie (thanks!) to check it.
I would feel more comfortable

cc @tstarling @aaron @Reedy maybe?

Volans added a subscriber: Volans.Oct 27 2017, 12:09 PM

I'm not very familiar with the beta-side of wmf-config, but from a quick look it seems to me that db-labs.php is used there.
To be on the safe side, could we apply this first in deployment-prep adding the :3306 to some of the DBs and verify that everything is still working as expected? Will this test the same code branches that will be used once in production?

Reedy added a comment.EditedOct 27 2017, 9:26 PM

I'm not very familiar with the beta-side of wmf-config, but from a quick look it seems to me that db-labs.php is used there.
To be on the safe side, could we apply this first in deployment-prep adding the :3306 to some of the DBs and verify that everything is still working as expected? Will this test the same code branches that will be used once in production?

FWIW, appending :3306 to $wgDBserver on my dev wiki makes no obvious difference

A quick dig into the code, and on the MySQLi wrapper, we cater for this 'hosname:port' pattern already

		$hostAndPort = IP::splitHostAndPort( $realServer );
		if ( $hostAndPort ) {
			$realServer = $hostAndPort[0];
			if ( $hostAndPort[1] ) {
				$port = $hostAndPort[1];
			}
		}

And we use DatabaseMysqli in WMF production too. Yay.

For the the older non mysqli wrapper, the PHP docs say this should work too

* @param string $server [optional] <p>
* The MySQL server. It can also include a port number. e.g. 
* "hostname:port" or a path to a local socket e.g. ":/path/to/socket" for 
* the localhost.

So... I'd be pretty sure this will work. Per @Volans we could just append ports to all servers (for consistency), on beta, and then deploy it and see ;). Or cherry pick it to deployment-tin (sync-common maybe? I can never remember), and use mwscript sql.php --wiki=enwiki and do a simple query and check it doesn't just break

Thanks @Reedy for checking this! :-)
Would you be able to quickly check it on beta? (I have never deployed/used beta so it might take me even more to figure it out than to actually test it!) :-)

Thanks @Reedy @Anomie and @tstarling for reviewing the patches, really appreciated.
As the three of you are really confident this should work, I will try to deploy it during this week early in the morning (and only for codfw) and leave it like that for a few day!
Thanks again - I will notify here once it is deployed

Change 387229 had a related patch set uploaded (by Reedy; owner: Reedy):
[operations/mediawiki-config@master] Add port numbers to beta db servers

https://gerrit.wikimedia.org/r/387229

Change 387229 merged by jenkins-bot:
[operations/mediawiki-config@master] Add port numbers to beta db servers

https://gerrit.wikimedia.org/r/387229

Reedy added a comment.Oct 30 2017, 1:01 PM

Beta is still working after that patch being deployed

Beta is still working after that patch being deployed

Thank you!! :-)

Change 386810 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Pool db2084 as multi-instance host

https://gerrit.wikimedia.org/r/386810

Mentioned in SAL (#wikimedia-operations) [2017-10-31T09:12:27Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Pool db2084 as multi-instance core host T178553 T178359 (duration: 00m 49s)

Marostegui closed this task as Resolved.Oct 31 2017, 9:56 AM
Marostegui claimed this task.

Crossposting from T178359#3722507:

This patch has been merged and deployed: https://gerrit.wikimedia.org/r/#/c/386810/ we know have db2084 as the first host "serving" (not really because it is codfw) with multi-instance on mediawiki-config.
@Volans and myself have been testing it using mwdebug and pointing `mw2099` just to db2084:3314 and 3315 and randomly browsing dewiki and commons and it all went fine.
Going to leave it there running for a few days before starting to add the other codfw which are ready to the config.


Thanks again @Volans for your help
Thanks @Anomie @Reedy and @tstarling for taking the time to test and review the patch.

I believe this can be closed now.

Change 385152 abandoned by Marostegui:
Draft: Setting a multi-instance host

Reason:
This was a draft that is no longer needed as the proper patch has been merged and deployed here: https://gerrit.wikimedia.org/r/#/c/386810/

Thanks Anomie for reviewing it!

https://gerrit.wikimedia.org/r/385152

Mentioned in SAL (#wikimedia-operations) [2017-11-06T06:46:10Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db2084 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-11-06T09:25:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2085 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-11-06T09:26:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db2085 as recentchanges multi-instance host on s3 and s5 T178553 T178359 (duration: 00m 46s)

jcrespo renamed this task from Support multi-instance hosts on mediawiki-config to Support multi-instance database hosts on mediawiki-config.Jan 10 2018, 10:02 AM