1007 to jessie
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MoritzMuehlenhoff
	Jan 15 2016, 12:48 PM

Description

Still on precise, migrate to jessie. This would also involve a postgresql update from 9.1 to 9.4

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Change osm master to be labsdb1007 on configuration	operations/puppet	production	+6 -7
tools: Make DNS point to labsdb1004 and not 1005	operations/puppet	production	+3 -3
labsdb-toolsdb: Cleaning up tls certificates	operations/puppet	production	+22 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production
Resolved	jcrespo	T123731 Migrate labsdb1005/1006/1007 to jessie
Resolved	Marostegui	T157358 labsdb1005 (mysql) maintenance for reimage
Duplicate	None	T157359 labsdb1006/1007 (postgresql) maintenance

Event Timeline

MoritzMuehlenhoff created this task.Jan 15 2016, 12:48 PM

MoritzMuehlenhoff raised the priority of this task from to Needs Triage.

MoritzMuehlenhoff updated the task description. (Show Details)

MoritzMuehlenhoff added a project: SRE.

MoritzMuehlenhoff subscribed.

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 15 2016, 12:48 PM

MoritzMuehlenhoff added a parent task: T123525: reduce amount of remaining Ubuntu 12.04 (precise) systems in production.Jan 15 2016, 12:51 PM

1005 as well:

labsdb1005.eqiad.wmnet: True
labsdb1006.eqiad.wmnet: True
labsdb1007.eqiad.wmnet: True

Dzahn added a project: Cloud-VPS.Apr 11 2016, 5:56 PM

Restricted Application added a project: Cloud-Services. · View Herald TranscriptApr 11 2016, 5:56 PM

Dzahn added a project: DBA.Apr 11 2016, 5:57 PM

one thought is we have an influx of new labsdb things coming I believe. This way sort itself out w/o a lot of in-place shuffling.

There is indeed a replacement for labsdb100[123] about to arrive. However, there are no short-term plans for these, as they have lower impact.

labsdb1005.eqiad.wmnet has already a jessie slave- so it should be able to have scheduled downtime soon.
Not much is happening for the postgres slaves.

Setting as stalled, though next steps look like this:

Flip tools master from labsdb1005 to labsdb1004
Decommission labsdb1005

Not sure about postgresql/osm steps and if the osm roles are jessie-ready yet, @akosiaris perhaps?

Change 318520 had a related patch set uploaded (by Jcrespo):
labsdb-toolsdb: Cleaning up tls certificates

https://gerrit.wikimedia.org/r/318520

gerritbot added a project: Patch-For-Review.Oct 28 2016, 11:15 AM

Change 318520 merged by Jcrespo:
labsdb-toolsdb: Cleaning up tls certificates

https://gerrit.wikimedia.org/r/318520

We need to schedule a downtime to do this move from labsdb1005 to labsdb1004. This should be a very short window of actual outage.

Need to sort out impact of this maint.

If we settle on a date and announce on labs-announce...

In T123731#2757001, @yuvipanda wrote:

If we settle on a date and announce on labs-announce...

@yuvipanda I think the asks here if you could think on it are:

Who do we let know toolsdb is going down
What are the impacts of toolsdb going down (just generally)

-1- assume labs-announce is fine

-2- I'm not sure? Are things using toolsdb going to be ok? Assuming things reconnect on failure (probably a bad assumption) it's a small window of issue but looking for input. There may be little we can do to shore that up other than verbose announcement.

I wonder if it'll be better to do this next quarter. We've already done a few bits of pretty disruptive maintenance, and have one coming up next week.

If not next quarter, how about 2nd week of December?

I wonder if it'll be better to do this next quarter.

I am ok with next quarter- let's set a time. I have workarounded the 5.5 support on puppet, so this is no longer a blocker.

Just one comment- note that in theory this maintenance is not disruptive.

Ok. Early January?

January ok, but after the 15th.

jcrespo moved this task from Triage to Meta/Epic on the DBA board.Nov 10 2016, 12:10 PM

jcrespo mentioned this in T138517: mysqldump is timing out preventing all tables from being included in the dump.Nov 15 2016, 7:01 PM

Early January is here and the 15th is coming up fast -- @yuvipanda rightfully mentioned above that this will need a (presumably advance) notice to labs-announce, so… friendly ping :)

@jcrespo How does Jan 25 / 26 work for you?

In T123731#2916980, @yuvipanda wrote:

@jcrespo How does Jan 25 / 26 work for you?

+1. let's meet at some point to organize the details of how to do it (there is several possibilities) and send an announcement.

Ping! Jan 25 is a week away from now, not a lot of time left for an announcement :)

mark raised the priority of this task from Medium to High.Jan 18 2017, 1:37 PM

I didn't manage to send out the announcement due to unforseen personal issues. I'll send it out now after checking with jynus.

tom29739 subscribed.Jan 18 2017, 5:17 PM

Update: Since I'll be travelling on the 25th, I'm going to push this out to early February instead. I'll ping @jcrespo when he's back from vacation next week to put a solid date on it and make an announcement.

+1, let's meet before to clarify impact.

Ping! Early February is now a week away.

jcrespo created subtask T157358: labsdb1005 (mysql) maintenance for reimage.Feb 6 2017, 5:29 PM

jcrespo created subtask T157359: labsdb1006/1007 (postgresql) maintenance.

jcrespo changed the task status from Stalled to Open.Feb 6 2017, 5:32 PM

We announced a while ago we're gonna do this on the 15th.

Let's use T157358 for this. Postgres is a different beast.

Change 337775 had a related patch set uploaded (by Yuvipanda):
tools: Make DNS point to labsdb1004 and not 1005

https://gerrit.wikimedia.org/r/337775

Change 337775 merged by Yuvipanda:
tools: Make DNS point to labsdb1004 and not 1005

https://gerrit.wikimedia.org/r/337775

Marostegui subscribed.Mar 1 2017, 1:38 PM

Marostegui closed subtask T157358: labsdb1005 (mysql) maintenance for reimage as Resolved.Mar 2 2017, 9:22 AM

I see that labs1005/1006/1007 are all either re-installed or down. They don't show up as precise anymore when checking with salt.

Is this resolved (besides a decom subtask maybe?)?

Dzhan- the "reinstall as jessie" part is done, but the setup of the passive replica is not 100% complete. It will take one commit to fix it and some extra time for reimport- but there is not way to revert it anymore. I just got distracted with more important ongoing issues.

Got it, and thank you very much.

Change 343670 had a related patch set uploaded (by Jcrespo):
[operations/puppet] Change osm master to be labsdb1007 on configuration

https://gerrit.wikimedia.org/r/343670

Change 343670 merged by Jcrespo:
[operations/puppet] Change osm master to be labsdb1007 on configuration

https://gerrit.wikimedia.org/r/343670