Page MenuHomePhabricator
Create Task
Maniphest T171851

Reimage ores* hosts with Debian Stretch
Closed, ResolvedPublic
Actions

Description

We'd like to start using the new stat* machines to build models. It looks like they are running Stretch but our new ORES machines were installed with Jessie. What can be done to bring these machines in alignment?

Details

SubjectRepoBranchLines +/-
Remove the cluster server group and related stuffmediawiki/services/ores/deploymaster+14 -22
Remove ORES profile from scboperations/puppetproduction+10 -61
Add the ORES cluster to wikimedia_clustersoperations/puppetproduction+6 -0
Disable notification for role::oresoperations/puppetproduction+3 -0
Add ores scap::dsh::groupsoperations/puppetproduction+7 -0
ores: Allow oresX00X to reach respective oresrdboperations/puppetproduction+18 -0
ores: Set oresX00X hosts as role::oresoperations/puppetproduction+45 -16
Remove ores::stresstest as its no longer neededoperations/puppetproduction+0 -30
Set ores1* as spare::systemoperations/puppetproduction+1 -1
Reimage ores100* as stretchoperations/puppetproduction+18 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedakosiarisT162039 Prepare to service applications from kubernetes
 ResolvedakosiarisT162041 Expand the infrastructure to codfw
 ResolvedRobHT161700 CODFW: (4) hardware access request for kubernetes
 ResolvedRobHT142578 codfw/eqiad:(9+9) hardware access request for ORES
Unknown Object (Task)
 ResolvedakosiarisT165170 rack/setup/install ores2001-2009
 ResolvedNoneT176324 Scoring platform team FY18 Q2
 DeclinedNoneT179501 Use external dsh group to list pooled ORES nodes
 ResolvedakosiarisT168073 Switch ORES to dedicated cluster
 ResolvedHalfakT185901 Preliminary deployment of ORES to new cluster
 ResolvedakosiarisT171851 Reimage ores* hosts with Debian Stretch
 ResolvedawightT169246 Stress/capacity test new ores* cluster
 ResolvedHalfakT174402 Review and fix file handle management in worker and celery processes
 ResolvedawightT177036 Clean up file handle and Redis connection management in ORES worker and celery processes
 ResolvedNoneT175736 Give ores admins read access to /srv/log/ores/main.log*
 ResolvedawightT179095 Wheels built on ores-misc-01 are incompatible with ores* and scb*
 ResolvedawightT181806 Problem with Redis server configuration on new ORES cluster
 ResolvedHalfakT182799 Make sure ORES is compatible with stretch
 ResolvedHalfakT184072 Rebuild ORES models on Stretch
 ResolvedHalfakT184073 Provision a Stretch box we can use to build ORES models
 ResolvedakosiarisT184074 Verify that all enchant/spelling dictionaries are available on Stretch. Port if needed.
 ResolvedHalfakT184135 Rebuild ORES wheels on Stretch
 ResolvedHalfakT184296 Convert CloudVPS instances to stretch.
ResolvedHalfakT184766 Convert ores-misc-01 to stretch
 ResolvedSumitT184765 Back up ores-misc-01 to ores-staging-01

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
akosiaris added a comment.Aug 14 2017, 1:16 PM
In T171851#3478511, @Halfak wrote:

I just talked to @fgiunchedi about this in IRC and he said that reimaging while they aren't getting traffic makes sense. @akosiaris, how difficult is this?

Quite easy. But let's do T169246 first

Halfak added a comment.Aug 14 2017, 1:51 PM

\o/ Sounds good.

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Aug 16 2017, 5:26 PM
Halfak edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Aug 21 2017, 4:46 PM
awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Sep 6 2017, 5:18 PM
Halfak added a subtask: T169246: Stress/capacity test new ores* cluster.Oct 27 2017, 3:45 PM
awight subscribed.EditedOct 27 2017, 4:55 PM

@akosiaris We ran into a little glitch, that wheels created on ores-misc-01 aren't compatible with the new ores* cluster. I'm thinking we should reorder the dependencies, to reimage ores-misc-01 and this cluster before we finish stress testing. Are you okay with that?

Update: I've worked around the wheels problem, see T179095

Halfak added a parent task: T168073: Switch ORES to dedicated cluster.Oct 30 2017, 6:05 PM
awight closed subtask T169246: Stress/capacity test new ores* cluster as Resolved.Dec 6 2017, 8:04 PM
Halfak mentioned this in T169246: Stress/capacity test new ores* cluster.Dec 6 2017, 8:39 PM
awight reopened subtask T169246: Stress/capacity test new ores* cluster as Open.Dec 7 2017, 8:05 PM
akosiaris added a parent task: T165170: rack/setup/install ores2001-2009.Dec 8 2017, 1:10 PM
awight mentioned this in T182799: Make sure ORES is compatible with stretch.Dec 13 2017, 4:59 PM
awight closed subtask T169246: Stress/capacity test new ores* cluster as Resolved.Dec 14 2017, 9:22 PM
gerritbot subscribed.Dec 21 2017, 9:53 AM

Change 399596 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Reimage ores100* as stretch

https://gerrit.wikimedia.org/r/399596

gerritbot added a project: Patch-For-Review.Dec 21 2017, 9:53 AM

Change 399596 merged by Alexandros Kosiaris:
[operations/puppet@production] Reimage ores100* as stretch

https://gerrit.wikimedia.org/r/399596

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Jan 29 2018, 4:31 PM

@akosiaris, is this done?

Halfak added a parent task: T185901: Preliminary deployment of ORES to new cluster.Jan 29 2018, 4:33 PM
akosiaris added a comment.Jan 29 2018, 7:48 PM

It's half done (codfw but not eqiad). I 've been stalling it on T182799 so that we don't get hosts in a non-working state alerting and all. We could do eqiad as well and follow the codfw way of just not having the roles applied but that would not really buy us much.

Halfak added a comment.Jan 29 2018, 7:53 PM

Sorry for the confusion. T182799 is done. Will resolve.

akosiaris added a subtask: T182799: Make sure ORES is compatible with stretch.Jan 31 2018, 3:34 PM
gerritbot added a comment.Jan 31 2018, 3:38 PM

Change 407018 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Set ores1* as spare::system

https://gerrit.wikimedia.org/r/407018

gerritbot added a comment.Jan 31 2018, 3:40 PM

Change 407018 merged by Alexandros Kosiaris:
[operations/puppet@production] Set ores1* as spare::system

https://gerrit.wikimedia.org/r/407018

Stashbot subscribed.Jan 31 2018, 3:44 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T15:44:19Z] <akosiaris> reimage ores100{1..9} T171851

Halfak added a comment.Feb 2 2018, 10:46 PM

Just checked on the hosts and it seems they are still in progress. @akosiaris, are you still working on the re-imaging?

akosiaris added a comment.Feb 3 2018, 10:11 AM

Yes I am, but currently in fosdem so it 'll take a bit more. ETA is probably Tuesday

Halfak added a comment.Feb 5 2018, 4:15 PM

Works for me. Thanks :)

gerritbot added a comment.Feb 6 2018, 5:14 PM

Change 408558 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Remove ores::stresstest as its no longer needed

https://gerrit.wikimedia.org/r/408558

gerritbot added a comment.Feb 6 2018, 5:14 PM

Change 408559 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] ores: Set oresX00X hosts as role::ores

https://gerrit.wikimedia.org/r/408559

gerritbot added a comment.Feb 6 2018, 5:14 PM

Change 408560 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Remove ORES profile from scb

https://gerrit.wikimedia.org/r/408560

gerritbot added a comment.Feb 6 2018, 5:49 PM

Change 408564 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] ores: Allow oresX00X to reach respective oresrdb

https://gerrit.wikimedia.org/r/408564

gerritbot added a comment.Feb 7 2018, 12:05 PM

Change 408558 merged by Alexandros Kosiaris:
[operations/puppet@production] Remove ores::stresstest as its no longer needed

https://gerrit.wikimedia.org/r/408558

gerritbot added a comment.Feb 7 2018, 12:05 PM

Change 408564 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Allow oresX00X to reach respective oresrdb

https://gerrit.wikimedia.org/r/408564

gerritbot added a comment.Feb 7 2018, 12:05 PM

Change 408559 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Set oresX00X hosts as role::ores

https://gerrit.wikimedia.org/r/408559

gerritbot added a comment.Feb 7 2018, 12:58 PM

Change 408799 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add ores scap::dsh::groups

https://gerrit.wikimedia.org/r/408799

gerritbot added a comment.Feb 7 2018, 12:58 PM

Change 408799 merged by Alexandros Kosiaris:
[operations/puppet@production] Add ores scap::dsh::groups

https://gerrit.wikimedia.org/r/408799

Stashbot added a comment.Feb 7 2018, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:17:18Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 1:18 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:18:39Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 01m 22s)

Stashbot added a comment.Feb 7 2018, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:19:31Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:20:26Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 00m 55s)

Stashbot added a comment.Feb 7 2018, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:32:51Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:33:04Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 00m 06s)

Stashbot added a comment.Feb 7 2018, 1:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:34:09Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:35:30Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 01m 21s)

Stashbot added a comment.Feb 7 2018, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:35:51Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T13:38:35Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 02m 45s)

Stashbot added a comment.Feb 7 2018, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T14:03:47Z] <akosiaris@tin> Started deploy [ores/deploy@eb0f776]: T171851

Stashbot added a comment.Feb 7 2018, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-07T14:05:34Z] <akosiaris@tin> Finished deploy [ores/deploy@eb0f776]: T171851 (duration: 01m 47s)

gerritbot added a comment.Feb 7 2018, 3:17 PM

Change 408817 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Disable notification for role::ores

https://gerrit.wikimedia.org/r/408817

akosiaris reopened subtask T182799: Make sure ORES is compatible with stretch as Open.Feb 7 2018, 3:21 PM
akosiaris added a comment.Feb 7 2018, 3:25 PM

ORES deploy has failed (see T182799#3952787 and T184135#3873517) but the topic of this task, which is to reimage the hosts to stretch has been successfully completed so I am thinking I should resolve this

gerritbot added a comment.Feb 7 2018, 3:37 PM

Change 408817 merged by Alexandros Kosiaris:
[operations/puppet@production] Disable notification for role::ores

https://gerrit.wikimedia.org/r/408817

Halfak closed subtask T182799: Make sure ORES is compatible with stretch as Resolved.Feb 8 2018, 5:07 PM
Halfak added a comment.Feb 9 2018, 9:58 PM

I found this in our deploy repo.

P6677 (An Untitled Masterwork)
1commit 354bff862627e9dbf6f617a72d438a22a4d81912
2Author: Alexandros Kosiaris <akosiaris@wikimedia.org>
3Date: Wed Feb 7 13:15:02 2018 +0000
4
5 Update ORES production scap configuration
6
7 Remove the cluster server group and related stuff
8 Add a new scb server group, to be removed soon
9 Remove the ores-worker server group from production
10 Move the production configs under [wmnet] stanza
11 Add checks for the default group
12 Pass --no-download to virtualenv as it looks like it's required in
13 stretch
14
15 Bug: T171851

Not sure what is going on as this change was not submitted to gerrit AFAICT

Halfak added a comment.Feb 9 2018, 10:24 PM

OK I put all of the changes in "alex_stuff"

halfak@tin:/srv/deployment/ores/deploy$ git branch -l 
  CELERY_4
  STABLE
  STABLE_REVSCORING_1
  alex_stuff
* master
akosiaris added a comment.Feb 12 2018, 12:45 PM
In T171851#3959677, @Halfak wrote:

I found this in our deploy repo.

P6677 (An Untitled Masterwork)
1commit 354bff862627e9dbf6f617a72d438a22a4d81912
2Author: Alexandros Kosiaris <akosiaris@wikimedia.org>
3Date: Wed Feb 7 13:15:02 2018 +0000
4
5 Update ORES production scap configuration
6
7 Remove the cluster server group and related stuff
8 Add a new scb server group, to be removed soon
9 Remove the ores-worker server group from production
10 Move the production configs under [wmnet] stanza
11 Add checks for the default group
12 Pass --no-download to virtualenv as it looks like it's required in
13 stretch
14
15 Bug: T171851

Not sure what is going on as this change was not submitted to gerrit AFAICT

Yes, that was me trying to clear up a bit scap.cfg so that I could deploy to ores[12]00[1-9] and then other things preempting me and not managing to finish it. The size limit is required since it seems like tin can't keep up with that many processes trying simultaneously to clone from it. I will also look into increasing apache MaxRequestWorkers on tin itself, but having scap behave nicely as well wouldn't hurt.

akosiaris mentioned this in rORESDEPLOYc18e42251dbb: Remove the cluster server group and related stuff Add a new scb server group….Feb 12 2018, 4:23 PM
akosiaris mentioned this in rORESDEPLOYb2950478078e: Remove the cluster server group and related stuff Add a new scb server group….Feb 12 2018, 4:27 PM
akosiaris mentioned this in rORESDEPLOY8aaa0cf97af2: Remove the cluster server group and related stuff Add a new scb server group….Feb 12 2018, 4:29 PM
akosiaris mentioned this in rORESDEPLOY9d7482fc9adf: Remove the cluster server group and related stuff Add a new scb server group….Feb 12 2018, 4:33 PM
awight added a comment.Feb 12 2018, 4:34 PM

@akosiaris Do you know whether that group_size change will apply to rollback as well? We want the rollback to be as fast as possible...

akosiaris added a comment.Feb 12 2018, 4:39 PM

Yes it will. Which is why I am experimenting already with fetch_batch_size [1]

[1] https://github.com/wikimedia/scap/blob/master/docs/scap3/repo_config.rst

gerritbot added a comment.Feb 12 2018, 5:08 PM

Change 409932 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[mediawiki/services/ores/deploy@master] Remove the cluster server group and related stuff

https://gerrit.wikimedia.org/r/409932

akosiaris mentioned this in rORESDEPLOYf613e60e518b: Remove the cluster server group and related stuff.Feb 12 2018, 5:11 PM
gerritbot added a comment.Feb 12 2018, 5:31 PM

Change 409962 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add the ORES cluster to wikimedia_clusters

https://gerrit.wikimedia.org/r/409962

gerritbot added a comment.Feb 12 2018, 5:32 PM

Change 409962 merged by Alexandros Kosiaris:
[operations/puppet@production] Add the ORES cluster to wikimedia_clusters

https://gerrit.wikimedia.org/r/409962

Stashbot added a comment.Feb 12 2018, 5:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T17:34:17Z] <akosiaris@tin> Started deploy [ores/deploy@f7e23f4]: T171851

Stashbot added a comment.Feb 12 2018, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T17:47:35Z] <akosiaris@tin> Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 13m 18s)

Stashbot added a comment.Feb 12 2018, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:00:00Z] <akosiaris@tin> Started deploy [ores/deploy@f7e23f4]: T171851

Stashbot added a comment.Feb 12 2018, 6:12 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:12:30Z] <akosiaris@tin> Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 12m 30s)

Stashbot added a comment.Feb 12 2018, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:27:53Z] <akosiaris@tin> Started deploy [ores/deploy@f7e23f4]: T171851

Stashbot added a comment.Feb 12 2018, 6:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:34:39Z] <akosiaris@tin> Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 06m 47s)

Stashbot added a comment.Feb 12 2018, 6:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:35:55Z] <akosiaris@tin> Started deploy [ores/deploy@f7e23f4]: T171851

Stashbot added a comment.Feb 12 2018, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:48:08Z] <akosiaris@tin> Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 12m 14s)

Stashbot added a comment.Feb 12 2018, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:50:20Z] <akosiaris@tin> Started deploy [ores/deploy@f7e23f4]: T171851

Stashbot added a comment.Feb 12 2018, 6:57 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T18:57:59Z] <akosiaris@tin> Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 07m 39s)

Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Feb 13 2018, 4:56 PM

It looks like this is done. Is that right?

akosiaris closed this task as Resolved.Feb 14 2018, 8:31 AM
akosiaris claimed this task.

Yes that's right. Resolving

akosiaris mentioned this in rORESDEPLOY34c361233e70: Remove the cluster server group and related stuff.Feb 14 2018, 8:37 AM
gerritbot added a comment.Feb 22 2018, 11:07 AM

Change 408560 merged by Alexandros Kosiaris:
[operations/puppet@production] Remove ORES profile from scb

https://gerrit.wikimedia.org/r/408560

Stashbot mentioned this in T168073: Switch ORES to dedicated cluster.Feb 22 2018, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-22T11:39:33Z] <akosiaris> purge ORES from scb hosts T168073 T171851

gerritbot added a comment.Apr 12 2018, 12:11 PM

Change 409932 abandoned by Alexandros Kosiaris:
Remove the cluster server group and related stuff

Reason:
I think whatever changes I had done on this one have already been done in the past 2 months.

https://gerrit.wikimedia.org/r/409932

awight mentioned this in Blog Post: Status Update (May 2, 2018).May 2 2018, 6:43 PM
akosiaris mentioned this in rORESDEPLOY72a8cfe6a120: Create change.Jun 10 2018, 12:22 PM
akosiaris mentioned this in rORESDEPLOY57fbb41bd5f0: Create patch set 2.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits