Preliminary deployment of ORES to new cluster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Jan 29 2018, 4:32 PM

Details

	Subject	Repo	Branch	Lines +/-
	Remove the cluster server group and related stuff	mediawiki/services/ores/deploy	master	+14 -22

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T176324 Scoring platform team FY18 Q2
Declined	None	T179501 Use external dsh group to list pooled ORES nodes
Resolved	akosiaris	T168073 Switch ORES to dedicated cluster
Resolved	Halfak	T185901 Preliminary deployment of ORES to new cluster
Resolved	akosiaris	T171851 Reimage ores* hosts with Debian Stretch
Resolved	awight	T169246 Stress/capacity test new ores* cluster
Resolved	Halfak	T174402 Review and fix file handle management in worker and celery processes
Resolved	awight	T177036 Clean up file handle and Redis connection management in ORES worker and celery processes
Resolved	None	T175736 Give ores admins read access to /srv/log/ores/main.log*
Resolved	awight	T179095 Wheels built on ores-misc-01 are incompatible with ores* and scb*
Resolved	awight	T181806 Problem with Redis server configuration on new ORES cluster
Resolved	Halfak	T182799 Make sure ORES is compatible with stretch
Resolved	Halfak	T184072 Rebuild ORES models on Stretch
Resolved	Halfak	T184073 Provision a Stretch box we can use to build ORES models
Resolved	akosiaris	T184074 Verify that all enchant/spelling dictionaries are available on Stretch. Port if needed.
Resolved	Halfak	T184135 Rebuild ORES wheels on Stretch
Resolved	Halfak	T184296 Convert CloudVPS instances to stretch.
Resolved	Halfak	T184766 Convert ores-misc-01 to stretch
Resolved	Sumit	T184765 Back up ores-misc-01 to ores-staging-01

Event Timeline

Halfak triaged this task as High priority.Jan 29 2018, 4:32 PM

Halfak created this task.

Halfak added a subtask: T171851: Reimage ores* hosts with Debian Stretch.

Looks like we're still blocked on the re-imaging. Hopefully I can pick this up next week.

Halfak claimed this task.Feb 2 2018, 10:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:26:58Z] <halfak@tin> Started deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:36:57Z] <halfak@tin> Finished deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901 (duration: 09m 59s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:41:59Z] <halfak@tin> Started deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901 (trying again)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:42:39Z] <halfak@tin> Finished deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901 (trying again) (duration: 00m 40s)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:47:13Z] <halfak@tin> Started deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901 (trying again again)

Mentioned in SAL (#wikimedia-operations) [2018-02-09T22:47:28Z] <halfak@tin> Finished deploy [ores/deploy@c98ec8b]: (non-production) experimenting with stretch deploy T185901 (trying again again) (duration: 00m 03s)

Posted this in #wikimedia-releng:

[16:48:53] <halfak> So, I just did a scap deploy that failed with a stream of "Timeout, server ores100*.eqiad.wmnet not responding." 
[16:49:07] <halfak> I checked and the ores hosts are online.  So I just re-tried. 
[16:49:25] <halfak> Now the scap just rockets through the deploy claiming to finish very quickly. 
[16:49:28] <halfak> But nothing is happening.

Looks like this is now done.

Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Feb 13 2018, 4:54 PM

Yes, this is done

Change 409932 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[mediawiki/services/ores/deploy@master] Remove the cluster server group and related stuff

https://gerrit.wikimedia.org/r/409932

gerritbot added a project: Patch-For-Review.Feb 14 2018, 8:33 AM

akosiaris mentioned this in rORESDEPLOY34c361233e70: Remove the cluster server group and related stuff.Feb 14 2018, 8:37 AM

Change 409932 abandoned by Alexandros Kosiaris:
Remove the cluster server group and related stuff

Reason:
I think whatever changes I had done on this one have already been done in the past 2 months.

https://gerrit.wikimedia.org/r/409932

awight mentioned this in Unknown Object (Phame Post).May 2 2018, 6:41 PM

awight mentioned this in Unknown Object (Phame Post).

awight mentioned this in Blog Post: Status Update (May 2, 2018).