Convert CloudVPS instances to stretch.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Jan 5 2018, 8:46 PM

Description

ores.wmflabs.org
ores-staging.wmflabs.org

Related Objects
Search...

Status	Assigned	Task
Resolved	akosiaris	T162039 Prepare to service applications from kubernetes
Resolved	akosiaris	T162041 Expand the infrastructure to codfw
Resolved	RobH	T161700 CODFW: (4) hardware access request for kubernetes
Resolved	RobH	T142578 codfw/eqiad:(9+9) hardware access request for ORES
		Unknown Object (Task)
Resolved	akosiaris	T165170 rack/setup/install ores2001-2009
Resolved	None	T176324 Scoring platform team FY18 Q2
Declined	None	T179501 Use external dsh group to list pooled ORES nodes
Resolved	akosiaris	T168073 Switch ORES to dedicated cluster
Resolved	Halfak	T185901 Preliminary deployment of ORES to new cluster
Resolved	akosiaris	T171851 Reimage ores* hosts with Debian Stretch
Resolved	awight	T186238 Migrate Scoring services to Debian Stretch
Resolved	Halfak	T182799 Make sure ORES is compatible with stretch
Resolved	Halfak	T184296 Convert CloudVPS instances to stretch.
Resolved	Halfak	T184766 Convert ores-misc-01 to stretch
Resolved	Sumit	T184765 Back up ores-misc-01 to ores-staging-01

Event Timeline

ores-staging-01 is back online as a stretch instance.

Halfak claimed this task.Jan 5 2018, 8:55 PM

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Working on this now :|

halfak@ores-staging-01:~$ virtualenv
-bash: virtualenv: command not found

So in staging, we import base. And in base we require 'virtualenv'. Hmm... I'm checking on puppet now.

Refs:

Running:

halfak@ores-staging-01:~$ sudo puppet agent -tv
Warning: Setting configtimeout is deprecated. 
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1146:in `issue_deprecation_warning')
Notice: Starting Puppet client version 4.8.2
...

Looks like I forgot to enable role::labs::ores::staging. Thought I'd clicked that :/

Now I'm looking at:

[ores-staging-01.eqiad.wmflabs] sudo: /srv/ores/venv/bin/pip install --use-wheel --no-deps /srv/ores/config/submodules/wheels/pip-*.whl
[ores-staging-01.eqiad.wmflabs] out: Requirement '/srv/ores/config/submodules/wheels/pip-*.whl' looks like a filename, but the file does not exist
[ores-staging-01.eqiad.wmflabs] out: pip-*.whl is not a valid wheel filename.
[ores-staging-01.eqiad.wmflabs] out:

So... what? Looks like we're supposed to include pip in our wheels.

And we're online! See https://ores-staging.wmflabs.org/v3/scores/fiwiki/1241

But it's returning an error. It seems like we have the wrong version of celery running.

{
  "fiwiki": {
    "models": {
      "damaging": {
        "version": "0.4.1"
      },
      "goodfaith": {
        "version": "0.4.1"
      }
    },
    "scores": {
      "1241": {
        "damaging": {
          "error": {
            "message": "Never call result.get() within a task!\nSee http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks\n",
            "type": "RuntimeError"
          }
        },
        "goodfaith": {
          "error": {
            "message": "Never call result.get() within a task!\nSee http://docs.celeryq.org/en/latest/userguide/tasks.html#task-synchronous-subtasks\n",
            "type": "RuntimeError"
          }
        }
      }
    }
  }
}

IT'S ALIVE https://ores-staging.wmflabs.org/v3/scores/fiwiki/1241241

Woops. I didn't do the main cluster. Leaving that for later.

OK so I'm thinking of just taking wmflabs cluster down half-way, installing, then doing the other half. Doesn't seem too crazy.

Halfak added a subtask: T184766: Convert ores-misc-01 to stretch.Jan 11 2018, 7:38 PM

ores-web-01 is created and configured, but not pooled. It seems to work fine to request scores from this node and it is able to talk to celery as planned.

Done! https://ores.wmflabs.org/v3/scores/enwiki/32456784/wp10?features

Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Jan 17 2018, 9:39 PM

awight mentioned this in Blog Post: Status Update (January 30, 2018).Jan 30 2018, 7:03 PM

Halfak closed this task as Resolved.Jan 30 2018, 8:30 PM

Halfak closed subtask T184766: Convert ores-misc-01 to stretch as Resolved.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu