Re-create ci slaves (March 2015)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Mar 4 2015, 5:39 PM

Description

~~integration-slave120x (precise)~~ and integration-140x (trusty) will be created.
~~integration-slave100x (precise)~~ and integration-101x (trusty) will be destroyed.

Notable this time:

The base image in labs has changed and now consolidates / and /var. Thus giving logs more breathing room. As well as other minor issues.
Lots of stuff weird crazy on our instances with untracked and forgotten live hacks.
Upstream puppet often breaks things that are not discovered until a new instance is created (e.g. permissions changes).
Puppet manifests are sometimes badly written and only tested on existing instances.

As always, document stuff at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup

Due to unresolved regressions, precise slaves will be skipping this round.

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T91524 Re-create ci slaves (March 2015)
Resolved	hashar	T90984 "/usr/local/bin/zuul-cloner" broken on new instances
Duplicate	hashar	T85969 On all slaves, /srv/deployment/integration/slave-scripts permissions went crazy
Resolved	akosiaris	T91525 "/usr/local/bin/tox: Permission denied" on integration-slave14xx instances
Resolved	hashar	T91832 Puppet (silently) fails to setup apache on new trusty instances
Resolved	coren	T92351 dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN
Declined	coren	T105297 Reenable resolv.conf ndots:2 option on CI instances
Resolved	Krinkle	T91526 Fix "Entry point ('console_scripts', 'tox') not found" on new slaves running Ubuntu Precise

Event Timeline

Krinkle created this task.Mar 4 2015, 5:39 PM

Krinkle claimed this task.

Krinkle raised the priority of this task from to High.

Krinkle updated the task description. (Show Details)

Krinkle added a project: Continuous-Integration-Infrastructure.

Krinkle subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2015, 5:39 PM

Krinkle added a subtask: T90984: "/usr/local/bin/zuul-cloner" broken on new instances.Mar 4 2015, 5:39 PM

Krinkle added a subtask: T85969: On all slaves, /srv/deployment/integration/slave-scripts permissions went crazy.

Krinkle renamed this task from Re-create integration slaves to Pool new integration-slave12xx and integration-slave14xx instances and delete old ones.Mar 4 2015, 5:43 PM

Krinkle updated the task description. (Show Details)

Krinkle set Security to None.

Krinkle updated the task description. (Show Details)

akosiaris closed subtask T91525: "/usr/local/bin/tox: Permission denied" on integration-slave14xx instances as Resolved.Mar 5 2015, 6:25 PM

Blocking issues for Ubuntu Trusty resolved. Our instance creation process as documented now works cleanly. New instances were created and pooled over the course of last week and the old instances deleted as of March 8.

https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL

Krinkle moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Mar 11 2015, 3:30 PM

Krinkle added subtasks: T91832: Puppet (silently) fails to setup apache on new trusty instances, T92351: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN.Mar 14 2015, 1:09 AM

coren closed subtask T92351: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN as Resolved.Mar 26 2015, 1:48 PM

scfc changed the status of subtask T92351: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN from Resolved to Declined.Mar 26 2015, 11:26 PM

Krinkle mentioned this in T94138: /var/lib/mysql/ filling up on old Precise slaves due to mysql usage.Mar 29 2015, 7:55 AM

Krinkle renamed this task from Pool new integration-slave14xx instances and delete old ones to Re-create ci slaves (March 2015).Apr 2 2015, 10:02 PM

Krinkle updated the task description. (Show Details)

Krinkle added a subtask: T91526: Fix "Entry point ('console_scripts', 'tox') not found" on new slaves running Ubuntu Precise.Apr 7 2015, 2:34 PM

Krinkle reopened subtask T92351: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN as Open.Apr 14 2015, 3:20 PM

Krinkle closed subtask T92351: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN as Resolved.Apr 14 2015, 3:56 PM

Krinkle closed subtask T91526: Fix "Entry point ('console_scripts', 'tox') not found" on new slaves running Ubuntu Precise as Resolved.