Page MenuHomePhabricator
Create Task
Maniphest T111377

nodepool can't update images
Closed, ResolvedPublic
Actions

Description

2015-09-03 15:22:56,272 ERROR nodepool.NodePool: Could not update image ci-jessie-wikimedia on wmflabs-eqiad
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nodepool/nodepool.py", line 1968, in updateImage
    self._updateImage(session, provider_name, image_name)
  File "/usr/lib/python2.7/dist-packages/nodepool/nodepool.py", line 1984, in _updateImage
    "Invalid image config. Must specify either "
Exception: Invalid image config. Must specify either a setup script, or a diskimage to use.

The nodepool configuration file lacks a setup: setup.sh under providers / ci-jessie-wikimedia.

The disk image builder element will need to populate it (T110733) and we need it under /etc/nodepool/scripts on the machine.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
nodepool 0.1.1-wmf4operations/debs/nodepooldebian+176 -1
0.1.1-wmf3: Use 'debian' user for setup stepoperations/debs/nodepooldebian+38 -0
nodepool: setup scripts are in integration/configoperations/puppetproduction+12 -2
Revert "nodepool: setup scripts are in integration/config"operations/puppetproduction+2 -10
nodepool: setup scripts are in integration/configoperations/puppetproduction+10 -2
nodepool: run ready.sh when finishing instancesoperations/puppetproduction+3 -1
nodepool: set instance hostname via sudointegration/configmaster+16 -2
nodepool: placeholder setup scriptintegration/configmaster+13 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Sep 3 2015, 3:25 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 3 2015, 3:25 PM
hashar mentioned this in T110733: Wrap diskimage-builder for easy image creations.Sep 3 2015, 3:26 PM
Krenair added projects: Nodepool, Continuous-Integration-Scaling.Sep 3 2015, 3:46 PM
Krenair subscribed.
hashar triaged this task as High priority.Sep 7 2015, 8:50 PM

Thank you @Krenair

hashar claimed this task.Sep 7 2015, 8:50 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
gerritbot subscribed.Sep 8 2015, 12:15 PM

Change 236769 had a related patch set uploaded (by Hashar):
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/236769

gerritbot added a project: Patch-For-Review.Sep 8 2015, 12:15 PM

Change 236770 had a related patch set uploaded (by Hashar):
nodepool: placeholder setup script

https://gerrit.wikimedia.org/r/236770

gerritbot added a comment.Sep 8 2015, 12:26 PM

Change 236770 merged by jenkins-bot:
nodepool: placeholder setup script

https://gerrit.wikimedia.org/r/236770

hashar mentioned this in rCICFe22ab133bec3: nodepool: placeholder setup script.Sep 8 2015, 12:28 PM
gerritbot added a comment.Sep 8 2015, 1:32 PM

Change 236776 had a related patch set uploaded (by Hashar):
nodepool: run ready.sh when finishing instances

https://gerrit.wikimedia.org/r/236776

gerritbot added a comment.Sep 8 2015, 1:32 PM

Change 236775 had a related patch set uploaded (by Hashar):
nodepool: set instance hostname via sudo

https://gerrit.wikimedia.org/r/236775

gerritbot added a comment.Sep 8 2015, 1:45 PM

Change 236775 merged by jenkins-bot:
nodepool: set instance hostname via sudo

https://gerrit.wikimedia.org/r/236775

hashar mentioned this in rCICFa4bb9d24df79: nodepool: set instance hostname via sudo.Sep 8 2015, 1:45 PM
gerritbot added a comment.Sep 8 2015, 9:01 PM

Change 236769 merged by Andrew Bogott:
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/236769

Andrew mentioned this in rOPUPb99563b343b0: nodepool: setup scripts are in integration/config.Sep 8 2015, 9:02 PM
Andrew mentioned this in rOPUP3020e9eb7dd8: nodepool: run ready.sh when finishing instances.
gerritbot added a comment.Sep 9 2015, 9:33 AM

Change 237062 had a related patch set uploaded (by Hashar):
Revert "nodepool: setup scripts are in integration/config"

https://gerrit.wikimedia.org/r/237062

gerritbot added a comment.Sep 9 2015, 9:42 AM

Change 237065 had a related patch set uploaded (by Hashar):
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/237065

gerritbot added a comment.Sep 9 2015, 2:01 PM

Change 237062 merged by Andrew Bogott:
Revert "nodepool: setup scripts are in integration/config"

https://gerrit.wikimedia.org/r/237062

Andrew mentioned this in rOPUPfb07206fde22: Revert "nodepool: setup scripts are in integration/config".Sep 9 2015, 2:01 PM
gerritbot added a comment.Sep 9 2015, 2:02 PM

Change 237065 merged by Andrew Bogott:
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/237065

Andrew mentioned this in rOPUP0d16246b00ae: nodepool: setup scripts are in integration/config.Sep 9 2015, 2:03 PM
gerritbot added a comment.Sep 10 2015, 2:21 PM

Change 237380 had a related patch set uploaded (by Hashar):
0.1.1-wmf3: Use 'debian' user for setup step

https://gerrit.wikimedia.org/r/237380

hashar added a comment.EditedSep 10 2015, 2:24 PM

I have found another oddity. When Nodepool refresh an image (setup step) it attempts to connect as root and on failure iterates over well known cloud-users. Diskimage-builder creates a debian user which is not looked at by Nodepool. Upstream patch is https://review.openstack.org/#/c/221873/ . https://gerrit.wikimedia.org/r/237380 is the Nodepool.

Filled T112100: Upload nodepool_0.1.1-wmf3 to apt.wikimedia.org and upgrade package on labnodepool1001.eqiad.wmnet

gerritbot added a comment.Sep 10 2015, 3:04 PM

Change 237380 merged by Hashar:
0.1.1-wmf3: Use 'debian' user for setup step

https://gerrit.wikimedia.org/r/237380

hashar mentioned this in rODEV8382610ad569: 0.1.1-wmf3: Use 'debian' user for setup step.Sep 10 2015, 3:24 PM
gerritbot added a comment.Sep 11 2015, 4:32 PM

Change 237700 had a related patch set uploaded (by Hashar):
0.1.1-wmf4: image creation chokes on /etc/nodepool

https://gerrit.wikimedia.org/r/237700

Andrew closed subtask T112100: Upload nodepool_0.1.1-wmf3 to apt.wikimedia.org and upgrade package on labnodepool1001.eqiad.wmnet as Resolved.Sep 11 2015, 4:43 PM
hashar closed this task as Resolved.Sep 11 2015, 9:16 PM

I made a few more changes, but in short it works now. Nodepool boot an instance from the provided image, runs the setup script and snapshot it for later usage.

hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.Sep 19 2015, 10:32 AM
greg added a project: Essential-Work.Sep 21 2015, 8:53 PM
hashar mentioned this in T118573: Upgrade Nodepool to 0.1.1-wmf4.Nov 13 2015, 5:08 PM
hashar added a subtask: T118573: Upgrade Nodepool to 0.1.1-wmf4.
hashar changed the status of subtask T118573: Upgrade Nodepool to 0.1.1-wmf4 from Open to Stalled.
hashar changed the status of subtask T118573: Upgrade Nodepool to 0.1.1-wmf4 from Stalled to Open.Feb 29 2016, 7:49 PM
gerritbot added a comment.Mar 7 2016, 8:23 PM

Change 237700 merged by Hashar:
nodepool 0.1.1-wmf4

https://gerrit.wikimedia.org/r/237700

hashar closed subtask T118573: Upgrade Nodepool to 0.1.1-wmf4 as Resolved.Mar 7 2016, 8:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits