Page MenuHomePhabricator
Create Task
Maniphest T111925

Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet
Closed, DeclinedPublic
Actions

Description

I wanted to use git-deploy to push integration/config.git from tin to labnodepool1001.eqiad.wmnet: https://gerrit.wikimedia.org/r/#/c/236769/ which is:

package { 'integration/config':
    provider => 'trebuchet',
    before   => Class['::nodepool'],
}

The labnodepool1001 is on the labs host subnet and it can't reach tin nor tin can reach it. Thus puppet kindly dies with:

redis.exceptions.ConnectionError: Error 110 connecting to tin.eqiad.wmnet:6379. Connection timed out

Trace:

Info: Caching catalog for labnodepool1001.eqiad.wmnet
Error: Execution of '/usr/bin/salt-call --log-level=quiet --out=json deploy.fetch integration/config' returned 1: Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 123, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 422, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 227, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 129, in call
    ret['return'] = func(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 494, in fetch
    _check_in('deploy.fetch', repo)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 44, in _check_in
    serv.sadd('deploy:repos', repo)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1421, in sadd
    return self.execute_command('SADD', name, *values)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 534, in execute_command
    connection.send_command(*args)
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 532, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 508, in send_packed_command
    self.connect()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 412, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 110 connecting to tin.eqiad.wmnet:6379. Connection timed out.
Error: /Stage[main]/Role::Nodepool/Package[integration/config]/ensure: change from purged to present failed: Execution of '/usr/bin/salt-call --log-level=quiet --out=json deploy.fetch integration/config' returned 1: Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 123, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 422, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 227, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 129, in call
    ret['return'] = func(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 494, in fetch
    _check_in('deploy.fetch', repo)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 44, in _check_in
    serv.sadd('deploy:repos', repo)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1421, in sadd
    return self.execute_command('SADD', name, *values)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 534, in execute_command
    connection.send_command(*args)
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 532, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 508, in send_packed_command
    self.connect()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 412, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 110 connecting to tin.eqiad.wmnet:6379. Connection timed out.
Notice: /Package[libguestfs-tools]: Dependency Package[integration/config] has failures: true
Warning: /Package[libguestfs-tools]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Apt::Pin[python-openstackclient]/File[/etc/apt/preferences.d/python-openstackclient.pref]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Apt::Pin[python-openstackclient]/File[/etc/apt/preferences.d/python-openstackclient.pref]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/usr/bin/nodepool-graceful-stop]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/usr/bin/nodepool-graceful-stop]: Skipping because of failed dependencies
Notice: /Package[uuid-runtime]: Dependency Package[integration/config] has failures: true
Warning: /Package[uuid-runtime]: Skipping because of failed dependencies
Notice: /Package[python-openstackclient]: Dependency Package[integration/config] has failures: true
Warning: /Package[python-openstackclient]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Exec[create_dib_base_path]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Exec[create_dib_base_path]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Apt::Pin[python-novaclient]/File[/etc/apt/preferences.d/python-novaclient.pref]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Apt::Pin[python-novaclient]/File[/etc/apt/preferences.d/python-novaclient.pref]: Skipping because of failed dependencies
Notice: /Stage[main]/Apt/Exec[apt-get update]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Apt/Exec[apt-get update]: Skipping because of failed dependencies
Notice: /Package[nodepool]: Dependency Package[integration/config] has failures: true
Warning: /Package[nodepool]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/srv/dib]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/srv/dib]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/srv/dib/images]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/srv/dib/images]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/srv/dib/tmp]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/srv/dib/tmp]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/etc/nodepool/nodepool.yaml]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/etc/nodepool/nodepool.yaml]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/srv/dib/cache]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/srv/dib/cache]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/etc/nodepool/logging.conf]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/etc/nodepool/logging.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/File[/etc/systemd/system/nodepool.service]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/File[/etc/systemd/system/nodepool.service]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/File[/lib/systemd/system/nodepool.service]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/File[/lib/systemd/system/nodepool.service]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/Exec[systemd reload for nodepool]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/Exec[systemd reload for nodepool]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/Service[nodepool]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/Base::Service_unit[nodepool]/Service[nodepool]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/var/lib/nodepool/.profile]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/var/lib/nodepool/.profile]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh/dib_jenkins_id_rsa.pub]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh/dib_jenkins_id_rsa.pub]: Skipping because of failed dependencies
Notice: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh/dib_jenkins_id_rsa]: Dependency Package[integration/config] has failures: true
Warning: /Stage[main]/Nodepool/File[/var/lib/nodepool/.ssh/dib_jenkins_id_rsa]: Skipping because of failed dependencies
Notice: Finished catalog run in 527.80 seconds

The workaround is to use git::clone and update the repo manually on deployment.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
nodepool: setup scripts are in integration/configoperations/puppetproduction+12 -2
Revert "nodepool: setup scripts are in integration/config"operations/puppetproduction+2 -10
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Sep 9 2015, 9:28 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: Deployments, Salt, Continuous-Integration-Scaling, Cloud-VPS, Cloud-Services.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2015, 9:28 AM
gerritbot subscribed.Sep 9 2015, 9:33 AM

Change 237062 had a related patch set uploaded (by Hashar):
Revert "nodepool: setup scripts are in integration/config"

https://gerrit.wikimedia.org/r/237062

gerritbot added a project: Patch-For-Review.Sep 9 2015, 9:33 AM
gerritbot added a comment.Sep 9 2015, 9:42 AM

Change 237065 had a related patch set uploaded (by Hashar):
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/237065

gerritbot added a comment.Sep 9 2015, 2:01 PM

Change 237062 merged by Andrew Bogott:
Revert "nodepool: setup scripts are in integration/config"

https://gerrit.wikimedia.org/r/237062

Andrew mentioned this in rOPUPfb07206fde22: Revert "nodepool: setup scripts are in integration/config".Sep 9 2015, 2:01 PM
gerritbot added a comment.Sep 9 2015, 2:02 PM

Change 237065 merged by Andrew Bogott:
nodepool: setup scripts are in integration/config

https://gerrit.wikimedia.org/r/237065

Andrew mentioned this in rOPUP0d16246b00ae: nodepool: setup scripts are in integration/config.Sep 9 2015, 2:03 PM
hashar added subscribers: Andrew, yuvipanda.Sep 22 2015, 6:31 PM

@Andrew and @yuvipanda i discovered we can not git-deploy from the deployment server (tin) to hosts in the lab subnet. My specific use case is worked around by git pulling on the host in lab. That is manageable since there is a single host and a single repo. So I can close this task.

The reason I am leaving it open is to ask whether we have other use cases for using git-deploy to labs hosts. In this case I guess I will fill a new task against Deployments. and abandon this one.

Thx!

Krenair subscribed.Oct 11 2015, 3:09 AM
hashar closed this task as Declined.Oct 27 2015, 2:31 PM
hashar claimed this task.

Solved by using a git clone directly where needed.

hashar moved this task from Backlog to Done on the Continuous-Integration-Scaling board.Nov 2 2015, 1:43 PM
greg added a project: OKR-Work.Jan 11 2016, 10:57 PM
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptJan 11 2016, 10:57 PM
mschwarzer subscribed.Jan 25 2017, 3:17 PM

@hashar

Hi. I'm having the same problem, when using Puppet to install role::elasticsearch::cirrus on a labs instance (hadoop000.math.eqiad.wmflabs). What should I git clone? Or is there any other work-around?

hashar added a comment.Jan 25 2017, 3:58 PM

context

Nodepool is a python software running on labnodepool1001.eqiad.wmnet. The software needs bits in the integration/config.git repository. Trebuchet worked by contacting the deployment server tin.eqiad.wmnet which is a different network and is not reachable. I ended up using simply git clone which is for a single server administered by only a handful of people was good enough.

role::elasticsearch::cirrus

That class invokes role::elasticsearch::common which ends up with:

modules/role/manifests/elasticsearch/common.pp
package { 'elasticsearch/plugins':
    provider => 'trebuchet',
}

Which mean that to deploy the plugins you would need a Trebuchet deployment server setup in the math labs project.

Long term fix

Trebuchet is legacy and all systems still using it are being migrated to the deployment system Scap. The task to migrate the elastic search plugins to it is T151996. Though since it is only updated once in a while by just two or three people, it is not much of a priority.

Short term

I believe the easiest to unblock you is to just clone the repository. But you would probably have troubles with the puppet manifest asking for provider => trebuchet. Though maybe when the repo is under /srv/deployment/elasticsearch/plugins Trebuchet will happily skip loading and puppet will continue.

Be awesome

If you really want to be the most awesome person of the month, you could attempt to migrate the deployment to use Scap via the migration guide https://wikitech.wikimedia.org/wiki/Scap3/Migration_Guide . But there are a bunch of prerequisites to add to the labs math project before it is possible.

Hope that clarify. Maybe fill another task copy pasting the above and we can subscribe the ElasticSearch people to it so they can assist ?

hashar mentioned this in T151996: Deploy elasticsearch plugins with scap3 (Trebuchet elasticsearch/plugins).Jan 25 2017, 3:59 PM
greg subscribed.Jan 25 2017, 6:06 PM

To be clear, git-deploy/trebuchet is deprecated as of March 11th, 2016. https://lists.wikimedia.org/pipermail/engineering/2016-March/000112.html

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:47 PM
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:47 PM
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban), Cloud-Services.Sep 26 2017, 11:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits