latest salt-cloud not installing minion during deployment

Discussion:

numb3rs1x

2013-11-22 21:17:05 UTC

I've been advised that salt in the develop branch was fixed with
salt-cloud. I can now deploy instances using salt-cloud, but the deployed
instance does not install salt-minion, although it show that a key has been
generated between the pair and excepted. The debug logs don't show any
errors, so I'm not sure if the process stops at some point or if there's
something in the networking that's wrong. I'm able to install salt-minion
on the new instance and make a connection to the master manually, but that
defeats the purpose of salt-cloud, which lately has been defeating its
purpose all on its own. :/

2013-11-22 16:05:28,023 [salt.loaded.int.cloud.ec2][INFO ] Created Cloud
VM 'vpc-instance'
2013-11-22 16:06:13,409 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:06:13,446 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:06:13,448 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:06:13,448 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:06:13,450 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:07:13,516 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:07:13,554 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:07:13,556 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:07:13,556 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:07:13,558 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:08:13,623 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:08:13,663 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:08:13,664 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:08:13,664 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:08:13,666 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:08:34,562 [salt.master
][INFO ] Clear payload received with command publish
2013-11-22 16:08:34,564 [salt.master
][INFO ] User sudo_ec2-user Published command test.ping with jid
20131122160834562412
2013-11-22 16:08:34,564 [salt.master
][DEBUG ] Published command details {'tgt_type': 'glob', 'jid':
'20131122160834562412', 'tgt': 'vpc-instance', 'ret': '', 'user':
'sudo_ec2-user', 'arg': [], 'fun': 'test.ping'}
2013-11-22 16:09:13,731 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:09:13,770 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:09:13,771 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:09:13,772 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:09:13,774 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:10:13,833 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:10:13,871 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:10:13,872 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:10:13,872 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:10:13,874 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:11:13,939 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:11:13,978 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:11:13,980 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:11:13,980 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:11:13,982 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:12:14,047 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:12:14,090 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:12:14,092 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:12:14,092 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:12:14,094 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:13:14,159 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:13:14,197 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:13:14,198 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:13:14,199 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:13:14,201 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:14:14,265 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:14:14,303 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:14:14,305 [salt.utils.event
][DEBUG ] MasterEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:14:14,305 [salt.utils.event
][DEBUG ] MasterEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:14:14,307 [salt.master
][DEBUG ] ckminions.connected_ids: []

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

David Boucha

2013-11-22 23:11:02 UTC

Permalink

Can you pastebin a sanitized copy of your cloud.profiles file?

Post by numb3rs1x
I've been advised that salt in the develop branch was fixed with
salt-cloud. I can now deploy instances using salt-cloud, but the deployed
instance does not install salt-minion, although it show that a key has been
generated between the pair and excepted. The debug logs don't show any
errors, so I'm not sure if the process stops at some point or if there's
something in the networking that's wrong. I'm able to install salt-minion
on the new instance and make a connection to the master manually, but that
defeats the purpose of salt-cloud, which lately has been defeating its
purpose all on its own. :/
2013-11-22 16:05:28,023 [salt.loaded.int.cloud.ec2][INFO ] Created
Cloud VM 'vpc-instance'
2013-11-22 16:06:13,409 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:06:13,446 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:06:13,448 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:06:13,448 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:06:13,450 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:07:13,516 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:07:13,554 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:07:13,556 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:07:13,556 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:07:13,558 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:08:13,623 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:08:13,663 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:08:13,664 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:08:13,664 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:08:13,666 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:08:34,562 [salt.master
][INFO ] Clear payload received with command publish
2013-11-22 16:08:34,564 [salt.master
][INFO ] User sudo_ec2-user Published command test.ping with jid
20131122160834562412
2013-11-22 16:08:34,564 [salt.master
'sudo_ec2-user', 'arg': [], 'fun': 'test.ping'}
2013-11-22 16:09:13,731 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:09:13,770 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:09:13,771 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:09:13,772 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:09:13,774 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:10:13,833 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:10:13,871 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:10:13,872 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:10:13,872 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:10:13,874 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:11:13,939 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:11:13,978 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:11:13,980 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:11:13,980 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:11:13,982 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:12:14,047 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:12:14,090 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:12:14,092 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:12:14,092 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:12:14,094 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:13:14,159 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:13:14,197 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:13:14,198 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:13:14,199 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:13:14,201 [salt.master
][DEBUG ] ckminions.connected_ids: []
2013-11-22 16:14:14,265 [salt.fileserver
][DEBUG ] Updating fileserver cache
2013-11-22 16:14:14,303 [salt.fileserver
][DEBUG ] diff_mtime_map: the maps are the same
2013-11-22 16:14:14,305 [salt.utils.event
ipc:///var/run/salt/master/master_event_pub.ipc
2013-11-22 16:14:14,305 [salt.utils.event
ipc:///var/run/salt/master/master_event_pull.ipc
2013-11-22 16:14:14,307 [salt.master
][DEBUG ] ckminions.connected_ids: []

numb3rs1x

2013-11-23 00:16:30 UTC

Permalink

http://pastebin.com/6Rte3N4D

numb3rs1x

2013-11-23 00:30:24 UTC

Permalink

Also, here is an example of console output, also sanitized.

http://pastebin.com/pykG5bmt

numb3rs1x

2013-11-23 00:35:23 UTC

Permalink

As well as a sanitized version of my cloud.profiles for good measure.

http://pastebin.com/66rfja5e

Joseph Hall

2013-11-23 13:45:43 UTC

Permalink

Thank you for those pastebins, numb3rs1x, they were incredibly helpful.

It looks like everything but the deploy script made it up there. The
next troubleshooting steps I would try are:

- Check the rest of your salt-cloud configs (including your map and
the /etc/salt/cloud file) and make sure there are no occurrences of
the `os` or `script` setting.
- Run the following command to ensure you have the latest bootstrap
script: salt-cloud -u
- Run your command again, but with the --keep-tmp option.
- Once the instance is spun up, look inside of /tmp/ and
/tmp/.saltcloud/. If the deploy script ran successfully, there should
be a log file.
- Look at /tmp/.saltcloud/deploy.sh. Make sure that /tmp/.saltcloud/
and /tmp/.saltcloud/deploy.sh are owned by the correct user, and have
proper execute permissions.
- Attempt to execute /tmp/.saltcloud/deploy.sh manually. Note any errors.

Post by numb3rs1x
As well as a sanitized version of my cloud.profiles for good measure.
http://pastebin.com/66rfja5e
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)
--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

numb3rs1x

2013-11-24 07:34:02 UTC

Permalink

sudo salt-cloud -u
Updating bootstrap-salt.sh.
Source: http://bootstrap.saltstack.org
Destination:
/usr/lib/python2.7/site-packages/salt/deploy/bootstrap-salt.sh
Usage: salt-cloud

salt-cloud: error: Failed to update the bootstrap script

numb3rs1x

2013-11-24 07:40:04 UTC

Permalink

- Run your command again, but with the --keep-tmp option.

Post by Joseph Hall
- Once the instance is spun up, look inside of /tmp/ and
/tmp/.saltcloud/. If the deploy script ran successfully, there should
be a log file.
sudo salt-cloud -m /etc/salt/salt_cloud_map --keep-tmp

[INFO ] salt-cloud starting
[INFO ] Applying map from '/etc/salt/salt_cloud_map'.
The following virtual machines are set to be created:

ls -la /tmp/ | grep salt
[ec2-***@ip-10-x-x-xx ~]$

Pedro Algarvio

2013-11-24 10:57:05 UTC

Permalink

Post by Joseph Hall
- Run your command again, but with the --keep-tmp option.
- Once the instance is spun up, look inside of /tmp/ and
/tmp/.saltcloud/. If the deploy script ran successfully, there should
be a log file.
sudo salt-cloud -m /etc/salt/salt_cloud_map --keep-tmp
[INFO ] salt-cloud starting
[INFO ] Applying map from '/etc/salt/salt_cloud_map'.
ls -la /tmp/ | grep salt
--
You received this message because you are subscribed to the Google
Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/groups/opt_out.

Could you update to latest develop and retry please?

Best Regards,
Pedro Algarvio.

numb3rs1x

2013-11-24 18:10:36 UTC

Permalink

Same results.

salt --version
salt 0.17.0-4959-gfd99a2c
[ec2-***@ip-10-0-0-11 salt]$ salt-cloud --version
salt-cloud 0.17.0-4959-gfd99a2c

numb3rs1x

2013-11-24 23:05:22 UTC

Permalink

Just to be clear, I ran another git pull and then setup.py install. I then
ran salt-cloud -u and came up with the same results. I think I see the
problem, but I don't know enough about debugging python code that is
designed to create binaries to execute to prove it.

I think it has to do with the 302 response that's being received from the
default URL that contains the bootstrap.

if self.options.update_bootstrap:
import urllib2
url = 'http://bootstrap.saltstack.org'
req = urllib2.urlopen(url)
if req.getcode() != 200:
self.error(
'Failed to download the latest stable version of the '
'bootstrap-salt.sh script from {0}. HTTP error: '
'{1}'.format(
url, req.getcode()
)
)

wget http://bootstrap.saltstack.org
--2013-11-24 18:05:07-- http://bootstrap.saltstack.org/
Resolving bootstrap.saltstack.org (bootstrap.saltstack.org)... 74.208.29.198
Connecting to bootstrap.saltstack.org
(bootstrap.saltstack.org)|74.208.29.198|:80... connected.
HTTP request sent, awaiting response... 302 Found

numb3rs1x

2013-11-25 04:58:15 UTC

Permalink

I was able to fix this by adding a missing directory. I though at first
that the URL being specified for the bootstrap download was the problem,
but after running through a debug, I found that it was failing silently
because the bootstrap file couldn't be written into the given path, which
is:

/usr/lib/python2.7/site-packages/salt/deploy

To fix it I ran 'sudo mkdir /usr/lib/python2.7/site-packages/salt/deploy'

The bootstrap was placed and salt-cloud deployed the instance properly.

numb3rs1x

2013-11-25 05:00:44 UTC

Permalink

I'll add that my python coding has not advanced to the point that I'm savvy
enough to push this as a patch or know where this exists in the
pre-compiled code.

Pedro Algarvio

2013-11-25 07:50:26 UTC

Permalink

Don

Pedro Algarvio @ Phone

----- Reply message -----
De: "numb3rs1x" <jonslusher-***@public.gmane.org>
Para: <salt-users-/***@public.gmane.org>
Assunto: [salt-users] Re: latest salt-cloud not installing minion during deployment
Data: seg, Nov 25, 2013 05:00
I'll add that my python coding has not advanced to the point that I'm savvy enough to push this as a patch or know where this exists in the pre-compiled code.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

Pedro Algarvio

2013-11-25 07:51:55 UTC

Permalink

Don't worry. I've created a ticket for this issue.

Pedro Algarvio @ Phone

----- Reply message -----
De: "numb3rs1x" <jonslusher-***@public.gmane.org>
Para: <salt-users-/***@public.gmane.org>
Assunto: [salt-users] Re: latest salt-cloud not installing minion during deployment
Data: seg, Nov 25, 2013 05:00
I'll add that my python coding has not advanced to the point that I'm savvy enough to push this as a patch or know where this exists in the pre-compiled code.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.