returner_sentry problem

[schlag]

Hi,

I am trying to hook up a sentry returner and have the following pillar
settings for a number of my minions:

raven:
servers:
- https://obfuscated-LGO0Z2YbA2cez710FhfsEgC/***@public.gmane.org/obfuscated
project: Saltstack
tags:
- host

I've created a team and project for this at getsentry called "Saltstack".

So far no events have come through. I wasn't quite following what the public_key
or secret_key were for in the documentation (http://docs.saltstack.com/ref/returners/all/salt.returners.sentry_return.html)
because I couldn't find anything related at getsentry.com.

I've tried testing this returner by using this: salt \* test.ping --return
sentry_return I see the output, but nothing arrives at getsentry.com. Any
ideas? Am curious how others tie into it.

Thanks in advance

[schlag]

meant to add that i've installed raven via pip (raven==4.0.3)

debian wheezy

salt --versions-report
Salt: 0.17.4
Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3

[Tim Messier]

The public_key and secret_key come from your sentry project. On
getsentry.com go to your Saltstack project settings page and click on API
Keys. It provides a url in the format of:

https://<public_key>:<secret_key>@app.sentry.com/<project>

You can set the servers field to just point to https://app.getsentry.com.
The keys are used by raven to authenticate and send messages to your
specific sentry project.

My pillar is something like this:

raven:
servers:
- https://app.getsentry.com
public_key: <key from API Keys page>
secret_key: <key from API Keys page>
project: <project id number>
tags:
- host
- os
- cpuarch

Post by schlag
Hi,
I am trying to hook up a sentry returner and have the following pillar
project: Saltstack
- host
I've created a team and project for this at getsentry called "Saltstack".
So far no events have come through. I wasn't quite following what the public_key
or secret_key were for in the documentation (
http://docs.saltstack.com/ref/returners/all/salt.returners.sentry_return.html)
because I couldn't find anything related at getsentry.com.
I've tried testing this returner by using this: salt \* test.ping --return
sentry_return I see the output, but nothing arrives at getsentry.com. Any
ideas? Am curious how others tie into it.
Thanks in advance
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[schlag]

thanks for that info!

so here is what i have in pillar now (keys & project id are obfuscated):

raven:
servers:
- https://app.getsentry.com
public_key: pubkey
secret_key: secretkey
project: project-id
tags:
- host
- os
- cpuarch

i also came across the sentry logging handler
here http://docs.saltstack.com/ref/configuration/logging/handlers/salt.log.handlers.sentry_mod.html
and set the following up in /etc/salt/master.d/sentry.conf:

sentry_handler:
dsn: https://pubkey:secretkey-LGO0Z2YbA2cez710FhfsEgC/***@public.gmane.org/project-id
project: project-id
public_key: pubkey
secret_key: secretkey
log_level: info

i've double & triple checked the keys and project-id. when i restart
salt-master after adding the sentry config, i get this error:

HTTPError: HTTP Error 400: BAD REQUEST
[ERROR ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST
(url: https://app.getsentry.com/api/project-id/store/, body: Missing
required attribute in authentication header: sentry_secret)
Traceback (most recent call last):
File
"/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line
104, in send_sync
super(ThreadedHTTPTransport, self).send(data, headers)
File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py",
line 41, in send
response = urlopen(req, data, self.timeout).read()
File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 435, in error
return self._call_chain(*args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)


In getsentry, I chose Django/python as the "platform" and tried "Other" as
well to see if that helped, but to no avail.

I tried simplifying the sentry_handler config in
/etc/salt/master.d/sentry.conf, by making it look like this:

sentry_handler:
dsn: https://pubkey:secretkey-LGO0Z2YbA2cez710FhfsEgC/***@public.gmane.org/project-id

and now get this warning after restarting salt-master:


Starting salt master control daemon:
salt-master/usr/local/lib/python2.6/dist-packages/raven/base.py:149:
DeprecationWarning: Manually configured connections are deprecated. Switch
to a DSN.
warnings.warn('Manually configured connections are deprecated. Switch to
a DSN.', DeprecationWarning)


So it seems to not be picked up, or maybe the error is just worded badly.
I am using a DSN in the handler config. Either way, I'm still not
receiving any events at getsentry.com.

[Tim Messier]

Hmm..first did the change for the minion's pillar data work? Does salt \*
test.ping --return sentry work now? (Also note, --return sentry not --return
sentry_return)

As for the DeprecationWarning, that is coming from raven (and is just a
warning, I am using raven==3.5.2 and don't get that warning, probably
because it isn't marked as deprecated in that version).

When you switched to just specifying just the dsn, did you also stop
specifying the log_level? Because by default the log level is error, so you
may not have had any errors for sentry to report.

On Tue, Jan 28, 2014 at 5:58 PM, schlag <teknophil1ac-***@public.gmane.org> wrote:

thanks for that info!

Post by Tim Messier
- https://app.getsentry.com
public_key: pubkey
secret_key: secretkey
project: project-id
- host
- os
- cpuarch
i also came across the sentry logging handler here
project: project-id
public_key: pubkey
secret_key: secretkey
log_level: info
i've double & triple checked the keys and project-id. when i restart
HTTPError: HTTP Error 400: BAD REQUEST
[ERROR ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST
(url: https://app.getsentry.com/api/project-id/store/, body: Missing
required attribute in authentication header: sentry_secret)
File
"/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line
104, in send_sync
super(ThreadedHTTPTransport, self).send(data, headers)
File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py",
line 41, in send
response = urlopen(req, data, self.timeout).read()
File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 435, in error
return self._call_chain(*args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
In getsentry, I chose Django/python as the "platform" and tried "Other" as
well to see if that helped, but to no avail.
I tried simplifying the sentry_handler config in
DeprecationWarning: Manually configured connections are deprecated. Switch
to a DSN.
warnings.warn('Manually configured connections are deprecated. Switch to
a DSN.', DeprecationWarning)
So it seems to not be picked up, or maybe the error is just worded badly.
I am using a DSN in the handler config. Either way, I'm still not
receiving any events at getsentry.com.
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[schlag]

yeah, so i've tried a few things. yeah, the first raven settings in pillar
didnt work. and yes, i tried removing log_level when i switched to dsn.
thanks for the info on the deprecated warning and '--return sentry'. i
must have seen '--return sentry_return' somewhere online.
just to recap, my pillar settings are as follows (my project, & keys are
obfuscated):

raven:
servers:
- https://app.getsentry.com
project: 12345
public_key: deadbeefdeadbeefdeadbeefdeadbeef
secret_key: beefdeadbeefdeadbeefdeadbeefdead
tags:
- os
- master
- saltversion
- cpuarch

verified that the minions see it with 'salt-call pillar.get raven'

eg:
salt-call pillar.get raven
local:
----------
project:
12345
public_key:
deadbeefdeadbeefdeadbeefdeadbeef
secret_key:
beefdeadbeefdeadbeefdeadbeefdead
servers:
- https://app.getsentry.com
tags:
- os
- master
- saltversion
- cpuarch


tested it with a 'salt \* test.ping --return sentry' on the master, still
nothing in getsentry. here is some debug output:

--snip

salt \* test.ping --return sentry -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/halite.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/halite.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/schedule.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/sentry.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/sentry.conf
[DEBUG ] Missing configuration file: /root/.salt
[DEBUG ] loading log_handlers in
['/var/cache/salt/master/extmods/log_handlers',
'/usr/lib/python2.6/dist-packages/salt/log/handlers']
[DEBUG ] Skipping /var/cache/salt/master/extmods/log_handlers, it is not
a directory
[DEBUG ] Loaded sentry_mod as virtual sentry
[INFO ] Processing `log_handlers.sentry`
/usr/local/lib/python2.6/dist-packages/raven/base.py:149:
DeprecationWarning: Manually configured connections are deprecated. Switch
to a DSN.
warnings.warn('Manually configured connections are deprecated. Switch to
a DSN.', DeprecationWarning)
[DEBUG ] Adding the 'sentry' provided logging handler:
<raven.handlers.logging.SentryHandler object at 0x26e0450>
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/halite.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/halite.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/schedule.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/sentry.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/sentry.conf
[DEBUG ] Missing configuration file: /root/.salt
[DEBUG ] LocalClientEvent PUB socket URI:
ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG ] LocalClientEvent PULL socket URI:
ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG ] loading output in ['/var/cache/salt/master/extmods/output',
'/usr/lib/python2.6/dist-packages/salt/output']
[DEBUG ] Skipping /var/cache/salt/master/extmods/output, it is not a
directory
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
minion1:
True
[DEBUG ] loading output in ['/var/cache/salt/master/extmods/output',
'/usr/lib/python2.6/dist-packages/salt/output']
[DEBUG ] Skipping /var/cache/salt/master/extmods/output, it is not a
directory
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
--snip

i do at least see this in the salt-master log after setting that output to
debug:

--snip
2014-01-30 14:12:03,444 [salt.master
][DEBUG ] Published command details {'tgt_type': 'glob', 'jid':
'20140130141203442417', 'tgt': 'ln*', 'ret': 'sentry', 'user': 'sudo_user1,
'arg': [], 'fun': 'test.ping'}
--snip

but I don't see any traffic going towards app.getsentry.com when i run the
test.ping.

we have a pretty hefty enterprise plan and are under our daily usage. i'll
contact them still though to see if they're seeing anything on their end.

one thing of note is that if i enable sentry_handler with a dsn and i
specify log_level: info (so that i can actually see stuff), i start seeing
these errors:

--snip
[ERROR ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST
(url: https://app.getsentry.com/api/12345/store/, body: Missing required
attribute in authentication header: sentry_secret)
Traceback (most recent call last):
File
"/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line
104, in send_sync
super(ThreadedHTTPTransport, self).send(data, headers)
File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py",
line 41, in send
response = urlopen(req, data, self.timeout).read()
File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 435, in error
return self._call_chain(*args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: BAD REQUEST
Sentry is attempting to send 3 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
--snip

i upgraded my master and all minions to 0.17.5

salt --versions-report
Salt: 0.17.5
Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3



my whole point of this exercise is to try to do the following:

1) get salt warnings & errors into getsentry (things that are normally
just logged to syslog locally) <-- i presume sentry_handler manages this

2) get salt events (state executions & their results, module executions,
etc..) into getsentry <-- i was thinking that a returner would help here,
but even if i get this part working, i'm not sure what is required to get
that data pushed into sentry w/o having to use '--return sentry' every
time. is my approach even right here?


thanks again!

[Tim Messier]

I did some more testing, I installed some newer version of raven on some
hosts (since that seems to be the only difference between your setup and
mine), and it seems that salt is failing with raven>3.5. I would file a bug
about this and while you want for that to be fixed, you could try
installing raven==3.5.2 (that is the version that is working for me).

As for 2: if you are running commands via the cli, I think you will always
need to specifiy --return sentry, I don't see an option to set a default
returner (feature request?). You do seem to be able to specify a returner
using the salt scheduler<http://docs.saltstack.com/topics/jobs/schedule.html>if
that is of any help.

Post by schlag
yeah, so i've tried a few things. yeah, the first raven settings in
pillar didnt work. and yes, i tried removing log_level when i switched to
dsn. thanks for the info on the deprecated warning and '--return sentry'.
i must have seen '--return sentry_return' somewhere online.
just to recap, my pillar settings are as follows (my project, & keys are
- https://app.getsentry.com
project: 12345
public_key: deadbeefdeadbeefdeadbeefdeadbeef
secret_key: beefdeadbeefdeadbeefdeadbeefdead
- os
- master
- saltversion
- cpuarch
verified that the minions see it with 'salt-call pillar.get raven'
salt-call pillar.get raven
----------
12345
deadbeefdeadbeefdeadbeefdeadbeef
beefdeadbeefdeadbeefdeadbeefdead
- https://app.getsentry.com
- os
- master
- saltversion
- cpuarch
tested it with a 'salt \* test.ping --return sentry' on the master, still
--snip
salt \* test.ping --return sentry -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/halite.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/halite.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/schedule.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/sentry.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/sentry.conf
[DEBUG ] Missing configuration file: /root/.salt
[DEBUG ] loading log_handlers in
['/var/cache/salt/master/extmods/log_handlers',
'/usr/lib/python2.6/dist-packages/salt/log/handlers']
[DEBUG ] Skipping /var/cache/salt/master/extmods/log_handlers, it is not
a directory
[DEBUG ] Loaded sentry_mod as virtual sentry
[INFO ] Processing `log_handlers.sentry`
DeprecationWarning: Manually configured connections are deprecated. Switch
to a DSN.
warnings.warn('Manually configured connections are deprecated. Switch to
a DSN.', DeprecationWarning)
<raven.handlers.logging.SentryHandler object at 0x26e0450>
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/halite.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/halite.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/schedule.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/sentry.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/sentry.conf
[DEBUG ] Missing configuration file: /root/.salt
ipc:///var/run/salt/master/master_event_pub.ipc
ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG ] loading output in ['/var/cache/salt/master/extmods/output',
'/usr/lib/python2.6/dist-packages/salt/output']
[DEBUG ] Skipping /var/cache/salt/master/extmods/output, it is not a
directory
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
True
[DEBUG ] loading output in ['/var/cache/salt/master/extmods/output',
'/usr/lib/python2.6/dist-packages/salt/output']
[DEBUG ] Skipping /var/cache/salt/master/extmods/output, it is not a
directory
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
--snip
i do at least see this in the salt-master log after setting that output to
--snip
2014-01-30 14:12:03,444 [salt.master
'20140130141203442417', 'tgt': 'ln*', 'ret': 'sentry', 'user': 'sudo_user1,
'arg': [], 'fun': 'test.ping'}
--snip
but I don't see any traffic going towards app.getsentry.com when i run
the test.ping.
we have a pretty hefty enterprise plan and are under our daily usage.
i'll contact them still though to see if they're seeing anything on their
end.
one thing of note is that if i enable sentry_handler with a dsn and i
specify log_level: info (so that i can actually see stuff), i start seeing
--snip
[ERROR ] Unable to reach Sentry log server: HTTP Error 400: BAD REQUEST
(url: https://app.getsentry.com/api/12345/store/, body: Missing required
attribute in authentication header: sentry_secret)
File
"/usr/local/lib/python2.6/dist-packages/raven/transport/threaded.py", line
104, in send_sync
super(ThreadedHTTPTransport, self).send(data, headers)
File "/usr/local/lib/python2.6/dist-packages/raven/transport/http.py",
line 41, in send
response = urlopen(req, data, self.timeout).read()
File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 435, in error
return self._call_chain(*args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: BAD REQUEST
Sentry is attempting to send 3 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
--snip
i upgraded my master and all minions to 0.17.5
salt --versions-report
Salt: 0.17.5
Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3
1) get salt warnings & errors into getsentry (things that are normally
just logged to syslog locally) <-- i presume sentry_handler manages this
2) get salt events (state executions & their results, module executions,
etc..) into getsentry <-- i was thinking that a returner would help here,
but even if i get this part working, i'm not sure what is required to get
that data pushed into sentry w/o having to use '--return sentry' every
time. is my approach even right here?
thanks again!
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[schlag]

pip installing raven==3.5.2 did the trick! thank you. i'll look into the
scheduler for specifying returners. very much appreciated!