[systemd-devel] directive for executing a script on service failure

[George Karakougioumtzis]

Since i couldn't find any info on the python's systemd-dbus api to get
the status of the services that i wanted i looked into python-systemd
and made a class (which is far from being ideal i must admit as i am a
python newbie) that monitors the journal for failed services. Class is
based upon info from python-systemd site and a post on stack overflow.
Here it is:
-----------------------------------
#!/usr/bin/python
import systemd
import select
import json
import time
import threading
from systemd import journal
from gi.repository import Notify, GObject


class LogReader():

 def __init__(self):
  pass
 def start_reading(self):
 
   j = journal.Reader()
   #j.this_boot(bootid=None)
   j.log_level(journal.LOG_INFO)

# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
   j.seek_tail()
  
   p = select.poll()
   p.register(j, j.get_events())
  
   while p.poll():
    waiting=j.process()
    if waiting == 1 or waiting == 2:
     j.get_next()     
     for entry in j:
      #print entry         
      if 'MESSAGE' in entry:
                 
       str2= "entered failed state"       
       try:
        string=entry['MESSAGE'].decode('utf-8')
        if string and str2 in string:
           Notify.init("myapp_name")
           n = Notify.Notification.new("journald:", string)
               n.show()       
           except Exception as e:
            raise


 def start_threads(self):
  for i in range(1,10):
   # Instatiates the thread
   # (i) does not make a sequence, so (i,)
   t = threading.Thread(target=self.start_reading(), args=(i,))
   # Sticks the thread in a list so that it remains accessible
   thread_list.append(t)

# Starts threads
   for thread in thread_list:
    thread.start()     
    thread_list = []
# This blocks the calling thread until the thread whose join() method is
called is terminated.
# From http://docs.python.org/2/library/threading.html#thread-objects
   for thread in thread_list:
    thread.join()

lr=LogReader()
lr.start_threads()


-------------------------------------------------------









On 02/10/2015 10:53 PM, Lennart Poettering wrote:
> On Fri, 06.02.15 21:23, Mantas Mikulėnas (grawity at gmail.com) wrote:
>
>> On Fri, Feb 6, 2015 at 5:26 PM, George Karakougioumtzis <
>> mad-proffessor at hotmail.com> wrote:
>>
>>> Hi. Congrats for the near perfect job on systemd! I was searching for a
>>> directive to execute a script upon systemd service failure. I would like
>>> to receive desktop notifications about such  failures. I stumbled upon
>>> OnFailure and FailureAction but these have hardcoded list of actions?
>> One of those actions is "start an arbitrary unit", which could handle
>> notifications... Unfortunately it doesn't actually pass any failure
>> information to that unit. 
>>
>> So, instead, you might have to watch the system
>> log (journal) for unit failure notices. (I would normally have said that
>> the journal makes it easy to filter for them, but quite strangely, these
>> ones don't have a MESSAGE_ID attached...)
> if you use OnFailure= I'd recommend checking the properties of the
> unit that failed instead of checking the journal. i.e. the equivalent
> of "systemctl show -p ..." on the unit that failed...
>
> And yeah, we should define a MESSAGE_ID for unit failure log messages.
>
>>> Any hints how to get notified (with notify-send most likely)?
>> systemd calling notify-send isn't going to happen – as a service, it runs
>> "outside" your desktop session, and has no access to its D-Bus session bus,
>> which tends to be started at a random address. (There might even be several
>> sessions, not just one!) And even considering the future "user bus" plans
>> (which would put the session bus at a known fixed location), the whole idea
>> of having a service inject something into a desktop session is really ugly.
>> Even Microsoft realized that and implemented "session 0 isolation"
>> in VIsta.
> Yes, that's correct. We do not call from lower layers into higher
> layers in systemd. Higher layers should instead subscribe to lower
> layers...
>
>> In other words, such a notifier would need to be started from within your
>> desktop session, connect *to* systemd (either by waiting for signals on the
>> system bus or by watching the journal messages), and idle in background
>> waiting for some unit to fail. With DBus it could be a simple 10-line
>> script, waiting for signals on one bus and calling Notify() on another...
> Another option would be to write a clinet that watches the journal
> constantly, and pops up a notification each time a message with
> LOG_ERR and higer is sent. After all, not only service errors should
> be interesting, all other errors should matter too..
>
> Lennart
>