[systemd-devel] Systemd Journald and audit logging causing journal issues

[Brad Zynda]

On 12/01/2017 12:31 PM, Lennart Poettering wrote:
> On Fr, 01.12.17 12:19, Brad Zynda (bradley.v.zynda at nasa.gov) wrote:
> 
>>
>>
>> On 12/01/2017 12:11 PM, Lennart Poettering wrote:
>>> On Fr, 01.12.17 09:05, Brad Zynda (bradley.v.zynda at nasa.gov) wrote:
>>>
>>>> Hey Lennart,
>>>
>>> Heya,
>>>
>>>> Just wanted to get your thoughts on this before we break the link..
>>>
>>> On what precisely? I am not sure what the original issue is, can you
>>> summarize this briefly here?
>>
>> Absolutely, here is the previous email chain:
> 
> Ah, completely forgot about that thread...
> 
> I am still wondering: why does auditd log to the journal (or syslog or
> stdout/stderr) at all? It maintains its own log record database to my
> knowledge. Moreover, journald is pulls audit data directly from the
> kernel anyway, hence there's no need to also push data in from auditd
> anyway....
> 
> Lennart
> 
Well here is what Steve has said on it:



On Friday, December 1, 2017 8:17:58 AM EST Brad Zynda wrote:
> Hey Steve,
>
> Just wanted to follow up on this and say we are still seeing services
> across the board that have:
>
> Warning: Journal has been rotated since unit was started. Log output is
> incomplete or unavailable
>
> basically created a script to check all unit file services/targets and
> grep status -l for Journal, it is effecting a large range of
> service.target's, service.service's and service.socket's
>
> If we restart the service or reboot we no longer see those messages
> about journal and everything appears to run as expected.

I have never looked at journald code and have no idea how it works or
why it
cares about audit events. My advice last email was to break the link if its
causing problems.

-Steve


> On 10/19/2017 04:13 PM, Steve Grubb wrote:
>> On Thursday, October 19, 2017 1:08:22 PM EDT Brad Zynda wrote:
>>>>> grep perm_mod /etc/audit/audit.rules
>>>>> -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F
auid>=1000
>>>>> -F auid!=4294967295 -k perm_mod
>>>>> -a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F
auid>=1000
>>>>> -F auid!=4294967295 -k perm_mod
>>>>> -a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F
>>>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>>>> -a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F
>>>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>>>> -a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S
>>>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>>>> auid!=4294967295 -k perm_mod
>>>>> -a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S
>>>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>>>> auid!=4294967295 -k perm_mod
>>>>>
>>>>> grep delete /etc/audit/audit.rules
>>>>> -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat
>>>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>>>> -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat
>>>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>>>> -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
>>>>> -a always,exit -F arch=b32 -S init_module -S delete_module -k modules
>>>>
>>>> These rules are well formed. Meaning no obvious holes that would cause
>>>> unintended events. The other ausearch/aureport commands I gave you
>>>> should
>>>> show you what is causing the events and to which files. This
information
>>>> may be used to create some kind of "never" rule that limits what gets
>>>> audited. If you do create some exclusion rule, it needs to be above the
>>>> perm_mod rules because audit is a "first match wins" system.
>>>>
>>>> -Steve
>>>>
>>>> -Steve
>>>
>>> Here is a peak report (at least in the last 24 hours) didnt include the
>>
>>> file summaries as that would make this email HUGE:
>> Well, the idea is to look for something that's getting hit a lot. What it
>> sounds like is that things are getting deleted and modified quite a bit
>> all
>> over the place. Does the executable report show a pattern such as one
>> application doing a lot? For example, with the rule you have, doing a yum
>> update will delete a whole lot of stuff and modify a whole lot of stuff.
>>
>> Its possible to put exceptions in the rules so that one program does not
>> flood the logs. But looking at the quantities below, the audit system can
>> easily handle that.
>>
>> Its also possible to exclude directories from auditing if the pattern is
>> that you have a daemon receiving and modifying files and then deleting
>> them. You should look at the executables & files and see if you can do
>> with auditing what they are doing because its not interesting.
>>
>> If this is causing you problems on the journald side where its causing
>> other tasks to fail. Then I'd break the link between auditd and journald.
>> The amount of event you show is highish but well within range of what
>> auditd can do. Just make sure flush is set to incremental_async and freq
>> is 100 or 200. You should be OK.
>>
>> -Steve
>>
>>> Key Summary Report
>>> ===========================
>>> total  key
>>> ===========================
>>> 8170  perm_mod
>>> 5390  delete
>>> 1073  access
>>> 56  time-change
>>> 26  session
>>> 12  privileged
>>> 7  logins
>>>
>>>
>>> Syscall Summary Report
>>> ==========================
>>> total  syscall
>>> ==========================
>>> 4250  fchmodat
>>> 1613  chmod
>>> 831  fchmod
>>> 521  fchown
>>> 97  chown
>>> 12  setxattr
>>>
>>>
>>> Syscall Summary Report
>>> ==========================
>>> total  syscall
>>> ==========================
>>> 2887  unlink
>>> 2189  rename
>>> 186  unlinkat
>>>
>>>
>>> so from the list the top 2 in each perm_mod and delete from the above
>>> list seem to be the culprits..



So it sounds as if know one knows how audit works?

Thanks,
Brad