Splunk Universal Forwarder foot print - What is that hourly task that causes CPU peaks ?

Hello, While terminating the new documentation of the TA-nmon, I carefully ran several steps of detailed load analysis to demonstrate and illustrate what are the costs related to the addon. This analysis implies analysing as well the foot print of the Universal Forwarder, doing that job carefully demonstrated that an hourly scheduled task generates peaks of CPU usage imputable to splunkd. The full story is available in the TA-nmon doc (currently testing): [http://ta-nmon.readthedocs.io/en/testing/footprint.html][1] **CPU usage statistics a Linux testing server:** ![alt text][2] **Some explanations:** - The first series represents the global CPU usage (in percent) with respectively UF + TA-nmon, UF with deployment server connection, UF without deployment server connection - The last serie represents the global CPU usage of the system not running anything but an independent nmon process used to collect performance data We can observe a scheduled hourly task that generates peaks of CPU usage (on this withness server that's CPU peaks up to 1.7% while the CPU usage between 2 tasks is approximately 0.1 to 0.2% of CPU. This peak occurs systematically, occurring every hour after the splunkd start, and it has been observed on various system (AIX, Linux, Solaris) and various processor architectures. (the following server is a Linux Power running on IBM Power8) **Looking at splunkd process:** ![alt text][3] We can as well clearly observe the hourly peak of CPU usage when splunkd is running, even without any application or configuration other than an ouputs.conf to forward internal data. While the level of the peak is relative, it is not negligible and be easily observed when looking carefully at system loads. I would be very curious to understand this behaviour, any one knows about an internal hourly task that could be at the root cause of these (small) CPU peaks ? Thanks, Guilhem [1]: http://ta-nmon.readthedocs.io/en/testing/footprint.html [2]: /storage/temp/194251-cpu-multi.png [3]: /storage/temp/194252-top-splunkd-cpu-multi.png