Background on layout
"NFS Share" - Central network storage for logs
"Source server" - Solaris zone running a JVM writing logs to the mounted "NFS Share"
"Indexing" - Solaris zone running Splunk Universal Forwarder which is reading logs on the mounted "NFS Share"
The JVM on the source server uses a class to rotate the logs every hour (org.jboss.logmanager.handlers.PeriodicRotatingFileHandler)
http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jboss.logmanager/jboss-logmanager/1.5.1.Final/org/jboss/logmanager/handlers/PeriodicRotatingFileHandler.java#PeriodicRotatingFileHandler.rollOver%28%29
While the class appears to do a move and recreate which should play nicely with Splunk we are losing a substantial amount of logs. As the "Indexing" side does not appear to detect that the logfile was rolled and indexing stops for an hour (sometimes 10 hours)
As you can see the performance logs here are showing large gaps when there is real data. Note all other log types are monitored via the same NFS method but rotated less frequently.
Screenshot is a timechart count of all events grouped by sourcetype
http://imgur.com/a/Kq3ZL
I checked the _internal logs and Splunk is not detecting the rotate performed by the remote JVM. Occasionally it works but most of the time it doesn't.
Ref: http://docs.splunk.com/Documentation/Splunk/6.4.2/Data/Howlogfilerotationishandled
We shouldn't be encountering this issue as the class is doing a move and recreate
https://answers.splunk.com/answers/185453/why-copytruncate-logrotate-does-not-play-well-with.html
Any ideas?
↧