Have been experimenting with Splunk Light for about a week so. I have 8 different devices running a java application that I need to monitor logs (log4j) from. The devices have Universal Forwarder installed and I have configured the forwarding from the Splunk Cloud web interface.
Today I discovered that some events have been duplicated multiple times on the index. I discovered this because that one the devices in a report suddenly had an unusual high event count. One event that occurred at around 19:30 yesterday has for some reason been reindexed about every 5 minutes between 19:30 and 03:30 tonight when it suddenly stopped. It has been duplicated 58 times! I have confirmed that there is only one entry in the actual log file.
The log4j uses RollingFileAppender and thus when exceeding the max size stores the old logs in app.log.n+1. I expected that the log rotation was causing the issue, but from what I have read Splunk should by default handle this scenario. But I see that others include the file in the path for the data input. When I created the data input I just added the path to the folder (...\logs) without specifying the actual file names. I just figured there won't be any other files in that folder so why not monitor the whole folder? Could this be the issue?
Even if that is the issue, I don't understand why it would get duplicated so many times? The log has only rotated twice since deployment and so contains app.log, app.log.1 and app.log.2.
Any suggestions?
↧