I have an input setup on a universal forwarder where I am monitoring a log file. The monitor on Splunk seems to read the file line-by-line and is truncating log entries way too early. Uploading the file into Splunk works just fine though.
Here is an example of a log entry that I am trying to read:
2019-07-08 22:25:42.314 INFO [MessageHandler.java:91] Processing the following message from Queue
------------------------------------
2019-07-07T23:11:39.000 2019-08-04T23:36:29.000 2019-07-07T22:49:51.000 2019-08-04T23:58:50.000 test.bsp ... ... ...
------------------------------------
However, when I search for this data, I find an entry that looks like this:
2019-07-11 17:00:27.192 INFO [MessageHandler.java:91] Processing the following message from Queue
------------------------------------
Then, when I search for the time within the `` XML tag, I've found this as a standalone event:2019-07-07T23:11:39.000
It seems that the monitor is reading the file line-by-line instead of respecting the line break rules defined in the props.conf, then the date parser takes the incorrect time as the log time. The logger might be flushing the file to disk after each line but that is something that is completely out of my control.
Here is the sourcetype in my props.conf:
[my-sourcetype]
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n\s]*)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\s[\w+\s\[\w\.]+(\:\d+)?\]
MAX_EVENTS = 2000
MAX_TIMESTAMP_LOOKAHEAD = 23
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX = ^
TRUNCATE = 50000
category = Custom
description = This is my source type yo
pulldown_type = true
When I test this out by uploading the file directly, this source type works just fine AND reads the entire log entry as a single one. How can I adjust this so it works for the monitor:// input type as well?
-----------------
Edit: here is the properties from the upload:
![screenshot of file upload props][1]
Edit 2: where does the event breaking and fitting to a sourcetype actually occur? I did change the sourcetype on my universal forwarder instance. Do I need to rename the sourcetype on the UF, or will the indexer/searcher update the sourcetype? Or does it not matter?
[1]: https://i.imgur.com/2AlJ6bJ.png
↧