I have a customer that wants to index psv files with headers. If I omit the props.conf file on the Universal Forwarder (UF), the entire psv file gets indexed as one event without any parsing. I have a props.conf on the indexer, but it's my understanding that the indexer does not parse forwarded structured data. However, when I add the props.conf to the UF's, no data is indexed. I have tried with UF versions 6.1.2, and 6.4 running on Linux and Sun. My inputs.conf and props.conf on the UF's are as follows:
inputs.conf
[monitor:///tmp/testmetrics*.txt]
crcSalt =
sourcetype = test_pri
index = test
disabled = 0
props.conf
[test_pri]
FIELD_DELIMITER=|
HEADER_FIELD_DELIMITER=|
HEADER_FIELD_LINE_NUMBER=1
INDEXED_EXTRACTIONS=psv
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIMESTAMP_FIELDS=DATETIME
TIME_FORMAT=%Y%m%d/%H%M%S
KV_MODE=none
The data is in this format with CRLF terminations after each line:
col1|col2|col3
row1|row11|row1111
row2|row22|row222
row3|row33|row333
splunkd.log:
12-02-2016 15:02:13.567 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/tmp/testmetrics.txt'.
12-02-2016 15:03:02.914 DEBUG TailingProcessor - File state notification for path='/tmp/testmetrics.txt' (first time).
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/etc/splunk.version' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/log/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/log/splunk/splunkd.log' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/spool/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Skipping itemPath='/tmp/testmetrics.txt', does not match path='/proj/unix/cen/tools/splunkforwarder/var/spool/splunk' :Not a directory :Not a symlink
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Item '/tmp/testmetrics.txt' matches stanza: /tmp/testmetrics*.txt.
12-02-2016 15:03:03.059 DEBUG TailingProcessor - Will use CRC salt='/tmp/testmetrics.txt' for this source.
12-02-2016 15:03:03.059 DEBUG FilesystemFilter - Testing path=/tmp/testmetrics.txt(real=/tmp/testmetrics.txt) with global blacklisted paths
12-02-2016 15:03:03.059 DEBUG TailReader - Will attempt to read file: /tmp/testmetrics.txt.
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG FileClassifierManager - Finding type for file: /tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|test_pri
12-02-2016 15:03:03.059 DEBUG WatchedFile - Storing pending metadata for file=/tmp/testmetrics.txt, sourcetype=test_pri, charset=UTF-8
12-02-2016 15:03:03.059 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|45
12-02-2016 15:03:03.060 DEBUG WatchedFile - Attempting to load indexed extractions config from conf=source::/tmp/testmetrics.txt|host::testhost|test_pri|45 ...
12-02-2016 15:03:03.060 DEBUG VerboseCrc - Checksumming salt_data="/tmp/testmetrics.txt".
12-02-2016 15:03:03.060 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|46
12-02-2016 15:03:03.060 DEBUG WatchedFile - Attempting to load indexed extractions config from conf=source::/tmp/testmetrics.txt|host::testhost|test_pri|46 ...
12-02-2016 15:03:03.060 DEBUG TailReader - About to read data (Opening file: /tmp/testmetrics.txt).
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=0
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=0
12-02-2016 15:03:03.060 DEBUG PropertiesMapConfig - Performing pattern matching for: source::/tmp/testmetrics.txt|host::testhost|test_pri|46
12-02-2016 15:03:03.060 DEBUG WatchedFile - seeking /tmp/testmetrics.txt to off=14598
12-02-2016 15:03:03.060 DEBUG WatchedFile - Reached EOF: fname=/tmp/testmetrics.txt fishstate=key=0x915b2ffd0a19e405 sptr=14598 scrc=0xf4eb0f294d1af3b2 fnamecrc=0x5fae16cea4aef038 modtime=1480708933
12-02-2016 15:03:03.060 DEBUG FilesystemChangeWatcher - inotify doing infrequent backup polling for healthy path="/tmp/testmetrics.txt"
Thanks.
↧
