We have a universal forwarder set up to forward incoming messages to logstash, TCP -> forwarder -> TCP:
outputs.conf:
[tcpout]
defaultGroup = logstash
[tcpout:logstash]
server=localhost:7777
sendCookedData = false
useACK = true
And inputs.conf:
[tcp://:9997]
sourcetype=_json
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true
[monitor://$SPLUNK_HOME/var/log/splunk/license_audit.log]
disabled = true
If we connect our load tester directly to logstash, then our messages appear correctly formatted. If we go via the universal formatter then messages are coming through with extra line breaks. It's as if Splunk isn't able to tell where one event stops and another begins.
Interestingly, if we send the same event multiple times, it seems to get truncated at the same point each time. All our messages originating from the load testing tool are json formatted, newline seperated. We have also tried terminating messages with EOT characters but to no avail.
What could be causing this? Is there a key piece of information/documentation we are missing?
↧