I'm running a Bro sensor with some (obviously) very high-volume log files that I'm monitoring with the Universal Forwarder. Some of these files are adding events at 500-2000 events/s. The forwarder is forwarding, but at a ridiculously low level, around 5-30 events/s. This is obviously not correct.
I initially set this up via forwarder management in the UI. I have tried both monitorings of all '.log' files in the target directory (/opt/bro/spool/bro), and during troubleshooting, even tried to isolate individual files with their own inputs (e.g., /opt/bro/spool/bro/conn.log). I have also added items directly to the 'inputs.conf' file on the server.
I've researched several options and the only one I've found that could possibly be relevant is the 'crcSalt' option. I've tried that but with no success. Nothing is working to get the forwarder to send these at the actual volume in real-time speed, as I would expect. My inputs.conf currently looks like this:
[monitor:///opt/bro/spool/bro/conn.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt =
[monitor:///opt/bro/spool/bro/dns.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt =
[monitor:///opt/bro/spool/bro/http.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt =
If I can't make this work, I will simply ditch the Universal Forwarder and go back to the rsyslog forwarding method I had previously that worked perfectly. Any thoughts or suggestions would be greatly appreciated.