Hi all,
we forward about 300GB per day from a single forwarder instance to an indexer cluster.
the forwarder is on a strong machine(24 cores, 130GB RAM, ssd) and we already configured limits.conf and server.conf for "unlimited" thruput (800MB/s) and parallelPipelines=2. also we increased the size of the parsing queue and structuredParsingQueue.
Normally we do ok, but sometimes, due to some kind of a problem data is being stacked in the machine and than the forwarder can't seem to close the gap.
the problem is that the universal forwarder lists all of the files first, and only then he forwards them to the indexers. what i want is for it to bo lazy, to list files and forward them at the same time.
Is there any config field i didn't change yet that will help me, it would be great.
Also, if there is anyone from Splunk reading this, you should know that this feature will drastically improve the universal forwarder and will change our life forever.
Thanks in advance,
↧