Quantcast
Viewing all articles
Browse latest Browse all 1551

How to troubleshoot why an indexer is only receiving data from 50% of forwarders in my environment?

I spent hours trying to figure this out Friday, and it's been bugging me all weekend. So, I'm hoping the community can help me figure this out! The info below is all from memory, hopefully I don't miss anything. First off, I'm completely new to Splunk... So if I butcher terminology or concepts, please understand! I am now trying to come in and fix something that appears to have never worked. Several months ago, the Splunk universal forwarder was pushed out to all of my Windows machines. I am fairly certain that it was pushed out using our patching solution "BigFix". Fast forward to today. I am receiving data from about 150 hosts. Unfortunately, I should be receiving data from closer to 350. My domain controllers are included in the list of the systems that are not forwarding data. The guy before me decided to set up a heavy forwarder, something about blowing through our license. I haven't looked into the heavy forwarder too much, but I'm assuming that it's working since half of the hosts are getting through to the indexer. 1 - So far I've compared the local/inputs.conf and the local/server.conf on the working system and the not-working system. According to the guy who did the install, those are the only files that he touched after the install. On each of the systems both the local/inputs.conf and the local/server.conf files are basically identical. 2 - Also, on the not-working system and the heavy forwarder I've run `NETSTAT -an` to verify that the 2 systems are establishing a connection between each other. 3 - I've dug through the `var/logs/splunkd.log` on both the working and the non-working system, and I didn't see anything obvious that would indicate what is wrong on the non-working system. 4 - I've spent hours making changes to the inputs.conf and the server.conf, then restarting the Splunk forwarder service, to no avail. Where else can I look...What else can I do... to try and figure out why only half of my systems are able to forward events to the indexer, and the other half cannot? Any and all help would be greatly appreciated. Thanks!

Viewing all articles
Browse latest Browse all 1551

Trending Articles