Here's my setup: I have three clustered indexers, two search heads, a deployment server, as well as several Heavy Forwarders (three Windows and three Linux). I've been collecting Windows logs remotely from the HF via WMI no problems for a while. This week, I decided to install a universal forwarder on two servers as a pilot in preparation for further deployments.
After installing, I found I was getting no log events at all. So I commenced troubleshooting.
First I checked to see if the indexers were receiving data by running tcpdump and I saw the logs and metrics coming over the wire to the indexers. CHECK
Then I checked to see if the records were in ANY index by running the following search:
index = * host=hostnames
This returned nothing. So I searched:
index=* hostnames
And while this returned multiple events, none were FROM those machines.
Then, I checked to see if there were records in the _internal index from those servers. CHECK
Then, I looked to see if any of those _internal records contained errors. No entries that said ERROR, so tentative CHECK
Then I looked on each server where where the UF was installed and looked in splunkd.log for errors. Just one:
AuditTrailManager - Private key error Error opening C:\Program Files\SplunkUniversalForwarder\etc\auth\audit\private.pem: The system cannot find the patch specified.
But I was kind of expecting this as I told the UF to use Splunk own internal certificate during install? Not sure if this is a factor....
So no other errors.
Here's C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_Windows\local\inputs.conf
[WinEventLog://Application]
disabled = 0
index = wineventlog
[WinEventLog://Security]
disabled = 0
index = wineventlog
[WinEventLog://System]
disabled = 0
index = wineventlog
[WinEventLog://Windows Powershell]
disabled = 0
index = wineventlog
Here's C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf
# BASE SETTINGS
[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = ip1:9997, ip2:9997, ip3:9997
## autolbsettings
autoLB = true
autoLBFrequency = 15
forceTimebasedAutoLB = true
Some other posts have mentioned that there could be a permissions issue. Is there a way to verify that? I installed this UF with the same domain admin account that the HF are using to pull logs via WMI so there shouldn't be a permissions issue?
What other steps can I take to fix this?
Thanks.
↧