Hello,
I'm trying to forward logs from azLog (Azure log integration) into my splunk indexer.
Both are running on AWS instances.
Everything seems to be configured correctly except that I don't see anything on the indexer.
Here is the investigation that I did so far:
My indexer has a receiver configured and enabled on 9997.
My instance which has the forwarder installed is able to connect there:
> PS C:\Users\Administrator>> Test-NetConnection xxx.xxx.xxx -Port 9997>> ComputerName : xxx.xxx.xxx> RemoteAddress : xx.xx.xx.xx> RemotePort : 9997 > InterfaceAlias : Ethernet> SourceAddress : xx.xx.xx.xx> TcpTestSucceeded : True
My inputs file looks like this:
[monitor://C:\Users\azlog\AzureActiveDirectoryJson]
disabled = false
crcSalt =
[monitor://C:\Users\azlog\AzureResourceManagerJson]
disabled = false
crcSalt =
[monitor://C:\Users\azlog\AzureSecurityCenterJson]
disabled = false
crcSalt =
My output file looks like this:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
disabled = false
server = xxx.xxx.xxx:9997
[tcpout-server://xxx.xxx.xxx:9997]
spunkd is running. Splunk list monitor shows the correct list of files.
Looking at the log for a specific file that should be forwarded I see :
05-29-2018 08:21:10.878 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - ****************************************
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - File state notification for path='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' (first time).
05-29-2018 08:21:13.878 +0000 DEBUG TailingProcessor - Returning disposition: 1
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Enqueued file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json in tailreader0
05-29-2018 08:21:13.878 +0000 DEBUG TailReader - Start reading file="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" in tailreader0 thread
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile - Reading for plain initCrc...
05-29-2018 08:21:13.878 +0000 DEBUG WatchedFile - Preserving seekptr and initcrc.
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Defering notification for file=C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log by 3.000ms
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Have seen this item before (since splunkd was restarted).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson' in tailreader0 thread, disposition=RECURSE_INTO_THIS_DIRECTORY, deferredBy=0.000
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Returning disposition=RECURSE_INTO_THIS_DIRECTORY for file=C:\Users\azlog\AzureResourceManagerJson
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Start reading file="C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json" in tailreader0 thread
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor - Skipping itemPath='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json', does not match path='C:\Users\azlog\AzureSecurityCenterJson' :Not a directory :Not a symlink
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor - Item 'C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' matches stanza: C:\Users\azlog\AzureResourceManagerJson.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor - Storing config 'C:\Users\azlog\AzureResourceManagerJson'.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor - Will use CRC salt='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' for this source.
05-29-2018 08:21:13.893 +0000 DEBUG TailingProcessor - Entry is associated with 1 configuration(s).
05-29-2018 08:21:13.893 +0000 DEBUG TailReader - Will attempt to read file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json.
05-29-2018 08:21:13.940 +0000 DEBUG TailReader - Got classified_sourcetype='json-6' and classified_charset='AUTO'.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Storing pending metadata for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json, sourcetype=json-6, charset=AUTO
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - setting trailing nulls to true via 'auto'
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Loading state from fishbucket.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|338 ...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Reading for plain initCrc...
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - initcrc has changed to: 0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Computed initCrc=5e4645810867b257 (old style).
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Normal record was not found for initCrc=0x5e4645810867b257.
05-29-2018 08:21:13.940 +0000 DEBUG WatchedFile - Creating new pipeline input channel with channel id: 339.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Attempting to load indexed extractions config from conf=source::C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json|host::EC2AMAZ-HOQE95P|json-6|339 ...
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - About to read data (Opening file: C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json).
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - seeking C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json to off=0
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Reading for plain initCrc...
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - initcrc changed to 0x5e4645810867b257 since file grew past initCrcLen.
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Applying pending meta data
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Clearing pending metadata
05-29-2018 08:21:13.956 +0000 DEBUG WatchedFile - Reached EOF: fname=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json fishstate=key=0x5e4645810867b257 sptr=12112 scrc=0xc11622e038ef0e51 fnamecrc=0xbe9301895b5e826a modtime=1527582073
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Skipping sending done key.
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Will doublecheck EOF (in 3000ms)..
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Finished reading file='C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json' in tailreader0 thread, disposition=NO_DISPOSITION, deferredBy=3.000
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - Defering notification for file=C:\Users\azlog\AzureResourceManagerJson\20180529T082113_3468468.0000000035.af2ac63e-756c-4c64-ad6d-b7dca46a0ceb.json by 3.000ms
05-29-2018 08:21:13.956 +0000 DEBUG TailReader - tailreader0 waiting for jobs
05-29-2018 08:21:14.893 +0000 DEBUG TailingProcessor - ****************************************
But absolutely nothing on the indexer in the main index.
In the internal index I see the log lines : e.g 05-29-2018 08:25:48.948 +0000 DEBUG TailReader - tailreader0 waiting for jobs
Any help with next steps here?
Thanks
↧