I'm having trouble getting the Splunk App for Stream working in a test environment with the following configuration:
1. Standalone Splunk Search Head and Indexer
2. Universal Forwarder (target for Splunk Stream)
I've tried a number of different configurations and even reinstalled Splunk from scratch, but still can't get this to work when following the documentation.
I tried using the deployment server, but was getting nowhere, so decided to manually configure the app on the Universal Forwarder, but still no luck.
The firewalls are off on both servers.
I am using the Splunk supplied SSL certs if that is an issue? Is there some flag that needs to be turned on to disable certificate verification? I used sslVerifyServerCert = false to be sure.
**/opt/splunkforwarder/etc/apps/Splunk_TA_stream/local/inputs.conf**
[streamfwd://streamfwd]
splunk_stream_app_location = https://xxx.xxx.xx.xx:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0
sslVerifyServerCert = false
Then tested this with a curl command (from UF) as recommended on a previous Splunk Answers post.
[root@media local]# curl -k https://xxx.xxx.xx.xx:8000/en-us/custom/splunk_app_stream/ping
{"id": "appsmeta", "api_versions": {"ping": 1, "captureipaddresses": 1, "vocabularies": 1, "httpinputs": 1, "streams": 1, "streamforwardergroups": 1, "indexers": 1, "users": 1}, "dateLastUpdated": 1469655789359, "version": "6.6.0", "_key": "appsmeta", "_user": "nobody"}
My NICs are not named weirdly (match en pattern), so should work out of the box. Saw this in another Splunk Answers post
[root@media local]# ifconfig
enp3s0: flags=4163 mtu 1500
inet 192.168.64.64 netmask 255.255.255.0 broadcast 192.168.64.255
inet6 fe80::be5f:f4ff:fe89:3152 prefixlen 64 scopeid 0x20
ether bc:5f:f4:89:xx:xx txqueuelen 1000 (Ethernet)
RX packets 2298867 bytes 1946057053 (1.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1590850 bytes 949878345 (905.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s1: flags=4099 mtu 1500
ether c8:3a:35:d9:xx:xx txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 365635 bytes 93720606 (89.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 365635 bytes 93720606 (89.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099 mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:26::xx:xx txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The forwarder is successfully sending its data to the SH/IDX (well everything apart from Splunk Stream data)
[root@media local]# cat /opt/splunkforwarder/etc/apps/cfgd_fwd_to_idx/local/outputs.conf
[tcpout]
defaultGroup= indexers
indexAndForward= false
[tcpout:indexers]
server= xxx.xxx.xx.xx:9997
autoLB= true
I also have a configuration to ensure that _internal is also passed through to the IDX/SH. I can see data from the UF when I search on index=_internal
[root@media local]# cat /opt/splunkforwarder/etc/apps/cfgd_do_not_log_to_internal/local/outputs.conf
# http://docs.splunk.com/Documentation/Splunk/latest/admin/outputsconf
[tcpout]
# If true, disables index filtering. Events for all indexes are then forwarded. Normally _internal indexes are not forwarded and are stored on the local Splunk instance
forwardedindex.filter.disable = true
# If set to true, data is indexed locally.
indexAndForward = false
I suppose from this point I'm not sure from the manual what I should do as I can't see this UF in the Splunk Stream interface on the Search Head?
I thought I should see it here at least shouldn't it?
http://imgur.com/a/86kRl
Where do I tell the UF that I want to monitor DNS and DHCP?
I've enabled the streams in "Configure Streams"
http://imgur.com/a/gvzn0
Creating a new group in Distributed Forwarder Management doesn't show my UF??
http://imgur.com/a/pXHEe
Any ideas or even a bit more detail in the documentation would be appreciated.
Edit: More information on Splunk Universal Forwarder server.
OS: Fedora 23
Kernel: 4.6.4-201.fc23.x86_64
Splunk running as root on all instances.
I am currently installing a standalone Splunk instance with Splunk App for Stream on this server to see if that will work purely for testing purposes.
↧