Hello All.
I am having existing setup for Splunk for the Aix servers and we just added some new servers to upgrade our application.
On our existing AIX Servers, the Splunk universal forwarder are installed and connected to Splunk app through config files and polling the logs files regularly.
The issue I am encountering right now is when I am trying to setup new servers with Splunk Universal Forwarder, the new hosts and source types are not showing up in my Splunk Search.
I have created a new index and configured that index as well on Splunk app.
I can see the new indexer is getting all the events when I check it from Splunk >Manager>Indexes
I have checked all the configuration file to ensure there is not typo and Splunk can read the log files on Splunk Universal Forwarder
Below what I see on my splunkd.log file as output when I start splunk on Aix server.
11-23-2016 12:08:45.685 -0700 INFO LicenseMgr - Initing LicenseMgr
11-23-2016 12:08:45.698 -0700 INFO ServerConfig - My GUID is C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:45.698 -0700 INFO ServerConfig - My server name is "iv54318p".
11-23-2016 12:08:45.698 -0700 INFO ServerConfig - My hostname is "iv54318p".
11-23-2016 12:08:45.702 -0700 INFO ServerConfig - Setting HTTP server compression state=on
11-23-2016 12:08:45.702 -0700 INFO ServerConfig - Setting HTTP client compression state=0 (false)
11-23-2016 12:08:45.702 -0700 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
11-23-2016 12:08:45.702 -0700 INFO LMConfig - serverName=iv54318p guid=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:45.703 -0700 INFO LMConfig - connection_timeout=30
11-23-2016 12:08:45.703 -0700 INFO LMConfig - send_timeout=30
11-23-2016 12:08:45.703 -0700 INFO LMConfig - receive_timeout=30
11-23-2016 12:08:45.703 -0700 INFO LMConfig - squash_threshold=1000
11-23-2016 12:08:45.703 -0700 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=false
11-23-2016 12:08:45.703 -0700 INFO LMStackMgr - closing stack mgr
11-23-2016 12:08:45.703 -0700 INFO LMSlaveInfo - all slaves cleared
11-23-2016 12:08:45.703 -0700 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
11-23-2016 12:08:45.703 -0700 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
11-23-2016 12:08:45.703 -0700 INFO LMStackMgr - init completed [C46EAB6B-7D93-4072-BDEA-D5D4DDED627A,Forwarder,runContext_splunkd=false]
11-23-2016 12:08:45.703 -0700 INFO LicenseMgr - StackMgr init complete...
11-23-2016 12:08:45.703 -0700 INFO LMTracker - this is not splunkd, will perform partial init
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=DistSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=RcvSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=ScheduledSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=Alerting state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=DeployServer state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LMTracker - setting feature=LocalSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO LicenseMgr - Tracker init complete...
11-23-2016 12:08:46.046 -0700 INFO ServerConfig - My GUID is C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.046 -0700 INFO ServerConfig - My server name is "iv54318p".
11-23-2016 12:08:46.046 -0700 INFO ServerConfig - My hostname is "iv54318p".
11-23-2016 12:08:46.051 -0700 INFO ServerConfig - Setting HTTP server compression state=on
11-23-2016 12:08:46.051 -0700 INFO ServerConfig - Setting HTTP client compression state=0 (false)
11-23-2016 12:08:46.051 -0700 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: virtual address space size: unlimited
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: data segment size: 134217728 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 WARN ulimit - Splunk may not work due to small data segment limit!
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: resident memory size: 33554432 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 WARN ulimit - Splunk may not work due to small resident memory size limit!
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: stack size: 33554432 bytes [hard maximum: 4294967296 bytes]
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: data file size: 1073741312 bytes
11-23-2016 12:08:46.058 -0700 WARN ulimit - Splunk may not work due to low file size limit
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: open files: unlimited
11-23-2016 12:08:46.058 -0700 INFO ulimit - Limit: cpu time: unlimited
11-23-2016 12:08:46.061 -0700 INFO loader - Splunkd starting (build 143156).
11-23-2016 12:08:46.061 -0700 INFO loader - System info: AIX, iv54318p, 1, 7, 00C948174C00.
11-23-2016 12:08:46.061 -0700 INFO loader - Detected 24 (virtual) CPUs and 49152MB RAM
11-23-2016 12:08:46.061 -0700 INFO loader - Arguments are: "splunkd" "-p" "8089" "start"
11-23-2016 12:08:46.061 -0700 INFO loader - Getting configuration data from: /u01/splunk/splunkforwarder/etc/myinstall/splunkd.xml
11-23-2016 12:08:46.062 -0700 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /u01/splunk/splunkforwarder/etc/modules
11-23-2016 12:08:46.062 -0700 INFO loader - loading modules from /u01/splunk/splunkforwarder/etc/modules
11-23-2016 12:08:46.064 -0700 INFO loader - Writing out composite configuration file: /u01/splunk/splunkforwarder/var/run/splunk/composite.xml
11-23-2016 12:08:46.069 -0700 INFO BundlesSetup - Setup stats for /u01/splunk/splunkforwarder/etc: cpuTimeUsed=0.011632 sharedServicesGeneration=1 sharedServicesPopulation=1
11-23-2016 12:08:46.121 -0700 INFO CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:08:46.122 -0700 INFO ClusteringMgr - initing clustering with: ht=60 rf=3 sf=2 ct=60 st=60 rt=60 rct=60 rst=60 rrt=60 rmst=600 rmrt=600 pe=1 im=0 is=0 mob=5 mor=5 pb=5 rep_port:
11-23-2016 12:08:46.149 -0700 INFO ClusteringMgr - clustering disabled
11-23-2016 12:08:46.172 -0700 WARN DeploymentClient - Property targetUri not found. DeploymentClient is disabled.
11-23-2016 12:08:46.173 -0700 INFO LicenseMgr - Initing LicenseMgr
11-23-2016 12:08:46.173 -0700 INFO LMConfig - serverName=iv54318p guid=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.173 -0700 INFO LMConfig - connection_timeout=30
11-23-2016 12:08:46.173 -0700 INFO LMConfig - send_timeout=30
11-23-2016 12:08:46.173 -0700 INFO LMConfig - receive_timeout=30
11-23-2016 12:08:46.173 -0700 INFO LMConfig - squash_threshold=1000
11-23-2016 12:08:46.173 -0700 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
11-23-2016 12:08:46.173 -0700 INFO LMStackMgr - closing stack mgr
11-23-2016 12:08:46.173 -0700 INFO LMSlaveInfo - all slaves cleared
11-23-2016 12:08:46.173 -0700 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
11-23-2016 12:08:46.173 -0700 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
11-23-2016 12:08:46.173 -0700 INFO LMStackMgr - init completed [C46EAB6B-7D93-4072-BDEA-D5D4DDED627A,Forwarder,runContext_splunkd=true]
11-23-2016 12:08:46.173 -0700 INFO LicenseMgr - StackMgr init complete...
11-23-2016 12:08:46.173 -0700 INFO LMTracker - init'ing slaveId=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A label=iv54318p [30,30,30,self,0,]
11-23-2016 12:08:46.176 -0700 INFO LMTracker - attempting to ping master=self from slave=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.176 -0700 INFO LMSlaveInfo - new slave='C46EAB6B-7D93-4072-BDEA-D5D4DDED627A' created
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO LMTracker - setting masterGuid='C46EAB6B-7D93-4072-BDEA-D5D4DDED627A'
11-23-2016 12:08:46.184 -0700 INFO LMTracker - attempting to ping master=self from slave=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A success
11-23-2016 12:08:46.184 -0700 INFO LicenseMgr - Tracker init complete...
11-23-2016 12:08:46.185 -0700 WARN DeploymentProcessor - License feature=DeployServer not enabled, cannot bring up Deployment Server
11-23-2016 12:08:46.187 -0700 INFO IndexProcessor - running splunkd specific init
11-23-2016 12:08:46.187 -0700 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
11-23-2016 12:08:46.187 -0700 INFO loader - Initializing from configuration
11-23-2016 12:08:46.189 -0700 WARN ThruputProcessor - unable to find out defaultDatabase from indexes.conf, defaulting to 'main'
11-23-2016 12:08:46.189 -0700 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
11-23-2016 12:08:46.190 -0700 INFO CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:08:46.190 -0700 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
11-23-2016 12:08:46.190 -0700 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
11-23-2016 12:08:46.190 -0700 INFO UDPInputProcessor - Registering metrics callback for: udpin_connections
11-23-2016 12:08:46.218 -0700 WARN ThruputProcessor - unable to find out defaultDatabase from indexes.conf, defaulting to 'main'
11-23-2016 12:08:46.219 -0700 INFO TcpOutputProc - Initializing with fwdtype=lwf
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to 10.62.64.53:9992
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to cgyut20841:9992
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - tcpout group group1 using Auto load balanced forwarding
11-23-2016 12:08:46.232 -0700 INFO TcpOutputProc - Group group1 initialized with maxQueueSize=512000 in bytes.
11-23-2016 12:08:46.232 -0700 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
11-23-2016 12:08:46.232 -0700 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
11-23-2016 12:08:46.232 -0700 INFO PipelineComponent - Launching the pipelines.
11-23-2016 12:08:46.232 -0700 INFO loader - Server supporting SSL v2/v3
11-23-2016 12:08:46.232 -0700 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
11-23-2016 12:08:46.253 -0700 INFO TailingProcessor - TailWatcher initializing...
11-23-2016 12:08:46.253 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
11-23-2016 12:08:46.257 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
11-23-2016 12:08:46.258 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
11-23-2016 12:08:46.258 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
11-23-2016 12:08:46.258 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
11-23-2016 12:08:46.259 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///u01/logs/soaserver1logs/soa_server1.out.
11-23-2016 12:08:46.259 -0700 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
11-23-2016 12:08:46.261 -0700 INFO WatchedFile - Will begin reading at offset=4129492 for file='/u01/logs/soaserver1logs/soa_server1.out'.
11-23-2016 12:08:46.532 -0700 INFO TcpOutputProc - Connected to idx=10.62.64.53:9992
11-23-2016 12:09:16.193 -0700 INFO CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:09:16.508 -0700 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
11-23-2016 12:09:17.285 -0700 INFO TcpOutputProc - Connected to idx=10.62.64.53:9992
11-23-2016 12:09:17.916 -0700 INFO TailingProcessor - ...continuing.
11-23-2016 12:10:53.156 -0700 INFO BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.1'.
11-23-2016 12:12:30.001 -0700 INFO BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.2'.
11-23-2016 12:13:46.353 -0700 INFO ThruputProcessor - Current data throughput (259 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
11-23-2016 12:14:07.886 -0700 INFO BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.3'.
11-23-2016 12:15:44.731 -0700 INFO BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.4'.
11-23-2016 12:17:21.578 -0700 INFO BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.5'.
Please help.
Thanks.
Ash
↧