the reason for this is because someone made a mix-up on the UF and then some hosts are indexing to the wrong index. Is there an easy way to find the Index to which each hosts are indexing different data?

We want to monitor Active Directory changes and security Events We are planning to deploy the Universal forwarder to each domain controller. I am confused by documentation. What is needed/best practice to accomplish this? Do we need to install ad-ons to the universal forwarder? Can we just monitor Windows Event security logs?

alt text I have installed universal forwarder on my windows host and the forwarder does forward the events to the Splunk server on port 997. As you can see in the wireshark picture the Splunk server recognizes the windows forwarder, but on the search, it can't seem to find any events related to windows or the windows host. What am I doing wrong?

I have universal forwarder with Splunk_TA_Stream and my app _server_app_audit where in inputs.conf I write `_TCP_Routing = mygroup1` or 2 at each app. After that, I write into outputs.conf `[tcpout:mygroup1 or 2]` ```server = index1:9997 or 2``` at each app but stream sends data to all indexes.

We want to monitor Active Directory changes and security Events We are planning to deploy the Universal forwarder to each domain controller. I am confused by the documentation. What is needed/best practice to accomplish this? Do we need to install add-ons to the universal forwarder? Can we just monitor Windows Event security logs?

I want to monitor log files and some custom files from /tmp/log_folder on a linux server. On the Linux box, the desired logs are scripted to /tmp/log_folder/ and this folder will be monitored by the UF. There is a script to clear out the folder every hour, any file older than 1 day. So far, I installed a UF on the server. Besides creating an inputs app (inputs.conf) on the UF and adding the monitoring stanza [monitor///tmp/log_folder/*] index=special_logs sourcetype = log_sourcetype ignoreOlderThan = 1d Do I need to add anything else? Thank you

Hello everyone, I have a lab in a Ubuntu VM. In this lab, I have the UF and the Splunk E. The forwarder monitors a folder that has a Catalina.out.bk file. The data arrives at Splunk E but it arrives at the Main index and it doesn't get all the records in (Only ~4.8k out of ~18k events) Here is my `inputs.conf` from `etc/apps/search/local/`: [monitor:///home/c137/Documents/fwrd] disabled = false index = idx-vru-test Here is my `input.conf` from `etc/system/local/`: [monitor:///home/c137/Documents/fwrd] disabled = false index = idx-vru-test I know I have different directories in each `inputs.conf`, the reason behind it is for testing stuff. This is my `outputs.conf` in `etc/apps/search/local` and `etc/system/local`: [tcpout] defaultGroup = idx-vru-test [tcpout:idx-vru-test] server = ubuntu:9997 [tcpout-server://ubuntu:9997] Attached below a picture of my Splunk indexes: ![Splunk E Indexes][1] [1]: /storage/temp/236624-indexes.png And for your ease I added the logs below for debugging: the logs added are `splunkd.log` and `metrics.log` from `var/logs/splunk`: #splunkd.log 04-02-2018 17:53:18.854 -0400 INFO PipelineComponent - Performing early shutdown tasks 04-02-2018 17:53:18.854 -0400 INFO loader - Shutdown HTTPDispatchThread 04-02-2018 17:53:18.854 -0400 INFO ShutdownHandler - Shutting down splunkd 04-02-2018 17:53:18.854 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin" 04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker" 04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore" 04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_KVStore" 04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput" 04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1" 04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Running shutdown level 1. Closing listening ports. 04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Shutting down listening ports 04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Setting up input quiesce timeout for : 90.000 secs 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ExecSendInitialSigterm" 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput" 04-02-2018 17:53:19.781 -0400 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput" 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput" 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput" 04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpInput" 04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - Cleaning up TCP connections 04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - Shutting down existing connections. 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_CacheManager" 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler" 04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - TCP connection cleanup complete 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput" 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput" 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot" 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_MainThread" 04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Exec" 04-02-2018 17:53:19.783 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Tailing" 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Received shutdown signal. 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Will reconfigure input. 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Calling addFromAnywhere in TailWatcher=0x7f9e6d3fb8b0. 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Shutting down with TailingShutdownActor=0x7f9e7302d940 and TailWatcher=0x7f9e6d3fb8b0. 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Pausing TailReader module... 04-02-2018 17:53:19.783 -0400 INFO TailReader - State transitioning from 0 to 1 (pseudoPause). 04-02-2018 17:53:19.783 -0400 INFO TailReader - State transitioning from 0 to 1 (pseudoPause). 04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Removing TailWatcher from eventloop... 04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - ...removed. 04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - Eventloop terminated successfully. 04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - Signaling shutdown complete. 04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_PeerManager" 04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager" 04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread" 04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor" 04-02-2018 17:53:19.808 -0400 INFO TailReader - State transitioning from 1 to 2 (signalShutdown). 04-02-2018 17:53:19.808 -0400 INFO TailReader - Shutting down batch-reader 04-02-2018 17:53:19.808 -0400 INFO TailReader - State transitioning from 1 to 2 (signalShutdown). 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_DeploymentClient" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AsyncQueuedMessageDispatcherThread" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_OfflineFlusher" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Slave" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SlaveSearch" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Captain" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Select" 04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_IdataDO_Collector" 04-02-2018 17:53:20.753 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput2" 04-02-2018 17:53:20.753 -0400 INFO PipeFlusher - Flushing pipelines... 04-02-2018 17:53:20.753 -0400 INFO TcpOutputProc - Shutting down auto load balanced connection strategy 04-02-2018 17:53:20.753 -0400 INFO PipeFlusher - Finished triggering pipeline flush. 04-02-2018 17:53:20.760 -0400 INFO TcpOutputProc - Auto load balanced connection strategy shutdown finished 04-02-2018 17:53:20.760 -0400 INFO TcpOutputProc - Received shutdown control key. 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_IndexerService" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Database1" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_LastIndexerLevel" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AWSMetering" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput2" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SearchDispatch" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_LoadLDAPUsers" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_MetricsManager" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Pipeline" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Queue" 04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClient" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_DmcProxyHttpClient" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Duo2FAHttpClient" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ApplicationLicenseChecker" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_S3ConnectionPoolManager" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TelemetryMetricBuffer" 04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - Shutdown complete in 1906.8 milliseconds 04-02-2018 17:53:21.751 -0400 INFO loader - All pipelines finished. 04-02-2018 17:53:24.226 -0400 INFO ServerConfig - My GUID is 1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED 04-02-2018 17:53:24.226 -0400 INFO ServerConfig - My server name is "ubuntu". 04-02-2018 17:53:24.226 -0400 INFO ServerConfig - Found no site defined in server.conf 04-02-2018 17:53:24.227 -0400 INFO ServerConfig - My hostname is "ubuntu". 04-02-2018 17:53:24.247 -0400 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000 04-02-2018 17:53:24.247 -0400 INFO ServerConfig - Setting HTTP server compression state=on 04-02-2018 17:53:24.247 -0400 INFO ServerConfig - Setting HTTP client compression state=0 (false) 04-02-2018 17:53:24.261 -0400 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 7677. The recommended value is: 16000. 04-02-2018 17:53:24.261 -0400 INFO loader - Regex JIT enabled 04-02-2018 17:53:24.261 -0400 INFO loader - using CLOCK_MONOTONIC 04-02-2018 17:53:24.262 -0400 INFO loader - Splunkd starting (build fa31da744b51). 04-02-2018 17:53:24.262 -0400 INFO loader - System info: Linux, ubuntu, 4.13.0-37-generic, #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018, x86_64. 04-02-2018 17:53:24.262 -0400 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 1970MB RAM 04-02-2018 17:53:24.262 -0400 INFO loader - Maximum number of threads (approximate): 985 04-02-2018 17:53:24.262 -0400 INFO loader - Arguments are: "-p" "8087" "restart" 04-02-2018 17:53:24.262 -0400 INFO loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml 04-02-2018 17:53:24.263 -0400 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules 04-02-2018 17:53:24.263 -0400 INFO loader - loading modules from /opt/splunkforwarder/etc/modules 04-02-2018 17:53:24.263 -0400 INFO loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml 04-02-2018 17:53:24.289 -0400 INFO ServerRoles - Declared role=universal_forwarder. 04-02-2018 17:53:24.291 -0400 INFO BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=9, cpu_time_used=0.00538, shared_services_generation=1, shared_services_population=1 04-02-2018 17:53:24.323 -0400 INFO LicenseMgr - Initing LicenseMgr 04-02-2018 17:53:24.323 -0400 INFO LMConfig - serverName=ubuntu guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED 04-02-2018 17:53:24.323 -0400 INFO LMConfig - connection_timeout=30 04-02-2018 17:53:24.323 -0400 INFO LMConfig - send_timeout=30 04-02-2018 17:53:24.323 -0400 INFO LMConfig - receive_timeout=30 04-02-2018 17:53:24.323 -0400 INFO LMConfig - squash_threshold=2000 04-02-2018 17:53:24.323 -0400 INFO LMConfig - strict_pool_quota=1 04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting='' 04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0 04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0 04-02-2018 17:53:24.323 -0400 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true 04-02-2018 17:53:24.323 -0400 INFO LMStackMgr - closing stack mgr 04-02-2018 17:53:24.326 -0400 INFO LMSlaveInfo - all slaves cleared 04-02-2018 17:53:24.327 -0400 INFO LMStack - Added type=forwarder license, from file=splunkforwarder.lic, to stack=forwarder of group=Forwarder 04-02-2018 17:53:24.327 -0400 INFO LMStackMgr - created stack='forwarder' 04-02-2018 17:53:24.327 -0400 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder 04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - added pool auto_generated_pool_free to stack free 04-02-2018 17:53:24.328 -0400 INFO ServerRoles - Declared role=license_master. 04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - Initialized hideQuotaWarning = "0" 04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - init completed [1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED,Forwarder,runContext_splunkd=true] 04-02-2018 17:53:24.328 -0400 INFO LicenseMgr - StackMgr init complete... 04-02-2018 17:53:24.328 -0400 INFO LMTracker - Setting default product type='enterprise' 04-02-2018 17:53:24.328 -0400 INFO LMTracker - init'ing slaveId=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED label=ubuntu [30,30,self] 04-02-2018 17:53:24.330 -0400 INFO LMTracker - enabling implicit feature set 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - attempting to ping master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED 04-02-2018 17:53:24.330 -0400 INFO LMSlaveInfo - new slave='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED' created 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AWSMarketplace state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AllowDuplicateKeys state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DisableQuotaEnforcement state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=HideQuotaWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SubgroupId state=DISABLED_DUE_TO_LICENSE (featureStatus=2) 04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1) 04-02-2018 17:53:24.331 -0400 INFO LMTracker - setting masterGuid='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED' 04-02-2018 17:53:24.331 -0400 INFO LMTracker - attempting to contact master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED success 04-02-2018 17:53:24.331 -0400 INFO LicenseMgr - Tracker init complete... 04-02-2018 17:53:24.331 -0400 INFO loader - Setting SSL configuration. 04-02-2018 17:53:24.331 -0400 INFO loader - Server supporting SSL versions TLS1.2 04-02-2018 17:53:24.331 -0400 INFO loader - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 04-02-2018 17:53:24.331 -0400 INFO loader - Using ECDH curves : prime256v1, secp384r1, secp521r1 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 50 parameters: start_from, use_old_eventlog_api, use_threads, thread_wait_time_msec, suppress_checkpoint, suppress_sourcename, suppress_keywords, suppress_type, suppress_task, suppress_opcode, current_only, batch_size, checkpointInterval, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, evt_resolve_ad_ds, evt_ad_cache_disabled, evt_ad_cache_exp, evt_ad_cache_exp_neg, evt_ad_cache_max_entries, evt_sid_cache_disabled, evt_sid_cache_exp, evt_sid_cache_exp_neg, evt_sid_cache_max_entries, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whitelist7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, key, suppress_text, renderXml 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 19 parameters: remoteAddress, process, user, addressFamily, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, sid_cache_disabled, sid_cache_exp, sid_cache_exp_neg, sid_cache_max_entries, disabled, index 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interval, disabled, index 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled, index, printSchema, baseline 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 12 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly, formatString 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule 04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token 04-02-2018 17:53:24.528 -0400 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts 04-02-2018 17:53:24.530 -0400 INFO DS_DC_Common - Initializing the PubSub system. 04-02-2018 17:53:24.530 -0400 INFO DS_DC_Common - Initializing core facilities of PubSub system. 04-02-2018 17:53:24.539 -0400 INFO DC:DeploymentClient - target-broker clause is missing. 04-02-2018 17:53:24.539 -0400 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config. 04-02-2018 17:53:24.539 -0400 INFO DS_DC_Common - Deployment Client not initialized. 04-02-2018 17:53:24.539 -0400 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder. 04-02-2018 17:53:24.539 -0400 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 is=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true 04-02-2018 17:53:24.539 -0400 INFO ClusteringMgr - clustering disabled 04-02-2018 17:53:24.539 -0400 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one. 04-02-2018 17:53:24.539 -0400 INFO SHClusterMgr - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10 04-02-2018 17:53:24.539 -0400 INFO SHClusterMgr - shpooling disabled 04-02-2018 17:53:24.545 -0400 INFO ulimit - Limit: virtual address space size: unlimited 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: data segment size: unlimited 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: resident memory size: unlimited 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited] 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] 04-02-2018 17:53:24.546 -0400 WARN ulimit - Core file generation disabled. 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: data file size: unlimited 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: open files: 64000 files [hard maximum: 1048576 files] 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: user processes: 7677 processes 04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: cpu time: unlimited 04-02-2018 17:53:24.546 -0400 INFO ApplicationLicense - app license disabled by conf setting. 04-02-2018 17:53:24.546 -0400 INFO IndexerInit - running splunkd specific init 04-02-2018 17:53:24.549 -0400 INFO IntrospectionGenerator:disk_objects - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false 04-02-2018 17:53:24.549 -0400 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600.000s 04-02-2018 17:53:24.549 -0400 INFO loader - Initializing from configuration 04-02-2018 17:53:24.561 -0400 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file 04-02-2018 17:53:24.563 -0400 INFO TcpInputProc - Registering metrics callback for: tcpin_connections 04-02-2018 17:53:24.654 -0400 INFO ChunkedLBProcessor - Initializing the chunked line breaking processor 04-02-2018 17:53:24.655 -0400 INFO TcpOutputProc - Initializing with fwdtype=lwf 04-02-2018 17:53:24.666 -0400 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .* 04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.* 04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry) 04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to ubuntu:9997 04-02-2018 17:53:24.669 -0400 INFO TcpOutputProc - tcpout group idx-vru-test using Auto load balanced forwarding 04-02-2018 17:53:24.669 -0400 INFO TcpOutputProc - Group idx-vru-test initialized with maxQueueSize=512000 in bytes. 04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file 04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file 04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file 04-02-2018 17:53:24.712 -0400 INFO PipelineComponent - Launching the pipelines for set 0. 04-02-2018 17:53:24.749 -0400 INFO TailingProcessor - TailWatcher initializing... 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/c137/Documents/fwrd. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/splunk/documents/frwd/. 04-02-2018 17:53:24.750 -0400 INFO TailReader - State transitioning from 1 to 0 (initOrResume). 04-02-2018 17:53:24.750 -0400 INFO TailReader - State transitioning from 1 to 0 (initOrResume). 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /home/c137/Documents/fwrd. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /home/splunk/documents/frwd. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk. 04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk. 04-02-2018 17:53:24.753 -0400 INFO loader - Limiting REST HTTP server to 21333 sockets 04-02-2018 17:53:24.753 -0400 INFO loader - Limiting REST HTTP server to 328 threads 04-02-2018 17:53:24.753 -0400 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: 04-02-2018 17:53:24.755 -0400 INFO TailReader - Registering metrics callback for: tailreader0 04-02-2018 17:53:24.755 -0400 INFO TailReader - Starting tailreader0 thread 04-02-2018 17:53:24.755 -0400 INFO TailReader - Registering metrics callback for: batchreader0 04-02-2018 17:53:24.755 -0400 INFO TailReader - Starting batchreader0 thread 04-02-2018 17:53:24.780 -0400 INFO WatchedFile - Resetting fd to re-extract header. 04-02-2018 17:53:24.892 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'. 04-02-2018 17:53:24.904 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'. 04-02-2018 17:53:24.905 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'. 04-02-2018 17:53:24.910 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'. 04-02-2018 17:53:24.912 -0400 INFO WatchedFile - Will begin reading at offset=200104 for file='/opt/splunkforwarder/var/log/splunk/audit.log'. 04-02-2018 17:53:24.913 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'. 04-02-2018 17:53:24.915 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'. 04-02-2018 17:53:24.916 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'. 04-02-2018 17:53:24.917 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'. 04-02-2018 17:53:24.919 -0400 INFO WatchedFile - Will begin reading at offset=3553 for file='/opt/splunkforwarder/var/log/splunk/conf.log'. 04-02-2018 17:53:24.920 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'. 04-02-2018 17:53:24.921 -0400 INFO WatchedFile - Will begin reading at offset=10937 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'. 04-02-2018 17:53:24.923 -0400 INFO WatchedFile - Will begin reading at offset=1353 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'. 04-02-2018 17:53:24.928 -0400 INFO TcpOutputProc - Connected to idx=127.0.1.1:9997, pset=0, reuse=0. 04-02-2018 17:53:24.948 -0400 INFO WatchedFile - Will begin reading at offset=4019773 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'. 04-02-2018 17:53:54.547 -0400 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views And #metrics.log 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000017, tail=0.000026, udpin=0.000000 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=5, cumulative_hits=2682 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7191, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=497.59, _tcp_KBps=0.49, _tcp_avg_thruput=1.57, _tcp_Kprocessed=751, _tcp_eps=0.21, kb=14.09 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4261380819697251, instantaneous_eps=0.1612942248602789, average_kbps=0.44710060274983593, total_k_processed=4546, kb=13.2099609375, ev=5 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.4261380682230032, instantaneous_eps=0.16129421965711366, average_kbps=0.4469039020887053, total_k_processed=4544, kb=13.2099609375, ev=5, load_average=0.28 04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000016, tail=0.000032, udpin=0.000000 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=6, cumulative_hits=251 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=6, cumulative_hits=251 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=6, cumulative_hits=253 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=6, cumulative_hits=251 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=54, inactive_channels=33, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_host_thruput, series="ubuntu", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=0, largest_size=7196, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=494.52, _tcp_KBps=0.48, _tcp_avg_thruput=0.87, _tcp_Kprocessed=519, _tcp_eps=0.21, kb=14.00 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4263004778073541, instantaneous_eps=0.19355528640617672, average_kbps=0.5274032732382895, total_k_processed=474, kb=13.21484375, ev=6 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.42630046405523603, instantaneous_eps=0.19355528016223547, average_kbps=0.5274026864157091, total_k_processed=474, kb=13.21484375, ev=6, load_average=0.28 04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000019, tail=0.000046, udpin=0.000000 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=13, cumulative_hits=2695 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=17, new_files_queued=0, fd_cache_size=3 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=56, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42685164384740165, eps=0.1612900676382802, kb=13.232421875, ev=5, avg_age=0.4, max_age=2 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.2595762026053572, eps=0.12903205411062416, kb=8.046875, ev=4, avg_age=0, max_age=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.6864278464527588, eps=0.2903221217489043, kb=21.279296875, ev=9, avg_age=0.2222222222222222, max_age=2 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7196, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=5, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=481.70, _tcp_KBps=0.47, _tcp_avg_thruput=1.51, _tcp_Kprocessed=765, _tcp_eps=0.20, kb=14.11 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4436421775932619, instantaneous_eps=0.354838137357843, average_kbps=0.44701626692097274, total_k_processed=4559, kb=13.7529296875, ev=11 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.6927282397198792, instantaneous_eps=0.38709616233187244, average_kbps=0.447604575234534, total_k_processed=4565, kb=21.474609375, ev=12, load_average=0.42 04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0 04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:thruput already registered 04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:idxSummary already registered 04-02-2018 17:53:24.909 -0400 INFO StatusMgr - Registering StatusListener StatusMgrLogger 04-02-2018 17:53:24.909 -0400 INFO StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_try, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor 04-02-2018 17:53:24.928 -0400 INFO StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_done, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 04-02-2018 17:53:52.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000018, tail=0.000039, udpin=0.000000 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=21, cumulative_hits=2716 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=3, cumulative_hits=41 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=3, cumulative_hits=39 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=23, new_files_queued=0, fd_cache_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=4, abandoned_channels=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.26965628382245843, eps=0.16128973985642373, kb=8.359375, ev=5, avg_age=0.6, max_age=3 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018617624268583287, eps=0.09677384391385424, kb=0.5771484375, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.7209210344949721, eps=0.09677384391385424, kb=22.3486328125, ev=3, avg_age=1, max_age=3 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.0091949425860138, eps=0.35483742768413223, kb=31.28515625, ev=11, avg_age=0.5454545454545454, max_age=3 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=24344, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=1498.03, _tcp_KBps=1.46, _tcp_avg_thruput=1.50, _tcp_Kprocessed=809, _tcp_eps=1.00, kb=43.89 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.2749452784111537, instantaneous_eps=0.7419329708721955, average_kbps=0.4494740527955077, total_k_processed=4598, kb=39.5234375, ev=23 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.0258594489110233, instantaneous_eps=0.5483851155118407, average_kbps=0.44927854429059444, total_k_processed=4596, kb=31.8017578125, ev=17, load_average=0.5 04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0 04-02-2018 17:53:54.547 -0400 INFO Metrics - adding new metrics group: tcpout_connections 04-02-2018 17:53:54.549 -0400 INFO Metrics - adding new metrics group: queue 04-02-2018 17:53:55.262 -0400 INFO Metrics - group=conf, action=base_initialize, count=1, wallclock_ms_total=9, wallclock_ms_max=9, cpu_total=0.00538, cpu_max=0.00538 04-02-2018 17:53:55.262 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 04-02-2018 17:53:55.262 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0 04-02-2018 17:53:55.262 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=instance, name=instance, instance_roles="universal_forwarder, license_master", index_cluster_label=none, index_cluster_status=non-clustered, license_status=ENABLED, instance_guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED, server_name=ubuntu 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=search_health_metrics, name=bundle_directory_reaper, bundle_dir_reaper_max_ms=0, bundle_dir_reaper_mean_ms=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=search_health_metrics, name=dispatch_directory_reaper, dispatch_dir_reaper_max_ms=0, dispatch_dir_reaper_mean_ms=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000000, tail=0.000113, udpin=0.000001 04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_host_thruput 04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_index_thruput 04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_source_thruput 04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_sourcetype_thruput 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=1, cumulative_hits=1 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=1, cumulative_hits=1 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=25, cumulative_hits=25 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=6, cumulative_hits=6 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=4, cumulative_hits=4 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=20, files_queued=37, new_files_queued=24, fd_cache_size=2 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=31, inactive_channels=10, new_channels=31, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_host_thruput, series="ubuntu", kbps=1.3457583647451972, eps=0.8167906810848301, kb=41.1904296875, ev=25, avg_age=0.16, max_age=3 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=1.3457583647451972, eps=0.7841190538414369, kb=41.1904296875, ev=24, avg_age=0.16666666666666666, max_age=3 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.31570874177087477, eps=0.16335813621696602, kb=9.6630859375, ev=5, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018856378614106818, eps=0.09801488173017961, kb=0.5771484375, ev=3, avg_age=0.3333333333333333, max_age=1 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.9943150306768611, eps=0.22870139070375242, kb=30.43359375, ev=7, avg_age=0.42857142857142855, max_age=3 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.3288801510618427, eps=0.490074408650898, kb=40.673828125, ev=15, avg_age=0.26666666666666666, max_age=3 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=527, largest_size=31828, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=1513.93, _tcp_KBps=1.48, _tcp_avg_thruput=1.48, _tcp_Kprocessed=44, _tcp_eps=0.77, kb=44.35 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.3417106131409169, instantaneous_eps=0.7514498326047483, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.06640625, ev=23 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.3457583647451972, instantaneous_eps=0.7841190538414369, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.1904296875, ev=24, load_average=0.46 04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0 04-02-2018 17:54:23.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0 Any help is appreciated!

We have only 2 GB of minimum disk space allocated for Splunk universal forwarder and my envirnoment team has asked to reduce the size consumed. I cleared splunk internal logs and also changes limit.conf but i found disk._objects file consuming more space. Is it safe to delete the file. the splunk version we are using is 7.0.0

I have installed Uf in one linux and splunk instance in another linux/windows. While trying to configure , uf is not forwarding data to linux/windows splunk,ping is working fine. Could you please help me on this.

Hello All, I am a newbie to distributed deployment. I was trying to specify the outputs.conf on the deployment server and the files get pushed on to the client. But there seems to be a syntax error on my outputs.conf file. My forwarders are listed on the UF as configured but not active. Following is my outputs.conf file. [tcpout] defaultGroup = indexers [tcpout:indexers] server = 192.168.1.144:9997 My status on the UF Your session is invalid. Please login. Splunk username: admin Password: Active forwards: None Configured but inactive forwards: 192.168.1.144:9997 This is what happens when i restart splunk UF on the machine Checking prerequisites... Checking mgmt port [8089]: open Checking conf files for problems... Invalid key in stanza [tek:tekgroup] in /opt/splunkforwarder/etc/apps/baseconfig/local/outputs.conf, line 2: server (value: 192.168.1.144:9997). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.0.3-fa31da744b51-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done

Hello What is the recommended way to clear an index present on all our indexers and then make all the universal forwarders re-send all the events on respective Windows server?

I've installed Splunk Enterprise on one VM and installed Universal Forwarder on another VM and I followed all the setup and all ports are opened but not able to get data onto Splunk Enterprise. I installed Universal Forwarder on the Splunk Server and followed the setup, where the data is passing to Splunk Enterprise. Could someone help me here that what is the issue to get logs data from another machine?

I have a folder which has multiple log files in format CalculationMgr-xxx(xx).log and EventMgr-xxx(xx).log where xx is a numeric value. I tried configuring 2 separate monitor stanza on UF to monitor these log files but it didn't work. So I have to configure a single stanza as below # Monitors CalculationMgr & EventMgr Log File [monitor://D:\Program Files (x86)\LogFiles\] disabled = false source = Log recursive = false queue = parsingQueue whitelist = (?i)CalculationMgr-\d+\(\d+\)\.log$|(?i)EventMgr-\d+\(\d+\)\.log$ _TCP_ROUTING = development_hf followTail = 0 ignoreOlderThan = 10d Now, I want to set separate source type for these 2 log files. So I tried doing this at both location UF and HF as per below configuration. But getting no success. On UF props.conf [source::.../LogFiles/EventMgr*.log] sourcetype = EventMgr1 [source::.../LogFiles/CalculationMgr*.log] sourcetype = CalculationMgr1 On HF props.conf [source::Log] TRANSFORMS-changesourcetype = set_sourcetype_calculationmgr, set_sourcetype_eventmgr transforms.conf [set_sourcetype_calculationmgr] REGEX = (?i)^CalculationMgr\S+ DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::CalculationMgr1 [set_sourcetype_eventmgr] REGEX = (?i)^EventMgr\S+ DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::EventMgr1 Any comment on what is wrong in this configuration? How can I achieve the results on Windows platform?

Created an app on the deployment server which is used to tell the Universal Forwarder which directories and logs to monitor. There is no issue with this aspect, the logs are being monitored as expected. What I would like to do is setup permissions on the Universal Forwarder so that other groups can read/write to the directories that are created by the UF. - Used RPM to install to `/opt/splunkforwarder` - `splunk:splunk`is used to own the files and run the service - `setgid` is configured on `/opt/splunkforwarder` - Setup File ACL permissions along with some defaults # file: opt/splunkforwarder/ # owner: splunk # group: splunk # flags: -s- user::rwx group::rwx group:splunk:rwx mask::rwx other::r-x default:user::rwx default:group::rwx default:group:splunk:rwx default:mask::rwx default:other::r-x However when an app is deployed to the UF, the mask is not set on the ACL stripping the newly created directory of the group permissions. `Access: (2700/drwx--S---) Uid: ( 205/ splunk) Gid: ( 205/ splunk)` # file: myapp/ # owner: splunk # group: splunk # flags: -s- user::rwx group::rwx #effective:--- group:splunk:rwx #effective:--- mask::--- other::--- default:user::rwx default:group::rwx default:group:splunk:rwx default:mask::rwx default:other::r-x Logging in interactively or non-interactive, the directories is created with the expected permissions. `Access: (2775/drwxrwsr-x) Uid: ( 205/ splunk) Gid: ( 205/ splunk)` # file: test/ # owner: splunk # group: splunk # flags: -s- user::rwx group::rwx group:splunk:rwx mask::rwx other::r-x default:user::rwx default:group::rwx default:group:splunk:rwx default:mask::rwx default:other::r-x I can manually add the mask `sudo setfacl -Rm m:rwX myapp/`and the effective permissions will be as intended. `Access: (2770/drwxrws---) Uid: ( 205/ splunk) Gid: ( 205/ splunk)`

hi all, we our splunk enterprise with this configuration: 1 universal forwarder 2 indexers in cluster 1 search head 1 SIEM how can i send traffic to our splunk based on syslog ? ""when we define input in our forwarder with F5 IP address and UDP port 514 we can receive data also the forwarder sends data to indexers and we can see them by our new defined index, but the data is not usable/readable because of mis-configuration in TA/add-on."" how can i configure add-on in such this structure ?

I have installed a universal forwarder on linux server and I have Splunk light cloud instance. I am able to find the forwarder in forwarder management but not in forwarder monitoring screen. I am also not able to add data from my forwarder to the search screen. Can some provide help with this.

Hi, I am trying to use one instance of Splunk Enterprise (Web) as a central place to be able to pull in resource usage data statistics for other servers/computers (CPU, Memory, HD, etc). I have set up a remote machine with the universal forwarder and have gotten it to report to Splunk Enterprise (Web) but it does not provide me with the real-time resource usage data I'm looking for. By default, the universal forwarders collect Windows logs but I'm pretty sure that doesn't include resource usage stats. How do I go about pulling in resource data stats from remote machines into a single instance of Spunk Enterprise so I can view everything from one spot? Thank you. Evan

I have attached screenshots of my search screen and universal forwarder monitoring screen. I can find them in the forwarder monitoring screen but not in the search screen. I followed the steps from below link. http://docs.splunk.com/Documentation/SplunkLight/7.0.3/GettingStarted/GettingdataintoSplunkLightusingLinux I could do till step 5, but not step 6. The New button is not available in search screen. ![alt text][1] [1]: /storage/temp/237585-splunk-forwardermonitor.png

I recently upgraded all of my Universal Forwarders (UF) to 7.0.3 from various version levels (some 6.3.3, some were 7.x). On one of the forwarders (AIX) when I run the command; `./splunk version` I get; "Splunk Universal Forwarder 7.0.3" But a search to list forwarder versions on the indexer lists a different version for the same host; `index=_internal sourcetype=splunkd group=tcpin_connections | stats first(version) by hostname` I get; "Version 7.0.2" Why are the versions being listed differently? After the upgrade, this is the only UF not listing 7.0.3 at the indexer.

Hi, I try to deploy a new forwarder since i've updated my indexer to 7.0.3. I got some problems and i found my answers on this forum. But I haven't been able to solve, below the error message in the splunkd.log 04-13-2018 13:22:44.069 +0000 INFO TcpOutputProc - Removing quarantine from idx=IPAddress:9997 04-13-2018 13:22:44.072 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer 04-13-2018 13:22:44.074 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer 04-13-2018 13:22:44.074 +0000 WARN TcpOutputProc - Applying quarantine to ip=IPAddress port=9997 _numberOfFailures=2 04-13-2018 13:22:51.491 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3 04-13-2018 13:22:51.503 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3 04-13-2018 13:23:51.505 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3 04-13-2018 13:23:51.517 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3 04-13-2018 13:24:17.921 +0000 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunkssl has been blocked for 600 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. And on my indexer : 04-13-2018 15:24:50.665 +0200 INFO ClientSessionsManager:Listener_AppEvents - Received count=1 AppEvent from DC ip=172.25.225.49 name=E4BC416F-983F-4CEF-AA47-45BA28ED0FF3 04-13-2018 15:26:42.372 +0200 ERROR TcpInputProc - Error encountered for connection from src=IPAddress:47781. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number Port 8089, 9997 listen and telnet in between works. **Forwarder outputs.conf** [tcpout] [tcpout:splunkssl] server = indexer:9997 [tcpout-server://indexer:9997] sslCertPath = /opt/splunkforwarder/etc/certs/splunk-sys-forwarder.pem sslCommonNameToCheck = indexer sslPassword = CaCertPassword sslRootCAPath = /opt/splunkforwarder/etc/certs/cacert.pem sslVerifyServerCert = false **Indexer inputs.conf** [splunktcp-ssl:9997] disabled = 0 connection_host = ip [SSL] serverCert = /opt/splunk/etc/certs/splunk-sys-indexer.pem sslPassword = CaCertPassword requireClientCert = false