Hello everyone,
I have a lab in a Ubuntu VM. In this lab, I have the UF and the Splunk E. The forwarder monitors a folder that has a Catalina.out.bk file. The data arrives at Splunk E but it arrives at the Main index and it doesn't get all the records in (Only ~4.8k out of ~18k events)
Here is my `inputs.conf` from `etc/apps/search/local/`:
[monitor:///home/c137/Documents/fwrd]
disabled = false
index = idx-vru-test
Here is my `input.conf` from `etc/system/local/`:
[monitor:///home/c137/Documents/fwrd]
disabled = false
index = idx-vru-test
I know I have different directories in each `inputs.conf`, the reason behind it is for testing stuff.
This is my `outputs.conf` in `etc/apps/search/local` and `etc/system/local`:
[tcpout]
defaultGroup = idx-vru-test
[tcpout:idx-vru-test]
server = ubuntu:9997
[tcpout-server://ubuntu:9997]
Attached below a picture of my Splunk indexes:
![Splunk E Indexes][1]
[1]: /storage/temp/236624-indexes.png
And for your ease I added the logs below for debugging: the logs added are `splunkd.log` and `metrics.log` from `var/logs/splunk`:
#splunkd.log
04-02-2018 17:53:18.854 -0400 INFO PipelineComponent - Performing early shutdown tasks
04-02-2018 17:53:18.854 -0400 INFO loader - Shutdown HTTPDispatchThread
04-02-2018 17:53:18.854 -0400 INFO ShutdownHandler - Shutting down splunkd
04-02-2018 17:53:18.854 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin"
04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker"
04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore"
04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_KVStore"
04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
04-02-2018 17:53:18.859 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1"
04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Running shutdown level 1. Closing listening ports.
04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Shutting down listening ports
04-02-2018 17:53:18.859 -0400 INFO TcpInputProc - Setting up input quiesce timeout for : 90.000 secs
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ExecSendInitialSigterm"
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
04-02-2018 17:53:19.781 -0400 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
04-02-2018 17:53:19.781 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpInput"
04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - Cleaning up TCP connections
04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - Shutting down existing connections.
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_CacheManager"
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
04-02-2018 17:53:19.782 -0400 INFO TcpInputProc - TCP connection cleanup complete
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput"
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_MainThread"
04-02-2018 17:53:19.782 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Exec"
04-02-2018 17:53:19.783 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Tailing"
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Received shutdown signal.
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Will reconfigure input.
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Calling addFromAnywhere in TailWatcher=0x7f9e6d3fb8b0.
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Shutting down with TailingShutdownActor=0x7f9e7302d940 and TailWatcher=0x7f9e6d3fb8b0.
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Pausing TailReader module...
04-02-2018 17:53:19.783 -0400 INFO TailReader - State transitioning from 0 to 1 (pseudoPause).
04-02-2018 17:53:19.783 -0400 INFO TailReader - State transitioning from 0 to 1 (pseudoPause).
04-02-2018 17:53:19.783 -0400 INFO TailingProcessor - Removing TailWatcher from eventloop...
04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - ...removed.
04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - Eventloop terminated successfully.
04-02-2018 17:53:19.807 -0400 INFO TailingProcessor - Signaling shutdown complete.
04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_PeerManager"
04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager"
04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread"
04-02-2018 17:53:19.807 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor"
04-02-2018 17:53:19.808 -0400 INFO TailReader - State transitioning from 1 to 2 (signalShutdown).
04-02-2018 17:53:19.808 -0400 INFO TailReader - Shutting down batch-reader
04-02-2018 17:53:19.808 -0400 INFO TailReader - State transitioning from 1 to 2 (signalShutdown).
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_DeploymentClient"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AsyncQueuedMessageDispatcherThread"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_OfflineFlusher"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Slave"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SlaveSearch"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Captain"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Select"
04-02-2018 17:53:20.752 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_IdataDO_Collector"
04-02-2018 17:53:20.753 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput2"
04-02-2018 17:53:20.753 -0400 INFO PipeFlusher - Flushing pipelines...
04-02-2018 17:53:20.753 -0400 INFO TcpOutputProc - Shutting down auto load balanced connection strategy
04-02-2018 17:53:20.753 -0400 INFO PipeFlusher - Finished triggering pipeline flush.
04-02-2018 17:53:20.760 -0400 INFO TcpOutputProc - Auto load balanced connection strategy shutdown finished
04-02-2018 17:53:20.760 -0400 INFO TcpOutputProc - Received shutdown control key.
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_IndexerService"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Database1"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_LastIndexerLevel"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_AWSMetering"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput2"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_SearchDispatch"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_LoadLDAPUsers"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_MetricsManager"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Pipeline"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Queue"
04-02-2018 17:53:20.760 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClient"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_DmcProxyHttpClient"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_Duo2FAHttpClient"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_ApplicationLicenseChecker"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_S3ConnectionPoolManager"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - shutting down level "ShutdownLevel_TelemetryMetricBuffer"
04-02-2018 17:53:20.761 -0400 INFO ShutdownHandler - Shutdown complete in 1906.8 milliseconds
04-02-2018 17:53:21.751 -0400 INFO loader - All pipelines finished.
04-02-2018 17:53:24.226 -0400 INFO ServerConfig - My GUID is 1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.226 -0400 INFO ServerConfig - My server name is "ubuntu".
04-02-2018 17:53:24.226 -0400 INFO ServerConfig - Found no site defined in server.conf
04-02-2018 17:53:24.227 -0400 INFO ServerConfig - My hostname is "ubuntu".
04-02-2018 17:53:24.247 -0400 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000
04-02-2018 17:53:24.247 -0400 INFO ServerConfig - Setting HTTP server compression state=on
04-02-2018 17:53:24.247 -0400 INFO ServerConfig - Setting HTTP client compression state=0 (false)
04-02-2018 17:53:24.261 -0400 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 7677. The recommended value is: 16000.
04-02-2018 17:53:24.261 -0400 INFO loader - Regex JIT enabled
04-02-2018 17:53:24.261 -0400 INFO loader - using CLOCK_MONOTONIC
04-02-2018 17:53:24.262 -0400 INFO loader - Splunkd starting (build fa31da744b51).
04-02-2018 17:53:24.262 -0400 INFO loader - System info: Linux, ubuntu, 4.13.0-37-generic, #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018, x86_64.
04-02-2018 17:53:24.262 -0400 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 1970MB RAM
04-02-2018 17:53:24.262 -0400 INFO loader - Maximum number of threads (approximate): 985
04-02-2018 17:53:24.262 -0400 INFO loader - Arguments are: "-p" "8087" "restart"
04-02-2018 17:53:24.262 -0400 INFO loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml
04-02-2018 17:53:24.263 -0400 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules
04-02-2018 17:53:24.263 -0400 INFO loader - loading modules from /opt/splunkforwarder/etc/modules
04-02-2018 17:53:24.263 -0400 INFO loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml
04-02-2018 17:53:24.289 -0400 INFO ServerRoles - Declared role=universal_forwarder.
04-02-2018 17:53:24.291 -0400 INFO BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=9, cpu_time_used=0.00538, shared_services_generation=1, shared_services_population=1
04-02-2018 17:53:24.323 -0400 INFO LicenseMgr - Initing LicenseMgr
04-02-2018 17:53:24.323 -0400 INFO LMConfig - serverName=ubuntu guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.323 -0400 INFO LMConfig - connection_timeout=30
04-02-2018 17:53:24.323 -0400 INFO LMConfig - send_timeout=30
04-02-2018 17:53:24.323 -0400 INFO LMConfig - receive_timeout=30
04-02-2018 17:53:24.323 -0400 INFO LMConfig - squash_threshold=2000
04-02-2018 17:53:24.323 -0400 INFO LMConfig - strict_pool_quota=1
04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
04-02-2018 17:53:24.323 -0400 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
04-02-2018 17:53:24.323 -0400 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
04-02-2018 17:53:24.323 -0400 INFO LMStackMgr - closing stack mgr
04-02-2018 17:53:24.326 -0400 INFO LMSlaveInfo - all slaves cleared
04-02-2018 17:53:24.327 -0400 INFO LMStack - Added type=forwarder license, from file=splunkforwarder.lic, to stack=forwarder of group=Forwarder
04-02-2018 17:53:24.327 -0400 INFO LMStackMgr - created stack='forwarder'
04-02-2018 17:53:24.327 -0400 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
04-02-2018 17:53:24.328 -0400 INFO ServerRoles - Declared role=license_master.
04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - Initialized hideQuotaWarning = "0"
04-02-2018 17:53:24.328 -0400 INFO LMStackMgr - init completed [1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED,Forwarder,runContext_splunkd=true]
04-02-2018 17:53:24.328 -0400 INFO LicenseMgr - StackMgr init complete...
04-02-2018 17:53:24.328 -0400 INFO LMTracker - Setting default product type='enterprise'
04-02-2018 17:53:24.328 -0400 INFO LMTracker - init'ing slaveId=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED label=ubuntu [30,30,self]
04-02-2018 17:53:24.330 -0400 INFO LMTracker - enabling implicit feature set
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - attempting to ping master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.330 -0400 INFO LMSlaveInfo - new slave='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED' created
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AWSMarketplace state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=AllowDuplicateKeys state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DisableQuotaEnforcement state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=HideQuotaWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SubgroupId state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.331 -0400 INFO LMTracker - setting masterGuid='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED'
04-02-2018 17:53:24.331 -0400 INFO LMTracker - attempting to contact master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED success
04-02-2018 17:53:24.331 -0400 INFO LicenseMgr - Tracker init complete...
04-02-2018 17:53:24.331 -0400 INFO loader - Setting SSL configuration.
04-02-2018 17:53:24.331 -0400 INFO loader - Server supporting SSL versions TLS1.2
04-02-2018 17:53:24.331 -0400 INFO loader - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
04-02-2018 17:53:24.331 -0400 INFO loader - Using ECDH curves : prime256v1, secp384r1, secp521r1
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 50 parameters: start_from, use_old_eventlog_api, use_threads, thread_wait_time_msec, suppress_checkpoint, suppress_sourcename, suppress_keywords, suppress_type, suppress_task, suppress_opcode, current_only, batch_size, checkpointInterval, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, evt_resolve_ad_ds, evt_ad_cache_disabled, evt_ad_cache_exp, evt_ad_cache_exp_neg, evt_ad_cache_max_entries, evt_sid_cache_disabled, evt_sid_cache_exp, evt_sid_cache_exp_neg, evt_sid_cache_max_entries, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whitelist7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, key, suppress_text, renderXml
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 19 parameters: remoteAddress, process, user, addressFamily, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, sid_cache_disabled, sid_cache_exp, sid_cache_exp_neg, sid_cache_max_entries, disabled, index
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interval, disabled, index
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled, index, printSchema, baseline
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 12 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly, formatString
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule
04-02-2018 17:53:24.520 -0400 INFO SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token
04-02-2018 17:53:24.528 -0400 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
04-02-2018 17:53:24.530 -0400 INFO DS_DC_Common - Initializing the PubSub system.
04-02-2018 17:53:24.530 -0400 INFO DS_DC_Common - Initializing core facilities of PubSub system.
04-02-2018 17:53:24.539 -0400 INFO DC:DeploymentClient - target-broker clause is missing.
04-02-2018 17:53:24.539 -0400 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
04-02-2018 17:53:24.539 -0400 INFO DS_DC_Common - Deployment Client not initialized.
04-02-2018 17:53:24.539 -0400 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.
04-02-2018 17:53:24.539 -0400 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 is=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true
04-02-2018 17:53:24.539 -0400 INFO ClusteringMgr - clustering disabled
04-02-2018 17:53:24.539 -0400 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.
04-02-2018 17:53:24.539 -0400 INFO SHClusterMgr - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10
04-02-2018 17:53:24.539 -0400 INFO SHClusterMgr - shpooling disabled
04-02-2018 17:53:24.545 -0400 INFO ulimit - Limit: virtual address space size: unlimited
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: data segment size: unlimited
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: resident memory size: unlimited
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
04-02-2018 17:53:24.546 -0400 WARN ulimit - Core file generation disabled.
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: data file size: unlimited
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: open files: 64000 files [hard maximum: 1048576 files]
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: user processes: 7677 processes
04-02-2018 17:53:24.546 -0400 INFO ulimit - Limit: cpu time: unlimited
04-02-2018 17:53:24.546 -0400 INFO ApplicationLicense - app license disabled by conf setting.
04-02-2018 17:53:24.546 -0400 INFO IndexerInit - running splunkd specific init
04-02-2018 17:53:24.549 -0400 INFO IntrospectionGenerator:disk_objects - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false
04-02-2018 17:53:24.549 -0400 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600.000s
04-02-2018 17:53:24.549 -0400 INFO loader - Initializing from configuration
04-02-2018 17:53:24.561 -0400 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
04-02-2018 17:53:24.563 -0400 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
04-02-2018 17:53:24.654 -0400 INFO ChunkedLBProcessor - Initializing the chunked line breaking processor
04-02-2018 17:53:24.655 -0400 INFO TcpOutputProc - Initializing with fwdtype=lwf
04-02-2018 17:53:24.666 -0400 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
04-02-2018 17:53:24.668 -0400 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to ubuntu:9997
04-02-2018 17:53:24.669 -0400 INFO TcpOutputProc - tcpout group idx-vru-test using Auto load balanced forwarding
04-02-2018 17:53:24.669 -0400 INFO TcpOutputProc - Group idx-vru-test initialized with maxQueueSize=512000 in bytes.
04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
04-02-2018 17:53:24.669 -0400 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
04-02-2018 17:53:24.712 -0400 INFO PipelineComponent - Launching the pipelines for set 0.
04-02-2018 17:53:24.749 -0400 INFO TailingProcessor - TailWatcher initializing...
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/c137/Documents/fwrd.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/splunk/documents/frwd/.
04-02-2018 17:53:24.750 -0400 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
04-02-2018 17:53:24.750 -0400 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /home/c137/Documents/fwrd.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /home/splunk/documents/frwd.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
04-02-2018 17:53:24.750 -0400 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
04-02-2018 17:53:24.753 -0400 INFO loader - Limiting REST HTTP server to 21333 sockets
04-02-2018 17:53:24.753 -0400 INFO loader - Limiting REST HTTP server to 328 threads
04-02-2018 17:53:24.753 -0400 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
04-02-2018 17:53:24.755 -0400 INFO TailReader - Registering metrics callback for: tailreader0
04-02-2018 17:53:24.755 -0400 INFO TailReader - Starting tailreader0 thread
04-02-2018 17:53:24.755 -0400 INFO TailReader - Registering metrics callback for: batchreader0
04-02-2018 17:53:24.755 -0400 INFO TailReader - Starting batchreader0 thread
04-02-2018 17:53:24.780 -0400 INFO WatchedFile - Resetting fd to re-extract header.
04-02-2018 17:53:24.892 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
04-02-2018 17:53:24.904 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
04-02-2018 17:53:24.905 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
04-02-2018 17:53:24.910 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
04-02-2018 17:53:24.912 -0400 INFO WatchedFile - Will begin reading at offset=200104 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
04-02-2018 17:53:24.913 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
04-02-2018 17:53:24.915 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
04-02-2018 17:53:24.916 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
04-02-2018 17:53:24.917 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
04-02-2018 17:53:24.919 -0400 INFO WatchedFile - Will begin reading at offset=3553 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
04-02-2018 17:53:24.920 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
04-02-2018 17:53:24.921 -0400 INFO WatchedFile - Will begin reading at offset=10937 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
04-02-2018 17:53:24.923 -0400 INFO WatchedFile - Will begin reading at offset=1353 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
04-02-2018 17:53:24.928 -0400 INFO TcpOutputProc - Connected to idx=127.0.1.1:9997, pset=0, reuse=0.
04-02-2018 17:53:24.948 -0400 INFO WatchedFile - Will begin reading at offset=4019773 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
04-02-2018 17:53:54.547 -0400 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
And
#metrics.log
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000017, tail=0.000026, udpin=0.000000
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7191, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=497.59, _tcp_KBps=0.49, _tcp_avg_thruput=1.57, _tcp_Kprocessed=751, _tcp_eps=0.21, kb=14.09
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4261380819697251, instantaneous_eps=0.1612942248602789, average_kbps=0.44710060274983593, total_k_processed=4546, kb=13.2099609375, ev=5
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.4261380682230032, instantaneous_eps=0.16129421965711366, average_kbps=0.4469039020887053, total_k_processed=4544, kb=13.2099609375, ev=5, load_average=0.28
04-02-2018 17:52:50.842 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000016, tail=0.000032, udpin=0.000000
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=54, inactive_channels=33, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_host_thruput, series="ubuntu", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=0, largest_size=7196, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=494.52, _tcp_KBps=0.48, _tcp_avg_thruput=0.87, _tcp_Kprocessed=519, _tcp_eps=0.21, kb=14.00
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4263004778073541, instantaneous_eps=0.19355528640617672, average_kbps=0.5274032732382895, total_k_processed=474, kb=13.21484375, ev=6
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.42630046405523603, instantaneous_eps=0.19355528016223547, average_kbps=0.5274026864157091, total_k_processed=474, kb=13.21484375, ev=6, load_average=0.28
04-02-2018 17:52:54.925 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000019, tail=0.000046, udpin=0.000000
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=17, new_files_queued=0, fd_cache_size=3
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=56, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42685164384740165, eps=0.1612900676382802, kb=13.232421875, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.2595762026053572, eps=0.12903205411062416, kb=8.046875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.6864278464527588, eps=0.2903221217489043, kb=21.279296875, ev=9, avg_age=0.2222222222222222, max_age=2
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7196, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=5, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=481.70, _tcp_KBps=0.47, _tcp_avg_thruput=1.51, _tcp_Kprocessed=765, _tcp_eps=0.20, kb=14.11
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4436421775932619, instantaneous_eps=0.354838137357843, average_kbps=0.44701626692097274, total_k_processed=4559, kb=13.7529296875, ev=11
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.6927282397198792, instantaneous_eps=0.38709616233187244, average_kbps=0.447604575234534, total_k_processed=4565, kb=21.474609375, ev=12, load_average=0.42
04-02-2018 17:53:21.842 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:thruput already registered
04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:idxSummary already registered
04-02-2018 17:53:24.909 -0400 INFO StatusMgr - Registering StatusListener StatusMgrLogger
04-02-2018 17:53:24.909 -0400 INFO StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_try, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor
04-02-2018 17:53:24.928 -0400 INFO StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_done, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:52.842 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000018, tail=0.000039, udpin=0.000000
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=3, cumulative_hits=41
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=23, new_files_queued=0, fd_cache_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=4, abandoned_channels=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_host_thruput, series="127.0.0.1", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.26965628382245843, eps=0.16128973985642373, kb=8.359375, ev=5, avg_age=0.6, max_age=3
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018617624268583287, eps=0.09677384391385424, kb=0.5771484375, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.7209210344949721, eps=0.09677384391385424, kb=22.3486328125, ev=3, avg_age=1, max_age=3
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.0091949425860138, eps=0.35483742768413223, kb=31.28515625, ev=11, avg_age=0.5454545454545454, max_age=3
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=24344, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=1498.03, _tcp_KBps=1.46, _tcp_avg_thruput=1.50, _tcp_Kprocessed=809, _tcp_eps=1.00, kb=43.89
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.2749452784111537, instantaneous_eps=0.7419329708721955, average_kbps=0.4494740527955077, total_k_processed=4598, kb=39.5234375, ev=23
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.0258594489110233, instantaneous_eps=0.5483851155118407, average_kbps=0.44927854429059444, total_k_processed=4596, kb=31.8017578125, ev=17, load_average=0.5
04-02-2018 17:53:52.852 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:54.547 -0400 INFO Metrics - adding new metrics group: tcpout_connections
04-02-2018 17:53:54.549 -0400 INFO Metrics - adding new metrics group: queue
04-02-2018 17:53:55.262 -0400 INFO Metrics - group=conf, action=base_initialize, count=1, wallclock_ms_total=9, wallclock_ms_max=9, cpu_total=0.00538, cpu_max=0.00538
04-02-2018 17:53:55.262 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:55.262 -0400 INFO Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0
04-02-2018 17:53:55.262 -0400 INFO Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=instance, name=instance, instance_roles="universal_forwarder, license_master", index_cluster_label=none, index_cluster_status=non-clustered, license_status=ENABLED, instance_guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED, server_name=ubuntu
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=search_health_metrics, name=bundle_directory_reaper, bundle_dir_reaper_max_ms=0, bundle_dir_reaper_mean_ms=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=search_health_metrics, name=dispatch_directory_reaper, dispatch_dir_reaper_max_ms=0, dispatch_dir_reaper_mean_ms=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000000, tail=0.000113, udpin=0.000001
04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_host_thruput
04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_index_thruput
04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_source_thruput
04-02-2018 17:53:55.263 -0400 INFO Metrics - adding new metrics group: per_sourcetype_thruput
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=1, cumulative_hits=1
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=1, cumulative_hits=1
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=6, cumulative_hits=6
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=20, files_queued=37, new_files_queued=24, fd_cache_size=2
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=map, name=pipelineinputchannel, current_size=31, inactive_channels=10, new_channels=31, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_host_thruput, series="ubuntu", kbps=1.3457583647451972, eps=0.8167906810848301, kb=41.1904296875, ev=25, avg_age=0.16, max_age=3
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_index_thruput, series="_internal", kbps=1.3457583647451972, eps=0.7841190538414369, kb=41.1904296875, ev=24, avg_age=0.16666666666666666, max_age=3
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.31570874177087477, eps=0.16335813621696602, kb=9.6630859375, ev=5, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018856378614106818, eps=0.09801488173017961, kb=0.5771484375, ev=3, avg_age=0.3333333333333333, max_age=1
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.9943150306768611, eps=0.22870139070375242, kb=30.43359375, ev=7, avg_age=0.42857142857142855, max_age=3
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.3288801510618427, eps=0.490074408650898, kb=40.673828125, ev=15, avg_age=0.26666666666666666, max_age=3
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=527, largest_size=31828, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=1513.93, _tcp_KBps=1.48, _tcp_avg_thruput=1.48, _tcp_Kprocessed=44, _tcp_eps=0.77, kb=44.35
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.3417106131409169, instantaneous_eps=0.7514498326047483, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.06640625, ev=23
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=1.3457583647451972, instantaneous_eps=0.7841190538414369, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.1904296875, ev=24, load_average=0.46
04-02-2018 17:53:55.263 -0400 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:54:23.842 -0400 INFO Metrics - group=deploy-connections, nCurrent=0
Any help is appreciated!
↧