How do I load the Universal Forwarder on a IBMi LPAR?
↧
How do I install a universal forwarder on an IBM Logical Partition (LPAR)?
↧
What are the main differences between the Universal forwarder and Heavy forwarder?
Can someone explain me in simply english the difference between there two forwards and where they are using?
↧
↧
Monitoring specific keys in the registry
I had the default registry monitoring turned on for our desktops for a day but it used way too much of our license so I had to disable it.
I am interested in monitoring a few keys but I am unclear on how to fill out the hive portion within the inputs.conf file.
Example of the keys I might monitor:
1. REGISTRY: Watch for the creation or modification of new registry keys and values
a. 4657 – Accesses: WriteData (or AddFile)
i. HKLM, HKCU & HKU – Software\Microsoft\Windows\CurrentVersion
1. Run, RunOnce
ii. HKLM\Software\Microsoft\Windows NT\CurrentVersion\Windows
1. Watch AppInit_Dlls
iii. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\EMDMgmt
1. Watch Connection time of USB Devices
iv. HKLM\System\CurrentControlSet\Services
1. Watch for NEW Services
v. HKLM\SYSTEM\CurrentControlSet\Enum\USBSTOR
1. Watch for NEW USB devices
↧
How do I reduce the number of Windows 4688 events generated by Splunk?
While logging Windows 4688 events I noticed that the Splunkd process is actually responsible for generating over 90% of the events. I am currently dropping the events generated by the Splunkd process at a heavy forwarder, but I'd like to stop Splunkd from generating them in the first place since they take up disk space on my end points.
I believe the Splunkd process keeps launching child processes that check the Windows event logs and terminate. This would mean that the Splunk UF is spending most of its time monitoring itself. I would like to know if there is a way to reduce the number of child processes generated by Splunkd so that my endpoints generate fewer 4688 (Process Created) events.
↧
Splunk universal forwarder will not start. ERROR AdminHandler:ServerControl - forcing shutdown since it did not complete in 360 seconds
Splunk Universal Forwarder running in windows. UF ver is 6.2.1
The very last entry in splunkd.log is 10-27-2017 16:19:19.825 -0400 ERROR AdminHandler:ServerControl - forcing shutdown since it did not complete in 360 seconds
When we try to start the service, it goes through the sequence of veryfing ports and configuration everything seems like is going to start then immediately get that the service has stopped.
Nothing gets written to logs except to splunkd-utility.log, when looked at this file I couldn't find anything obvious.
Tried to enable debug pretty much everywhere but again nothing gets written to logs.
Rebooting the windows server where this forwarder is installed seems to resolved the issue but I'm interested in knowing what condition could the forwarder be encountering during startup to cause this behavior.
The messages in splunkd prior to stop are below
10-27-2017 16:13:18.839 -0400 INFO DeployedApplication - Checksum mismatch 12852313919059407491 <> 17817335707338687324 for app=X. Will reload from='deploymentserver:8089/services/streams/deployment?name=default:myapp:myapp_inputs'
10-27-2017 16:13:19.198 -0400 INFO DeployedApplication - Downloaded url=deploymentserver:8089/services/streams/deployment?name=default:myapp:myapp_inputs to file='C:\Program Files\SplunkUniversalForwarder\var\run\myapp\myapp_inputs-1509132847.bundle' sizeKB=10
10-27-2017 16:13:19.214 -0400 INFO DeployedApplication - Installing app=myapp_inputs to='C:\Program Files\SplunkUniversalForwarder\etc\apps\myapp_inputs'
10-27-2017 16:13:19.667 -0400 WARN DC:DeploymentClient - Restarting Splunkd...
10-27-2017 16:13:41.245 -0400 INFO TcpOutputProc - Connected to idx=1.1.1.1:9996 using ACK.
10-27-2017 16:14:12.167 -0400 INFO TcpOutputProc - Closing stream for idx=1.1.1.1:9996
10-27-2017 16:19:19.825 -0400 ERROR AdminHandler:ServerControl - forcing shutdown since it did not complete in 360 seconds
↧
↧
Missing events when using a universal forwarder
I was trying to do a batch input with a bunch of CSVs using a universal forwarder, really simple thing:
inputs.conf:
[batch://]
move_policy=sinkhole
index=myindex
sourcetype=mysourcetype
props.conf:
[mysourcetype]
INDEXED_EXTRACTIONS = CSV
The file gets indexed, but when I'm running searches against my data, some lines are missing from the file.
I've waited for several hours (even tried a day after), but still missing, so no forwarding delay I guess. Also, newer files are already indexed.
The problematic file I've used for checking contains 190K lines. I don't see any errors in the internal logs, only a warning about too much events (100K+) with the same timestamp. My universal forwarder version is 6.6.1
I've moved the same files and the same configs to a heavy forwarder, and tested: works perfectly.
I see the same warning about the timestamp, but the whole content of the file is properly indexed.
Is there a limitation with universal forwarders for such things? Or where should I look to solve this?
↧
Data is not getting indexed through Universal Forwarder
Hello All, We are forwarding data to indexer from Universal forwarder for couple of months perfectly. Recently we are facing issues that the forwarder is not sending files to indexer and I observed log errors as
10-30-2017 12:29:04.614 +0530 ERROR BTree - 64th child has invalid offset: indexsize=134928 recordsize=291776, (Leaf)
10-30-2017 12:29:04.614 +0530 ERROR BTreeCP - addUpdate CheckValidException caught: BTree::Exception: Validation failed in checkpoint
10-30-2017 12:29:04.676 +0530 ERROR BTree - reading one headers failed: Cannot create a file when that file already exists.
10-30-2017 12:29:04.676 +0530 ERROR BTree - verifyHeaders failed
10-30-2017 12:29:04.676 +0530 ERROR TailReader - Ignoring path="C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log" due to: BTree::Exception: failed to restore checkpoint
10-31-2017 14:09:54.581 +0530 ERROR BTreeCP - open failed to restore checkpoint in btree='C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\splunk_private_db', itmay be corrupted -- run `SPLUNK_HOME/bin/btprobe -d '' -r` to attempt to repair .
Please let me know the actions to remove this error.
Thanks in Advance.
↧
Using Splunk Universal Forwarder to collect from ElasticSearch/Logstash
one of our end-user clients have massive information stored in ELK stack. Our company needs to collect those data into Splunk using Splunk Universal forwarder . They can't send us fluentd due to firewall restrictions.
- How can Splunk UF read from logstash? Does it have to query ELK api to do this?
- Can Splunk UF do polling to get data on a regular basis?
Worse case I'm asking them to write the data into a file , but wanted to see Splunk UF native intergration to ELK if its present
↧
Securing Communications between Deployment Server to Forwarder (upgraded from 6.5.3 to 6.6.x (Deprecated SSL Stanzas) Windows 2012, Windows 2008R2
I have recently upgraded from 6.5.x to 6.6.x and I am now encountering openSSL communication errors between my Deployment Server and Universal Forwarder.
SSL encryption is configured for DS to FWD and FWD to IDX. The configuration was working fine until we upgraded from 6.5 to 6.6.1 at this point the communication from the DS to the FWD stopped working. No changes were made to the certificates.
The communication between FWD to IDX via 9998 is still working fine but FWD to Deployment Server does not work with the verifyServerCert = true enabled in the server.conf of the DS configuration.
We receive the following error message CA: WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='unknown CA'.
↧
↧
Symantec 14.0 and Splunk 7.0.0 (splunkd) not playing well together
Good afternoon,
I have a problem with Symantec 14.0 and splunk 7 Universal Forwarder not playing well together. Whenever the forwarder is running, Symantic use goes to 99% for every 10 seconds out of 60. This has killed our performance on the production servers. Let me know what information you might need and I can post it. Thank you!
↧
Splunk Universal Forwarder fails port scans on AIX 7.1 servers
I have several AIX servers (AIX 7.1) with Splunk Universal Forwarder 6.5.2 that all fail Nessus port scans for allowing TLS1.0 on port 8089. All configs, verified by btool, have "sslVersions" and "sslVersionsForClient" set to tls1.2 and Splunk has been restarted many a time after making sure these configs are correct.
Have several Red Hat Linux servers (RH 6) with same Universal Forwarder version and same SSL settings in the same configs, that do NOT fail port scans for any issues related to port 8089.
Anyone encounter the same issue on AIX platforms?
The AIX platforms all have OpenSSL 1.0.1e, while the Linux platforms have OpenSSL 0.9.8e-fips-rhel5
I appreciate any insight.
↧
ignoreOlderThan in inputs.conf
Hi All,
We have Splunk environment with nearly 1000 Universal Forwarders sending logs to Indexers. These Universal Forwarders are managed by Deployment Server.
Now the issue is few of the logs from a folder are missing on Indexers. By running the btool and troubleshooting commands, we came to know that the file is being ignored with the below error:
type = ignored (modtime fell behind window of 604800 seconds ago)
But the files are modified and updated even today. So could you please help us what could be the issue and how to resolve this? The inputs stanza configured is as below:
[monitor:///myfolder/]
whitelist = (a|b|c)\.log
index = myindex
sourcetype = mysourcetype
ignoreOlderThan = 7d
recursive = true
Thanks in advance.
↧
Universal Forwarderで時間指定のログ転送
日本語ですみません。
業務要件として、1日1回決められた時間（リアルタイムではなく）にUniversal Forwarderでログ転送する必要があります。
Universal Forwarderの機能で、決められた時間にログ転送する事は可能でしょうか？
現在は、決められた時間直前にUniversal Forwarderを起動し、
ログの転送が終わる時間を見込んでUniversal Forwarderを停止する運用を考えています。
↧
↧
How to get a list of all hosts installed with Universal Forwarder
I have a bunch of agents(hosts) in Appdynamics, I wanted to figure out that the Universal Forwarder is installed or not in all those hosts to collect logs to Splunk.
**Is there any way that I can get the list of hosts that installed with UF.**
Thanks in advance.
↧
How do I FULLY uninstall Splunk Universal Forwarder
I'm running Splunk Universal Forwarder with a Splunk Enterprise deployment. On a new install, all information is populating correctly into the Splunk App for Windows Infrastructure, including the Windows Update history. However, for forwarders that previously had Splunk installed from the last Enterprise installation, this information is not being reported to the indexer.
The apps are deploying correctly, and are receiving information, but are missing this tidbit (and maybe a few others, I have not dug in too much yet). What I have done is uninstalled the Unifersal Forwarder 6.6.4 both through the Control Panel and by right clicking on the Installer. However, in both of these circumstances a lot of registry keys mentioning "Splunk" and "UniversalForwarder" are left over. I believe one of these keys is the culprit to my installation problems.
Does anyone have a suggestion as how to completely remove Splunk keys from the registry upon uninstalling?
↧
Splunk Universal forwarder not reporting data
we have some Universal forwarder Agents installed in servers in different domains , server team done patching on those servers and post patching or Server reboot , these are not reporting logs . Ours is distributed environment with search head cluster . i have checked for splunk agent services those are running fine and tried restarting the services still no logs have been reported . below is the log which it is generating
11-20-2017 23:57:27.697 +0530 WARN TcpOutputProc - Cooked connection to ip=XXXXXXX timed out
11-20-2017 23:57:47.714 +0530 WARN TcpOutputProc - Cooked connection to ip=XXXXXXXX timed out
11-20-2017 23:58:07.715 +0530 WARN TcpOutputProc - Cooked connection to ip=XXXXXXXX timed out
11-20-2017 23:58:27.043 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected .
I just tried re installing the agent still nothing . configuration end nothing has changed .
↧
incorrect epoch times in netflow data from universal forwarder
I have Stream application installed on Universal Forwarder and I've setup streamfwd as a receiver for Netflow. To be more precise, my architecture is following one:
- network traffic is mirrored to the server where UF is running
- server runs pmacctd which ingests mirrored traffic from network interface, generates Netflow data and sends it to UF
- UF sends Netflow to indexers.
Flows are comming fine, but I noticed that values of flow_start_time and flow_end_time are wrong. According to the documentation [1], these fields should have absolute time in Epoch seconds, but what I get is something entirely different. For example, 1757846, which corresponds to Wednesday, January 21, 1970 8:17:26 AM in GMT. Before Splunk, I was sending these flows to another collector, and had no issues with timestamps. Any ideas where to start troubleshooting?
I did notice that values of these fields increase when time goes by, as expected. It's just the values which are wrong. Sounds like relative value, but it's not clear where it starts from. Date on the server which runs UF is correct.
[1]: https://docs.splunk.com/Documentation/StreamApp/7.1.1/DeployStreamApp/FlowProtocols
↧
↧
How can I exclude data from being ingested by the universal forwarder?
Hello all,
I have recently set up Splunk to monitor /var/log/messages.
There is one event in this log that I would like to exclude.
The event itself really does not matter.
I would just like to know how I can keep certain types of data
from getting into Splunk, without ignoring the files which the data comes from.
Please help.
↧
Trouble getting the Windows universal forwarder to forward data
Hello all, I can't seem to get the windows universal forwarder to forward data.
- Splunk indexer (7.x.x) is on CentOS7, 8089 and 9997 open on firewall
- Latest Splunk forwarder installed on windows 10
- Did not go into customize on windows installer GUI, but did put the win event stanza from documentation into the forwarder inputs.conf (system local).
- opened 9997 data input in webui
- Turned off windows firewall for troubleshooting.
- Downloaded various windows apps/add-ons to splunk indexer thinking it was a deployment thing
What am I missing?
↧
Universal Forwarder, Server Class.
I install UF on linux client.
Than I
./splunk set deploy-poll *.*.*.*:8089
Client did not appear in Forwarder Management in Clients.
What i miss?
↧