We have a Universal Forwarder that is sending a huge amount of data. We need to only index events that contain any of these words-- "EnvisionResponse" or "EnvisionRequest" or "TransactionStatusDetail".
The "EnvisionRequest" event is multiple lines so I need all the lines for the event:
here is an example
2017-02-23 12:00:02,982 INFO (http-139.61.194.230-8380-24) EnvisionRequest version="1"
referenceNbr 869dc644e461b01
messageType P
Our Splunk Indexer is version 6.1
Can this be done in the props.conf and transforms.conf on the Indexer without adding to the daily license volume?
↧
Can data being sent from a Universal Forwarder be filtered at the indexer level for only certain events?
↧
For perfmon metrics, is it possible to specify an index at universal forwarder installation time or after installation with the CLI?
Hello
Is it possible to specify an index when you install an universal forwarder for perfmon's metrics or after with the CLI?
I don't want to modify directly the .conf file.
By default, the data is sent to "main" index.
↧
↧
Why does the Universal Forwarder index a CP1251 encoded file twice?
Hello!
I'm trying to pre-filter and forward structured .csv file from Universal Forwarder (UF) to Splunk Enterprise server. This file is CP1251 encoded, not UTF-8.
I've made a new sourcetype and copied it to props.conf file on the UF:
[lg_csv]
CHARSET = CP1251
FIELD_NAMES = time,servername,pid,tid,index4,index5,index6,k_f,address,identifier,description
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
Inputs.conf file on the UF:
[monitor://c:\cs_L]
disabled = false
index = log_test
sourcetype = lg_csv
When I put my .csv log file into c:\cs_L\*.csv, the UF indexes it and forwards to the Splunk server, but every event is duplicated. If I add data to this file and save it, added events are forwarded to Splunk server and NOT duplicated.
I tried to modify props.conf and figured out:
1. If there is no CHARSET definition in props.conf on the UF, I get every event only once - no duplication. But events fields are wrong encoded (like "\xD2\xE5.......")
2. If there is only CHARSET = CP1251 and NO_BINARY_CHECK = true definitions in props.conf on the UF, I get every event only once - no duplication. But events are not indexed on the UF and cannot be pre-filtered by TRANSFORMS-null= setnull and so on.
After running UF in the normal and debug modes and analyzing splunkd.log it seems to me, that UF indexes this file like UTF-8 encoded, computes CRC, than opens file like CP1251 encoded, computes different CRC and indexes once again.
Does anyone have any idea of solving this problem?
Version of UF is 6.5.2, version of Splunk is 6.5.0. UF is installed on Windows 2008 R2 Enterprise 64bit.
↧
Why are the Index and SourceType names in our Active Directory forests not matching?
We have two Active Directory forests in our enterprise with Universal Forwarders installed on all of our domain controllers. The sourcetype and index names in one forest do not match up with the sourcetype and index names of the other forest. Why is that and how can I get the names to be the same? I don't want to have to build different reports for the same thing because the sourcetype and index names are different.
SourceType name in Forest A: "wineventlog"
SourceType name in Forest B: "main"
Index name in Forest A --> "XmlWinEventLog: Security"
Index name in Forest B --> "WinEventLog:Security"
↧
Can I replace Splunk Universal Forwarder with Apache NiFi?
NiFi has a putSplunk processor that should do what I want (send data to an indexer)
BUT it doesn't have any place for me to specify sourcetype, or index, and it only has one "Host" field, whereas I usually use autolb with 2 indexers.
Can I do this? If so, how?
Thanks!
↧
↧
Is it possible to configure Splunk to show the filename only and not the contents of the file we're monitoring?
In the Splunk deployment we have, I'm using the Splunk universal forwarder to monitor changes to a folder, specifically when a file is added, on an sftp server. So far this is working, however it's showing not only that there has been a change, but the contents of the files in that directory. Is there a way to show the filename only and not the contents of the file, as there is sensitive information contained therein?
↧
Why does migrating universal forwarder to 6.4.3 display "unconfigured/disabled/deleted index='wineventlog'" message and event logs stopped forwarding?
Hi all,
I've 3 Splunk 6.4.1 Indexers and a Splunk 6.4.1 Search Head + Distributed Management Console (DMC) on Linux Red Hat 6.6.
I've tested Windows Event Log in Windows 2008 R2 Domain Controller Servers in Preproduction environment with 1 Splunk 6.4.1 Indexer + Search Head + DMC, forwarded Event Logs all ok.
I've migrated Splunk Universal Forwarder (SUF) in Production from 5.0.2 to 6.4.3 with clean Installation (Uninstall SUF 5.0.2, Reboot Server and Reinstall SUF 6.4.3), and before with SUF 5.0.2 Windows Events was forwarded with no problem, after SUF Clean Upgrade to 6.4.3 I receive once following message:
> Received event for unconfigured/disabled/deleted index='wineventlog' with source='source::WinEventLog:Security' host='host::my-host' sourcetype='sourcetype::WinEventLog:System' (1 missing total)
and Event Logs stopped to be forwarded.
I haven't changed configuration on my Indexers and Search Head, below my configuration:
* $SPLUNK_HOME/etc/system/local/serverclass.conf
* List item
> [serverclass:domain_controller]> host = my-dc-host> [serverclass:domain_controller:app:domain_controller]
* $SPLUNK_HOME/etc/deployment-apps/domain_controller/default/inputs.conf> [WinEventLog://Security]> disabled = 0
(I've also tried to add "index = main" on bottom of above stanza with no results).
And other configurations to send logs globally from deployment clients to deployment server...
I've tried to uninstall and reinstall SUF 6.4.3, but no issue resolved, I've also read all Splunk Answers on same problem, but before SUF upgrade Windows Event Logs was Forwarded with no problem, and in Preproduction all works fine.
Any suggestion?
Regards.
↧
Can SSL be configured when sending data to Universal Forwarder through TCP connection from an external source?
Hi,
Data is sent to Splunk Universal Forwarder (UF) through the TCP connection. From UF, data is forwarded to indexers. As we know SSL is supported by Splunk when Data is sent to Indexers. But can SSL be configured when sending data to Splunk UF through TCP connection from an external source?
↧
How to configure the forwarding of Microsoft Windows Print Server logs?
Hi Guys,
I have installed universal forwarder on Print server, Windows Server 2012 R2 and configured the receiver IP and Port on it.
On the Splunk deployment server, I tried to configure Windows Event Logs (Collect event logs from forwarders.) under Data Inputs however I don't see PrinterServer logs.
Screenshot: https://imgur.com/mEj1Kp5
I have configured the inputs.conf under local directory with the following and restarted the splunkuniversalforwarder service
[default]
host = PrintServer2012
[WinEventLog://Microsoft-Windows-PrintService/Operational]
disabled = 0
renderXml = 1
checkpointInterval = 5
evt_resolve_ad_obj = 1
start_from = newest
# only index events with these event IDs.
whitelist = 307,805
Any ideas how to get the logs into the Splunk?
↧
↧
How to resolve missing DHCP logs when my connections and configurations seem fine?
This is for troubleshooting of our Splunk Enterprise and/or Splunk universal forwarder. We have missing logs on two of our servers, Splunk universal forwarder is installed on the said two servers, config files are okay. We performed initial troubleshooting and the results are the following:
1. The connectivity from the two servers are established both in our DS and HF and yet we still haven't got any logs
2. The log file is right and currently active during this time
3. Configs on inputs and outputs are also proper
We are not sure what seems to be the problem here and hopefully someone on Splunk community can help. I have also shared the diag file on the two servers.
https://drive.google.com/open?id=0Bx_oXq4bXGfyREg5VzNmVmlUWE0
Thanks in advance.
↧
Why does using the Splunk Forwarder with Splunk Free display message "This feature is not available with your installed set of licenses"?
From my understanding the Splunk free license still lets you forward logs from other servers using the Splunk universal forwarder.
On my indexer web interface, I can view the Splunk forwarder server being connected but when I go to add data from a forwarder the page just says `This feature is not available with your installed set of licenses.` yet everything I've read seems to indicate this should be possible.
What am I missing? Thanks for any help you can provide.
↧
Is it possible to collect logs from Active/Standby application server pair without log duplication?
Hello,
We have an application which runs on 2 servers, 1 is the active server and one is a hot standby so if one server fails the other automatically picks up, we can also force it to fail over as part of normal maintenance tasks.
The problem is, the application generates logs on the currently active server, but periodically the log directory in synchronized so that we have a full set of history on both machines to make sure if one ever goes down catastrophically we can recover.
Setting up a Splunk Universal Forwarder on each of the machines will send 2 copies of the logs to Splunk.
Is there some method people have used to stop ingesting duplicate log files/entries from what is essentially 2 separate systems?
Thanks,
Tony
↧
How to disable processes run frequently by Splunk universal forwarder?
I see that these commands are executed every minute:
splunk-powershell.exe
splunk-winprintmon.exe
splunk-regmon.exe
splunk-netmon.exe
splunk-admon.exe
splunk-MonitorNoHandle.exe
The first one actually twice per minute.
Is there a way to disable these? are these some scripted inputs? I cannot locate them in the config.
I tried adding this for example to my config, but did not seem to change the anything:
[WinNetMon]
disabled = 1
[WinPrintMon]
disabled = 1
[WinRegMon]
disabled = 1
↧
↧
How to set up a Universal Forwarder to allow it to receive logs via the REST API?
We have a Universal Forwarder (UF) installation on premises that collects logs from various UF Agents and sends them to Splunk Cloud.
But we also want to be able to send logs via an API to the forwarder. These are logs that are coming from other sources that don't have the agent. How do we set up the receiver on the universal forwarder to allow it to receive logs via the REST API ?
These are all Windows based.
↧
Cannot See Universal Forwarder from Splunk Enterprise
Hello,
I have installed splunk enterprise in a windows environment. I have installed Universal Forwarder on a separate machine. Before running the ./splunk add forward_server command (to add the indexer), I ran ipconfig from the windows box where splunk enterprise is. Using that IPv4 address (lets call it xxx.xx.xxx.xxx). I then successfully pinged that address from where I installed the forwarder (a linux machine). Then, using the default forwarder port (9997), I ran the command as:
./splunk add forward-server xxx.xx.xxx.xxx:9997
which ran successfully. I then restarted forwarder like:
./splunk restart
and the forwarder successfully restarted. I verified that the outputs.config file in the splunk_home/etc/system/local had the correct settings:
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = xxx.xx.xxx.xxx:9997
[tcpout-server://xxx.xx.xxx.xxx:9997]
I then logged into the splunk enterprise web interface, and selected "Add Data" link, and then the "forward" link. At the top is says "Select Forwarders", but beneath that there is a red triangle that says "There are currently no forwarders configured as deployment clients to this instance".
Am I doing something wrong? If so, how do I diagnose and correct? Grateful for any response!
↧
WIndows 7 support
I was trying to download the universal forwarder for windows 7 32 bit OS, but i can see only windows 8, 8.1, 10 OS.
Is Splunk supporting windows 7? If I will download universal forwarder for windows 8. Will it take the data from window 7 system?
↧
Why am I unable to install Splunk universal forwarder on Windows server 2012 R2?
Hi
Unable to install Splunk universal forwarder on Windows server 2012 R2, please help to solve this issue.
Logs
04-04-2017 21:49:01.089 +0530 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
04-04-2017 21:49:01.089 +0530 INFO ServerConfig - Host name option is "".
04-04-2017 21:49:03.538 +0530 INFO loader - Running utility: "check-transforms-keys"
04-04-2017 21:49:03.538 +0530 INFO loader - Getting configuration data from: C:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
04-04-2017 21:49:03.538 +0530 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:03.538 +0530 INFO loader - loading modules from C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:03.553 +0530 INFO loader - Writing out composite configuration file: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
04-04-2017 21:49:05.363 +0530 INFO loader - Splunkd starting (build 67571ef4b87d).
04-04-2017 21:49:05.363 +0530 INFO loader - System info: Windows, TSMSRV2, 2, 6, x64.
04-04-2017 21:49:05.363 +0530 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 16383MB RAM
04-04-2017 21:49:05.363 +0530 INFO loader - Maximum number of threads (approximate): 8191
04-04-2017 21:49:05.363 +0530 INFO loader - Arguments are: "rest" "--noauth" "POST" "/services/apps/local/SplunkUniversalForwarder/enable"
04-04-2017 21:49:05.363 +0530 INFO loader - Getting configuration data from: C:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
04-04-2017 21:49:05.363 +0530 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:05.363 +0530 INFO loader - loading modules from C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:05.363 +0530 INFO loader - Writing out composite configuration file: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
04-04-2017 21:49:05.379 +0530 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
04-04-2017 21:49:05.379 +0530 INFO ServerConfig - Host name option is "".
04-04-2017 21:49:05.394 +0530 WARN AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
04-04-2017 21:49:05.410 +0530 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
04-04-2017 21:49:06.720 +0530 ERROR LimitsHandler - Configuration from app=SplunkUniversalForwarder does not support reload: limits.conf/[thruput]/maxKBps
04-04-2017 21:49:06.720 +0530 ERROR ApplicationUpdater - Error reloading SplunkUniversalForwarder: handler for limits (access_endpoints /server/status/limits/general): Bad Request
04-04-2017 21:49:06.720 +0530 ERROR ApplicationUpdater - Error reloading SplunkUniversalForwarder: handler for server (http_post /replication/configuration/whitelist-reload): Application does not exist: Not Found
04-04-2017 21:49:06.720 +0530 ERROR ApplicationUpdater - Error reloading SplunkUniversalForwarder: handler for web (http_post /server/control/restart_webui_polite): Application does not exist: Not Found
04-04-2017 21:49:06.720 +0530 WARN LocalAppsAdminHandler - User 'splunk-system-user' triggered the 'enable' action on app 'SplunkUniversalForwarder', and the following objects required a restart: default-mode, limits, server, web
04-04-2017 21:49:07.095 +0530 INFO loader - Splunkd starting (build 67571ef4b87d).
04-04-2017 21:49:07.095 +0530 INFO loader - System info: Windows, TSMSRV2, 2, 6, x64.
04-04-2017 21:49:07.095 +0530 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 16383MB RAM
04-04-2017 21:49:07.095 +0530 INFO loader - Maximum number of threads (approximate): 8191
04-04-2017 21:49:07.095 +0530 INFO loader - Arguments are: "rest" "--noauth" "POST" "/servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server" "name=192.168.6.74:9997"
04-04-2017 21:49:07.095 +0530 INFO loader - Getting configuration data from: C:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
04-04-2017 21:49:07.095 +0530 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:07.095 +0530 INFO loader - loading modules from C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:07.095 +0530 INFO loader - Writing out composite configuration file: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
04-04-2017 21:49:07.126 +0530 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
04-04-2017 21:49:07.126 +0530 INFO ServerConfig - Host name option is "".
04-04-2017 21:49:07.142 +0530 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
04-04-2017 21:49:07.563 +0530 INFO loader - Splunkd starting (build 67571ef4b87d).
04-04-2017 21:49:07.563 +0530 INFO loader - System info: Windows, TSMSRV2, 2, 6, x64.
04-04-2017 21:49:07.563 +0530 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 16383MB RAM
04-04-2017 21:49:07.563 +0530 INFO loader - Maximum number of threads (approximate): 8191
04-04-2017 21:49:07.563 +0530 INFO loader - Arguments are: "rest" "--noauth" "POST" "/servicesNS/nobody/SplunkUniversalForwarder/admin/deploymentclient/deployment-client" "targetUri=192.168.6.74:8089"
04-04-2017 21:49:07.563 +0530 INFO loader - Getting configuration data from: C:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
04-04-2017 21:49:07.563 +0530 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:07.563 +0530 INFO loader - loading modules from C:\Program Files\SplunkUniversalForwarder\etc\modules
04-04-2017 21:49:07.563 +0530 INFO loader - Writing out composite configuration file: C:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
04-04-2017 21:49:07.563 +0530 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
04-04-2017 21:49:07.563 +0530 INFO ServerConfig - Host name option is "".
04-04-2017 21:49:07.594 +0530 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
04-04-2017 21:49:07.610 +0530 WARN DC:PhonehomeThread - Phonehome thread is now shutdown.
↧
↧
CSV and TSV File Inputs on Universal Forwarder - Do I need to configure both INDEXED_EXTRACTIONS and FIELD_DELIMITER?
I am going to be forwarding CSV and TSV files, and was wondering if I need to configure **both** INDEXED_EXTRACTIONS and FIELD_DELIMITER in props.conf for the sourcetype on the Universal Forwarder.
It seems redundant to tell it
INDEXED_EXTRACTIONS= csv **and** FIELD_DELIMITER= ,
and
INDEXED_EXTRACTIONS= tsv **and** FIELD_DELIMITER= \t
If it is a csv it should be obvious the field delimiter is a comma.
And if it is a tsv it should be obvious the field delimiter is a tab.
Is there a reason to configure both? Or if only one is needed is there a reason to use one over the other?
↧
Forward a log to a different indexer without forwarding _internal index to that indexer
I have a universal forwarder (version 6.2.5) that is forwarding a monitored log file to an indexer. I want to add another monitored log file that should be sent to a different indexer.
I got this to work by adding a [tcpout:indexer2] stanza to the outputs.conf and using _TCP_ROUTING = indexer2 in inputs.conf for the new log file. However, the _internal index (splunkd.log etc.) is now being sent to both the original indexer and indexer2. I want the _internal index to be sent only to the original indexer. How can I configure the forwarder to make this happen?
Here are the outputs.conf and inputs.conf settings I am currently using:
**outputs.conf**
[tcpout]
defaultGroup = indexer1
[tcpout:indexer1]
server = server1:9997
autoLB = true
[tcpout:indexer2]
server = server2:9997
autoLB = true
**inputs.conf**
[monitor:///var/log/test1.log]
disabled = false
index = test
sourcetype = access_combined
[monitor:///var/log/test2.log]
_TCP_ROUTING = indexer2
disabled = false
index = test
sourcetype = access_combined
↧
Collecting logs from the fortified network over the firewall with the forwarders
Greetings, a beginner Splunk administrator here.
So I have the case where within my network there are two isolated network zones. One such that could be classified as intranet-oriented or less fortified one, and the other highly fortified with servers holding records needed for eventual collection.
Communication between them occurs through the firewall. Intranet zone holds the crucial indexer.
How could I collect logs from the fortified zone over the firewall between these zones with Splunk forwarder while not opening too many network ports or compromising the network in any way.
From your experience, I'd like to hear several ways in approaching this.
↧