Background: Externally-hosted server infrastructure feeds event data to the MSP's ArcSight implementation as a non-negotiable part of their service offering - you give them systems to manage, their event data goes into their ArcSight, because they use it for monitoring etc. with your infrastructure getting its own ArcSight Connectors to aggregate event data before passing it further up the chain.
In-house Splunk deployment then receives these systems' event data via ArcSight Connector network output - Universal Forwarders are installed on the Connector hosts, and configured with a network-monitor input to receive the event data output by the Connector application over a local TCP connection.
The Connectors are segregated by event source type, so Windows servers all feed event data to one Connector, Linux servers to another, network appliances to another, etc.
Problem: One of the Connectors receives application event data via daily batch transfer, and when it passes it on to the Splunk UF, it just keeps doing it, resulting in multiple copies - anything from 2 to a comical 230-odd - of the same event being indexed by Splunk.
The issue has been analysed by the MSP via packet capture and direct inspection of the log files received by the Connector, and it's been definitively proven that events are being resent for some unknown reason.
Luckily, the event source is reasonably quiescent, so the license and storage overhead incurred in dealing with these event duplicates is manageable, but we're mindful of what would happen if, say, our Windows event data Connector went to plaid along similar lines.
Has anyone seen anything like it before?
↧