hello, I have a strange question, This question is described as a bit rough.
I have a single site cluster that contains 5 indexers, 4 search heads, a deploye, a cluster master, some deployment servers, some heavy forwarders, and some universal forwarders. The deployment server also acts as the role of a heavy forwarder.
The search factor of indexer clustering is 2 and replication factor is 3. Universal forwarder monitor log files then forward to HF, then hf forward it to indexers cluster.
Strange things always happen unreasonably. When the cluster is running for a period of time, some sourcetype event will be duplicated, Sometimes, each event is repeated 5 times. if I restart heavy forwarders. The repetition of the phenomenon will disappear. The whole cluster will return to normal but sometimes I need to restart their universal forwarder for it to work.
Some soucetype events have been duplicate again and I will need to restart HF OR UF to return to normal state.
I tried to find out the reason from the indexer's splunkd.log, but I didn't find any clues.
I think index replication has a problem but I couldn't find any error logs. Why does it return to normal when I restart HF or UF?
↧