I have an application which writes .json files into a directory. I would like to be able to monitor the directory and forward all files to the indexers. The files are written once, and never updated, so I don't need to monitor the file for changes, just make sure that any new files added to the directory are forwarded for indexing. The size of the files will vary, they can be anywhere between 20K and several MB in size.
I know that I can use the [monitor:] input to do this, but it will not clean up the files. I see that the [batch:] input will cleanup the files, but I'm unclear if batch monitors the directory for new files as well. If it does, will adding useACK=true to the stanza guarantee that the file will not be deleted until the ACK is received from the indexer? My stanza in inputs.conf would look like this (I think):
[batch:///ingest/data]
index=foo
sourcetype=foo_json
useACK=true
move_policy=sinkhole
↧