I have a 50G dev license sandbox where I've installed NMON on the indexer and TA_nmon on one of the universal forwarders (manually since my dev instance doesn't seem to allow a deployment server). But I never see data arrive at the indexer.
On the forwarder, I can see csv files cyclically come and go in
`/opt/splunkforwarder/var/log/nmon/var/csv_repository/`
But nothing ever shows up on the indexer. E.g., `index=mon` or `index=*mon*` show no results.
*[Note that the above us under* .../var/log/ *on my install and not* .../var/run/ *per the trouble shooting article*]
If I search on `index=_internal host=myUFHost *nmon*` I see lots of results saying things like:
WatchedFile - WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_57_VM.nmon.csv'.
and
WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_11_VM.nmon.csv'.
If I constrain the search for a given `file=`, I can see that at least some these messages repeat roughly hourly for a given file name. (I'm guessing the numbers are minute w/in the given hour?)
I did some searching on these messages and saw some suggestion that perhaps the UF tries to read the file before it's populated? Or perhaps it's getting deleted before processing completes?
With some help from folks on the Splunk Slack#getting-data-in channel I blithely tried `index=_internal "drop" "index"` and got a few hits like this on sourcetype=mongod:
2019-07-18T22:01:01.226Z I STORAGE [conn967] dropCollection: s_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c (7bdb7e61-4fa5-48ff-bf30-2fe97841eaa6) - index namespace 's_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c.$_UserAndKeyUniqueIndex' would be too long after drop-pending rename. Dropping index immediately.
Any guidance would be greatly appreciated.
Platform:
- Splunk Enterprise 7.0.3
- Linux RHEL5 64bit (2.6.18-419.el5)
Places I've looked:
- https://answers.splunk.com/answers/400165/nmon-performance-monitor-for-unix-and-linux-system-5.html
- http://nmonsplunk.wikidot.com/documentation:userguide:troubleshoot:troubleguide
- https://answers.splunk.com/answers/126878/what-more-can-i-do-to-solve-file-too-small-to-check-seekcrc-probably-truncated-will-
Thanks!
↧
