Splunk Universal Forwarder agent keeps crashing - Agent version 6.3.0 ...Server is Linux x86_64
crashlog updated:
[splunk@ftdcslsapp638 splunk]$ cat crash-2016-10-12-11:52:08.log
[build aa7d4b1ccb80] 2016-10-12 11:52:08
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 5484 running under UID 5043.
Crashing thread: tailreader0
Registers:
RIP: [0x0000003B25A325E5] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x000000000000156C]
RSI: [0x000000000000159E]
RBP: [0x000000000187DC28]
RSP: [0x00007F72847FE868]
RAX: [0x0000000000000000]
RBX: [0x00007F726E3FF000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0xFEFEFEFEFEFEFEFF]
R9: [0x00007F7290F49F60]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x0000000001813019]
R13: [0x000000000187EFE0]
R14: [0x00007F7288BD79B8]
R15: [0x00007F72847FEDE8]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace:
[0x0000003B25A325E5] gsignal + 53 (/lib64/libc.so.6)
[0x0000003B25A33DC5] abort + 373 (/lib64/libc.so.6)
[0x0000003B25A2B70E] ? (/lib64/libc.so.6)
[0x0000003B25A2B7D0] __assert_perror_fail + 0 (/lib64/libc.so.6)
[0x0000000000A503EA] ? (splunkd)
[0x0000000000A4E6C3] _ZNK11TailWatcher12setupConfigsER15WatchedTailFile + 1507 (splunkd)
[0x0000000000A4E7D2] _ZNK11TailWatcher19initializeFileStateER15WatchedTailFileRK8Pathname + 66 (splunkd)
[0x0000000000A679F5] _ZN10TailReader10handleFileEP15WatchedTailFileP11TailWatcher + 69 (splunkd)
[0x0000000000A6A2DA] _ZN12ReaderThread4mainEv + 378 (splunkd)
[0x000000000109F0EE] _ZN6Thread8callMainEPv + 62 (splunkd)
[0x0000003B25E07AA1] ? (/lib64/libpthread.so.0)
[0x0000003B25AE8AAD] clone + 109 (/lib64/libc.so.6)
Linux / ftdcslsapp638.ftiz.cummins.com / 2.6.32-642.1.1.el6.x86_64 / #1 SMP Fri May 6 14:54:05 EDT 2016 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2016-10-11 16:01:49.002 -0500 splunkd started (build aa7d4b1ccb80)
splunkd: /home/build/build-src/ember/src/pipeline/input/Tailing.h:178: bool StatWrap::isDir() const: Assertion `_valid' failed.
2016-10-12 10:18:26.775 -0500 splunkd started (build aa7d4b1ccb80)
2016-10-12 10:20:58.906 -0500 Interrupt signal received
2016-10-12 10:21:16.753 -0500 splunkd started (build aa7d4b1ccb80)
2016-10-12 10:25:49.955 -0500 Interrupt signal received
2016-10-12 10:26:03.211 -0500 splunkd started (build aa7d4b1ccb80)
2016-10-12 10:37:14.366 -0500 Interrupt signal received
2016-10-12 10:37:31.114 -0500 splunkd started (build aa7d4b1ccb80)
splunkd: /home/build/build-src/ember/src/pipeline/input/Tailing.h:178: bool StatWrap::isDir() const: Assertion `_valid' failed.
/etc/redhat-release: Red Hat Enterprise Linux Server release 6.8 (Santiago)
glibc version: 2.12
glibc release: stable
Last errno: 2
Threads running: 35
Runtime: 4477.407956s
argv: [splunkd -p 8089 restart]
Thread: "tailreader0", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f72888ab610:
00000000 00 f7 7f 84 72 7f 00 00 |....r...|
00000008
ReaderThread: mode=0, queueSize=89704, shutdown=N, reconfigure=N, mode=0
Reading File-WatchedTailFile-WatchedFileState: path="/opt/splunkforwarder/var/run/nmon/var/csv_repository/ftdcslsapp638_12_OCT_2016_101833_CPUnn_51942_20161012103734.nmon.csv", flags=0x24213
First 144 bytes of PathnameStat @0x7f727ab95608:
00000000 00 00 00 00 00 00 00 00 80 55 b9 7a 72 7f 00 00 |.........U.zr...|
00000010 00 00 00 00 00 00 00 00 a0 ba 1f 85 72 7f 00 00 |............r...|
00000020 00 b3 8a 88 72 7f 00 00 40 a2 f1 80 72 7f 00 00 |....r...@...r...|
00000030 00 00 00 00 00 00 00 00 30 a0 97 01 00 00 00 00 |........0.......|
00000040 00 57 b9 7a 72 7f 00 00 28 b8 1f 85 72 7f 00 00 |.W.zr...(...r...|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 77 5b fe 57 00 00 00 00 0f aa 09 00 00 00 00 00 |w[.W............|
00000070 00 00 00 00 00 00 00 00 50 bd 96 01 00 00 00 00 |........P.......|
00000080 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000090
FilesystemChangeWatcher: _timeoutActive=N, _throttled=N, _waitingForNotifyCount=89705
EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y
Timeout: _when = 1476287351.633359, _initialMsec = 0
file-in: _initialized=Y, _lastCharWasNewline=N, _lastReadHadNulls=N, _wasCrcConflict=N, _warned=N
_nullsWarned=N, _wasTooNew=N, _exists=N, _noDebug=N
_hadExplicitSource=N, _crossedInitCrcLenBoundary=N, _classifiedAtLeastOnce=N, _fileReplaced=N, _readPathAfterRealEOF=N
_onlyNotifiedOnce=Y, _isArchive=N, _isCached=111213, _unowned=N, _deleteOnEOF=N
_overrideDeleteOnEOF=N, _doNotDeleteChildren=N, _readFromEnd=N, _readIrregardless=N
_fileCheckMethod=0, _crcSalt=, _origPath=
_bytesRead=0, _storingBytesRead=0, _initCrc=0x0, _seekCrc=0x0
_filenameCrc=0xc260a7a429dae89a, _fallbackCrc=0x0, _lastEOFTime=, _modTime=
_eofSeconds=3, _ignoreThresh=, _initCrcBytes=256, _initCrcForBatch=0x0
_pendingMetadata=
_prevFd=-1, _pdModels=[0 PDs]
_rescheduleDelay=1000, _rescheduleTarget=, _name=/opt/splunkforwarder/var/run/nmon/var/csv_repository/ftdcslsapp638_12_OCT_2016_101833_CPUnn_51942_20161012103734.nmon.csv, _statusName=
_st=[dev=633625, ino=0, mode=0, size=0, mtime=0, owner=0, group=1]
_toStringPrefix=state=0x0x7f727ab95580, _backoff=0
_stdataInputHeaderProcessing=[]
_detectTrailingNulls=N, _detectReadingFromOffSet=N, _readAndSkipHeader=N, _uniqueId=0
_rawPath=
x86 CPUID registers:
0: 0000000A 756E6547 6C65746E 49656E69
1: 000006F1 06010800 80002201 0FABFBFF
2: 76036301 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000077 00000002 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 20100800
80000002: 20202020 746E4920 52286C65 65582029
80000003: 52286E6F 50432029 35452055 3536342D
80000004: 76204C37 20402032 30342E32 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...
↧