[Glass] Semaphore error in stone?

[Brodbeck Andreas]

Hi all!

I have this "once/twice a day" problem, where my GsDevKit-based application terminates, and I can't find the root cause. According to the logs, the stone terminates. I am not sure, wether the reclaim gem or the stone is generating the error. Both logs have similar error messages. These lines from the stone log are probably the most important ones:

from stone log:
=============================
ViaSiteTdu.log.1-ERROR:Semaphore failure errno 43 (stnSemIdx = 102)
ViaSiteTdu.log.1:--- 04/19/21 22:33:17.502 CEST ---
ViaSiteTdu.log.1-    SHUTDOWN command was generated by  Stone failed in goToSleep, semaphore failed.
ViaSiteTdu.log.1-Shared memory SemClear idx 102 failure: errno=22,EINVAL, Invalid argument (programmer error) for lock: noLock
=============================

Environment:
GemStone 3.4.5
2GB shared cache
500MB temp obj space
Ubuntu 18.04.5 LTS

(Below are some more log files excerpts and informations)

After searching online and the manuals I am pretty clueless. Some sort of semaphore problem, it seems.

Any tiny clues?

Cheers,
Andreas

-- -- -- -- -- -- -- --
Andreas Brodbeck
Softwaremacher
www.mindclue.ch
Threema ID: JTJWSSBN

Brot? www.brotrezept.ch!




Stone log:
========================================================================
ViaSiteTdu.log.1---- 04/19/21 19:28:51.831 CEST ---
ViaSiteTdu.log.1-    Starting a SymbolCreation Gem process.
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-    Session 3 with processId 16505 is the Symbol Creation Gem.
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-    Session 4 in processId 16501 is the Master Reclaim session
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-    Session 2 with processId 16503 is the AdminGem.
ViaSiteTdu.log.1-    First login used IcuLibraryVersion version 51.2
ViaSiteTdu.log.1-ERROR:Semaphore failure errno 43 (stnSemIdx = 102)
ViaSiteTdu.log.1:--- 04/19/21 22:33:17.502 CEST ---
ViaSiteTdu.log.1-    SHUTDOWN command was generated by  Stone failed in goToSleep, semaphore failed.
ViaSiteTdu.log.1-Shared memory SemClear idx 102 failure: errno=22,EINVAL, Invalid argument (programmer error) for lock: noLock
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-HostCallDebugger: invoked at: Mon Apr 19 22:33:18 CEST 2021
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-UTL_GUARANTEE failed, File /kufi2/users/buildgss/34/build47617/src/shrpcclient.c line 645
ViaSiteTdu.log.1-
ViaSiteTdu.log.1-Begin attempt to print C-level stack for process 16472 at: Mon Apr 19 22:33:18 CEST 2021
ViaSiteTdu.log.1---- 04/19/21 22:33:18.508 CEST ---
ViaSiteTdu.log.1-   Starting checkpoint for clean shutdown.
========================================================================

reclaim gc gem log:
========================================================================
--- 04/19/21 22:31:10.677 CEST Changing to reclaimAll configuration
--- 04/19/21 22:31:10.694 CEST Restoring previous configuration
--- 04/19/21 22:33:17.514 CEST Logging out
Shared memory SemClear idx 102 failure: errno=22,EINVAL, Invalid argument (programmer error) for lock: noLock
HostCallDebugger: invoked at: Mon Apr 19 22:33:17 CEST 2021
UTL_GUARANTEE failed, File /kufi2/users/buildgss/34/build47617/src/shrpcclient.c line 645
Begin attempt to print C-level stack for process 16501 at: Mon Apr 19 22:33:17 CEST 2021
gdb is /usr/bin/gdb
===--- start gdb stacks using /opt/GsDevKit_home/server/stones/ViaSiteTdu/product/bin/pstack.gdb
========================================================================



All gems then forcibly terminate:
========================================================================
seaside8084_server-8084.log.1:--- 04/19/21 22:33:17.815 CEST Logging out
seaside8084_server-8084.log.1------------------------------------------------------
seaside8084_server-8084.log.1-GemStone: Error         Fatal
seaside8084_server-8084.log.1-Your session has been forcibly terminated, , stone shutting down
seaside8084_server-8084.log.1-Error Category: 231169 [GemStone] Number: 4059  Arg Count: 1 Context : 20 exception : 20
========================================================================


================================================
Linux: ipcs -l

------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 2621440
max total shared memory (kbytes) = 18014398509481980
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767
================================================