[Glass] Fwd: Swazoo server hangs

[Otto Behrens]

Hi,

Ok, so this is not an obvious one - no responses.

We are getting desperate as it is happening constantly now. Any ideas
are welcome, please.

We thought that we could reproduce this problem with the stack trace
below, but this appears to be less useful because it only shows the
stack of the one active thread. It does mean that other threads could
be in an unhappy state, but we can't see it. Do you know if sigusr1
will dump more in GS 3?

Is the Zinc HTTP server perhaps a better option (in stead of Swazoo)?

Thanks
Otto

---------- Forwarded message ----------
From: Otto Behrens <otto at finworks.biz>
Date: Wed, Oct 30, 2013 at 11:50 AM
Subject: Swazoo server hangs
To: "glass at lists.gemtalksystems.com" <glass at lists.gemtalksystems.com>


Hi,

We are running GS 2.4.4.4 with Seaside30 3.0.7 and Swazoo2 2.2.0.4.

We run 4 swazoo servers reverse proxied behind nginx. The problem is
that our Swazoo server hangs up. There is a socket listening on the
known port. The process is idle, but it does not respond to requests -
connecting to the port times out.

Sending kill -USR1 <pid> gives us the output below. We have a
monitoring process that picks up this condition and kills the process.
But this causes the site to be unresponsive until it starts up again.

Do you have any suggestion how to solve this? Have you seen this
problem on your applications? Will an upgrade to GS 3 help?

Thanks
Otto

GemStone signal handler: signal 10 (SIGUSR1), received from process
28660 userId 1000
  si_code: 0, SI_USER, signal from kill(2), sigsend(2), raise(3C) or abort(3C)

Begin attempt to print C-level stack at: Wed Oct 30 11:28:01 SAST 2013


End of C-level stack:

----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------

Printing Smalltalk stack for memory usage diagnosis:
Smalltalk stack: printing to topaz .out file at [10/30/2013 11:28:01
AM.783 SAST]
    iS->ARStackPtr = 0x7f6298ea70a0, offset from base = 20
1 = TOP OF STACK,   stackDepth = 10

1  ProcessorScheduler >> _reapEvents: @IP 132  [GsMethod 498429953]
   16: 0x7f6265153ff0 (cls:66817 Array) size:0)
   15: 10 (SmallInteger 1)
   14: 10 (SmallInteger 1) <--framePtr=0x7f6298ea7090 AR[18]
  VC at 0x7f6265153f60   VC.unwindBlock= 20 (OOP_NIL)  VC.serialNum=
3005013794875392082 (SmallInteger 375626724359424010)
   13: 3005013794875392082 (SmallInteger 375626724359424010)
   12: 20 (OOP_NIL)
   11: 20 (OOP_NIL)
   10: 0x7f6265153ff0 (cls:66817 Array) size:0)
   9: 0x7f6265153ff0 (cls:66817 Array) size:0)
   8: 268 (OOP_TRUE)
   7: 2 (SmallInteger 0)
   6: 26 (SmallInteger 3)
   5: 80002 (SmallInteger 10000)
   4: 10 (SmallInteger 1)
   3: 11065002175282 (SmallInteger 1383125271910)
   2: 0x7f627f5ea790 (cls:92929 SortedCollection) size:3)
   1: 268 (OOP_TRUE)
rcvr: 0x7f627f49d758 (cls:116481 ProcessorScheduler) size:11)
[framePtr=0x7f6298ea7090 AR[18]]

2  ProcessorScheduler >> _findReadyProcess @IP 13  [GsMethod 498434561]
   1: 20 (OOP_NIL)
rcvr: 0x7f627f49d758 (cls:116481 ProcessorScheduler) size:11)
<--framePtr=0x7f6298ea7080 AR[16]

3  ProcessorScheduler >> _reschedule @IP 13  [GsMethod 498439425]
   2: 20 (OOP_NIL)
   1: 0x7f627f5ec4a8 oid:289184257 (cls:99841 GsProcess) size:22)
rcvr: 0x7f627f49d758 (cls:116481 ProcessorScheduler) size:11)
<--framePtr=0x7f6298ea7068 AR[13]

4  GsProcess >> _wait @IP 13  [GsMethod 260628481]
rcvr: 0x7f627f5ec4a8 oid:289184257 (cls:99841 GsProcess) size:22)
<--framePtr=0x7f6298ea7060 AR[12]

5  Delay >> wait @IP 54  [GsMethod 498471937]
rcvr: 0x7f6265153f10 (cls:115969 Delay) size:3)
<--framePtr=0x7f6298ea7058 AR[11]

6  WAGsSwazooAdaptor >> start @IP 20  [GsMethod 3440353793]
rcvr: 0x7f627f50e858 oid:20135281153 (cls:42137601
FinWorksGsSwazooAdaptor) size:5) <--framePtr=0x7f6298ea7050 AR[10]

7  WAServerAdaptor (C)  >> startOn: @IP 27  [GsMethod 3494994433]
   2: 0x7f627f50e858 oid:20135281153 (cls:42137601
FinWorksGsSwazooAdaptor) size:5)
   1: 64026 (SmallInteger 8003)
rcvr: 0x7f6291984060 oid:42137601 (cls:42128385
FinWorksGsSwazooAdaptor (C) ) size:19) <--framePtr=0x7f6298ea7038
AR[7]

8  WAGemStoneRunSeasideGems >> startOn: @IP 13  [GsMethod 3494978817]
   1: 64026 (SmallInteger 8003)
rcvr: 0x7f627f5f99e8 (cls:1470662657 WAGemStoneRunSeasideGems) size:3)
<--framePtr=0x7f6298ea7028 AR[5]

9  WAGemStoneRunSeasideGems (C)  >> startGemServerOn: @IP 21
[GsMethod 3494925057]
   2: 0x7f627f50e948 (cls:1470662657 WAGemStoneRunSeasideGems) size:3)
   1: 64026 (SmallInteger 8003)
rcvr: 0x7f62919839b0 oid:1470662657 (cls:1470665729
WAGemStoneRunSeasideGems (C) ) size:19) <--framePtr=0x7f6298ea7010
AR[2]

10  (Executed Code) @IP 71  [GsMethod 0x7f6294094058]
rcvr: 20 (OOP_NIL) <--framePtr=0x7f6298ea7008 AR[1]