[Glass] problem update: Zinc server with Seaside (3.1)

Sat Dec 21 01:51:10 PST 2013

Ha, a second try does yield better results. The start in topaz does not have the out-of-mem.
I do see a connection being dropped every other request…. but there is progress!

Don’t know what made it fail similarly the first time. The only change I did between this email and the one below is a restart of the stone...

Johan

On 21 Dec 2013, at 10:42, Johan Brichau <johan at yesplan.be> wrote:

> Dale,
> 
> Unfortunately I’m getting the same result in topaz and less stable behaviour from gemtools (connections are dropped)
> Looking into it…
> 
> Johan
> 
> On 20 Dec 2013, at 04:31, Dale K. Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> 
>> Johan,
>> 
>> I moved the package SocketStream to the repo (changed baseline) and commit on socketstream_fix ... let me know if this works for you..
>> 
>> Dale
>> 
>> ----- Original Message -----
>> | From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
>> | To: "Johan Brichau" <johan at yesplan.be>
>> | Cc: glass at lists.gemtalksystems.com
>> | Sent: Thursday, December 19, 2013 5:45:26 PM
>> | Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
>> | 
>> | I think I've got things working now ... I will push a branch of
>> | glassdb/zinc with my changes ... I need to clean things up a bit and
>> | test with topaz ... right now its running in tODE ... I'm still
>> | getting two connections per request, but it's much more stable now
>> | ...
>> | 
>> | Dale
>> | 
>> | ----- Original Message -----
>> | | From: "Johan Brichau" <johan at yesplan.be>
>> | | To: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
>> | | Cc: glass at lists.gemtalksystems.com
>> | | Sent: Thursday, December 19, 2013 7:46:37 AM
>> | | Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
>> | | 
>> | | Hi Dale,
>> | | 
>> | | Thanks for these pointers.
>> | | I will look into it asap.
>> | | 
>> | | One thing I noticed is that after I merged the latest changes into
>> | | Seaside3.1 [1], a lot less processes were spawning and it took more
>> | | time before the system ran out of memory.
>> | | But it might just have been a coincidence...
>> | | 
>> | | Johan
>> | | 
>> | | [1] https://github.com/glassdb/Seaside31/pull/5
>> | | 
>> | | On 19 Dec 2013, at 06:38, Dale K. Henrichs
>> | | <dale.henrichs at gemtalksystems.com> wrote:
>> | | 
>> | | > Just a touch more info:
>> | | > 
>> | | > ZnMultiThreadedServer>>listenLoop sents #repeat to a block ...
>> | | > 
>> | | > if `serverSocket isValid` returns false, the loop is exited but
>> | | > it
>> | | > launches another
>> | | > #listenLoop ... if we get consistent invalid serverSockets, this
>> | | > guy will spin out of control..
>> | | > 
>> | | > Inside the block, ZnMultiThreadedServer>>serveConnectionsOn: is
>> | | > called (right before the repeat).
>> | | > 
>> | | > In ZnMultiThreadedServer>>serveConnectionsOn: if
>> | | > #waitForAcceptFor:
>> | | > returns a non-nil response, we are off to the races with a forked
>> | | > process and a return to ZnMultiThreadedServer>>listenLoop which
>> | | > itself spins up a new call to
>> | | > ZnMultiThreadedServer>>serveConnectionsOn: ....
>> | | > 
>> | | > Soooo, it looks like SocketStreamSocket>>waitForAcceptFor: had
>> | | > better function correctly .... and serverSocket had better not go
>> | | > into terminal invalidity...
>> | | > 
>> | | > Some halts or breakpoints in and around these particular methods
>> | | > should yield some good information ... or point us in a more
>> | | > productive direction.
>> | | > 
>> | | > Dale
>> | | > 
>> | | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
>> | | > To: "Johan Brichau" <johan at yesplan.be>
>> | | > Cc: glass at lists.gemtalksystems.com
>> | | > Sent: Wednesday, December 18, 2013 8:59:47 PM
>> | | > Subject: Re: [Glass] problem update: Zinc server with Seaside
>> | | > (3.1)
>> | | > 
>> | | > Johan,
>> | | > 
>> | | > I've collected more info on two processes[1] ... first the one
>> | | > that
>> | | > gets an "index out of range error" and the other process that was
>> | | > sitting in the process list waiting for more input from the
>> | | > socket, but the socket was closed (at leasted #isConnected is
>> | | > false).
>> | | > 
>> | | > I hit the out of memory earlier but didn't collect any meaningful
>> | | > info ... I'm assuming that the multi-threaded boy is hitting an
>> | | > error somewhere and spinning away like crazy ...
>> | | > 
>> | | > Need to catch the error in progress and perhaps find out from
>> | | > Sven
>> | | > if there is supposed to be a "deadman switch" on this guy to
>> | | > avoid
>> | | > infinite errors ..
>> | | > 
>> | | > Dale
>> | | > 
>> | | > [1]
>> | | > https://github.com/dalehenrich/todeHome/blob/master/home/seaside3.1.0/stacks
>> | | > 
>> | | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
>> | | > To: "Johan Brichau" <johan at yesplan.be>
>> | | > Cc: glass at lists.gemtalksystems.com
>> | | > Sent: Wednesday, December 18, 2013 6:39:13 PM
>> | | > Subject: Re: [Glass] problem update: Zinc server with Seaside
>> | | > (3.1)
>> | | > 
>> | | > Here's the stack:
>> | | > 
>> | | > 1. OffsetError(AbstractException)>>_signalWith: @5 line 25
>> | | > 2. OffsetError(AbstractException)>>signal @2 line 47
>> | | > 3. ByteArray(Object)>>_error:args: @15 line 11
>> | | > 4. ByteArray(Object)>>_errorIndexOutOfRange: @2 line 6
>> | | > 5. ByteArray>>at: @4 line 13
>> | | > 6. SocketStream>>next @13 line 10
>> | | > 7. ZnLineReader>>processNext @5 line 4
>> | | > 8. ZnLineReader>>nextLine @3 line 3
>> | | > 9. ZnRequestLine>>readFrom: @3 line 3
>> | | > 10. ZnRequestLine class>>readFrom: @3 line 3
>> | | > 11. ZnRequest>>readHeaderFrom: @2 line 2
>> | | > 12. ZnRequest(ZnMessage)>>readBinaryFrom: @2 line 3
>> | | > 13. ZnRequest class(ZnMessage class)>>readBinaryFrom: @3 line 3
>> | | > 14. [] in
>> | | > ExecBlock1(ZnZincServerAdaptor)>>configureServerForBinaryReading
>> | | > @2 line 4
>> | | > 15. [] in
>> | | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
>> | | > @3 line 6
>> | | > 16. [] in
>> | | > ExecBlock0(ZnSingleThreadedServer)>>withMaximumEntitySizeDo: @2
>> | | > line 6
>> | | > 17. [] in ZnMaximumEntitySize class(DynamicVariable
>> | | > class)>>value:during: @3 line 9
>> | | > 18. ZnMaximumEntitySize class(ExecBlock)>>ensure: @2 line 12
>> | | > 19. ZnMaximumEntitySize class(DynamicVariable
>> | | > class)>>value:during:
>> | | > @6 line 10
>> | | > 20.
>> | | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>withMaximumEntitySizeDo:
>> | | > @5 line 5
>> | | > 21.
>> | | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
>> | | > @2 line 6
>> | | > 22. [] in
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
>> | | > @2 line 4
>> | | > 23. ZnManagingMultiThreadedServer(ExecBlock)>>on:do: @3 line 42
>> | | > 24.
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
>> | | > @3 line 5
>> | | > 25.
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeOneRequestResponseOn:
>> | | > @2 line 7
>> | | > 26. [] in
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
>> | | > @2 line 10
>> | | > 27. [] in ZnCurrentServer class(DynamicVariable
>> | | > class)>>value:during: @3 line 9
>> | | > 28. ZnCurrentServer class(ExecBlock)>>ensure: @2 line 12
>> | | > 29. ZnCurrentServer class(DynamicVariable class)>>value:during:
>> | | > @6
>> | | > line 10
>> | | > 30.
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
>> | | > @4 line 8
>> | | > 31. [] in
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
>> | | > @2 line 9
>> | | > 32. ZnManagingMultiThreadedServer(ExecBlock)>>ensure: @2 line 12
>> | | > 33. [] in
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
>> | | > @2 line 10
>> | | > 34. [] in ExecBlock0(ExecBlock)>>ifCurtailed: @2 line 6
>> | | > 35. ExecBlock0(ExecBlock)>>ensure: @2 line 12
>> | | > 36. ZnManagingMultiThreadedServer(ExecBlock)>>ifCurtailed: @3
>> | | > line
>> | | > 8
>> | | > 37. [] in
>> | | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
>> | | > @2 line 11
>> | | > 38. GsProcess>>_start @7 line 16
>> | | > 39. UndefinedObject(GsNMethod class)>>_gsReturnToC @1 line 1
>> | | > 
>> | | > 
>> | | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
>> | | > To: "Johan Brichau" <johan at yesplan.be>
>> | | > Cc: glass at lists.gemtalksystems.com
>> | | > Sent: Wednesday, December 18, 2013 6:26:01 PM
>> | | > Subject: Re: [Glass] problem update: Zinc server with Seaside
>> | | > (3.1)
>> | | > 
>> | | > Johan,
>> | | > 
>> | | > I've finally started monkeying with this and the first thing I
>> | | > ran
>> | | > into (running this under tODE and trying to duplicate the
>> | | > GemTools
>> | | > error condition) was an error involving SocketStream and an
>> | | > apparent buffer overflow ... it looks to me that
>> | | > SocketStream>>next has no logic to refill the buffer once it is
>> | | > full ... at least I got a stack in which the ByteArray buffer was
>> | | > accessed at slot 4097 (1 larger than the buffer):
>> | | > 
>> | | > There are three other processes floating around as well ... I'll
>> | | > keep digging ...
>> | | > 
>> | | > Dale
>> | | > 
>> | | > 
>> | | > From: "Johan Brichau" <johan at yesplan.be>
>> | | > To: glass at lists.gemtalksystems.com
>> | | > Sent: Sunday, December 15, 2013 1:20:20 PM
>> | | > Subject: [Glass] problem update: Zinc server with Seaside (3.1)
>> | | > 
>> | | > Hi all,
>> | | > 
>> | | > I’m working on the Zinc server adaptor for Seaside 3.1 but I got
>> | | > stuck.
>> | | > 
>> | | > Launching the following from Gemtools brings up the server and
>> | | > makes it respond to requests nicely. Trying to do the same from a
>> | | > topaz session though, immediately throws an out-of-mem exception.
>> | | > 
>> | | > WAGsZincAdaptor startOn: 8383
>> | | > 
>> | | > If you break the blocking call in Gemtools and run the line above
>> | | > again, you also run out-of-mem after a couple of times (but only
>> | | > after _also_ connecting to the server from your web browser).
>> | | > First question about this is why that does not happen in gem
>> | | > tools?
>> | | > I tried turning the auto commit off but that did not change
>> | | > anything.
>> | | > 
>> | | > Following advice in [1], I started investigating from gemtools
>> | | > and
>> | | > got this for the byte sizes in temp obj memory on 75% full:
>> | | > 
>> | | > #'ByteArray'->22153440, #'GsMethodDictionary'->3073088,
>> | | > #'GsMethodLookupCache'->2224064, #'GsProcess'->562224,
>> | | > #'String'->415504, #'Array'->340168, #'SocketStream'->322680,
>> | | > #'ExecBlock'->201664, #'VariableContext'->175552,
>> | | > #'SocketStreamSocket'->150696,  …0
>> | | > 
>> | | > The process browser shows over 100 threads displayed as follows:
>> | | > 
>> | | > (priority=25) ready [oop= …. ]
>> | | > 
>> | | > I guess this is where Dale started having nightmares. The stack
>> | | > traces show the out-of-mem always happens in the
>> | | > ZnNetworkingUtils>>setSocketStreamParameters: method.
>> | | > 
>> | | > All of this is using the current Zinc version on github for
>> | | > gemstone 3.1 [2]  and my version of the Seaside 3.1 port [3]
>> | | > To try, load Seaside from my repo, load the ‘Zinc-Seaside’ group
>> | | > from the baseline and hit the above.
>> | | > 
>> | | > Any ideas are welcome
>> | | > 
>> | | > Johan
>> | | > 
>> | | > [1]
>> | | > http://gemstonesoup.wordpress.com/2008/11/19/gemstone-101-managing-out-of-memory-situations/
>> | | > [2] https://github.com/glassdb/zinc/tree/gemstone3.1
>> | | > [3] https://github.com/jbrichau/Seaside31
>> | | > 
>> | | > _______________________________________________
>> | | > Glass mailing list
>> | | > Glass at lists.gemtalksystems.com
>> | | > http://lists.gemtalksystems.com/mailman/listinfo/glass
>> | | 
>> | | 
>> | 
> 