[Glass] problem update: Zinc server with Seaside (3.1)

Thu Dec 19 19:31:15 PST 2013

Johan,

I moved the package SocketStream to the repo (changed baseline) and commit on socketstream_fix ... let me know if this works for you..

Dale

----- Original Message -----
| From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| To: "Johan Brichau" <johan at yesplan.be>
| Cc: glass at lists.gemtalksystems.com
| Sent: Thursday, December 19, 2013 5:45:26 PM
| Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| 
| I think I've got things working now ... I will push a branch of
| glassdb/zinc with my changes ... I need to clean things up a bit and
| test with topaz ... right now its running in tODE ... I'm still
| getting two connections per request, but it's much more stable now
| ...
| 
| Dale
| 
| ----- Original Message -----
| | From: "Johan Brichau" <johan at yesplan.be>
| | To: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| | Cc: glass at lists.gemtalksystems.com
| | Sent: Thursday, December 19, 2013 7:46:37 AM
| | Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| | 
| | Hi Dale,
| | 
| | Thanks for these pointers.
| | I will look into it asap.
| | 
| | One thing I noticed is that after I merged the latest changes into
| | Seaside3.1 [1], a lot less processes were spawning and it took more
| | time before the system ran out of memory.
| | But it might just have been a coincidence...
| | 
| | Johan
| | 
| | [1] https://github.com/glassdb/Seaside31/pull/5
| | 
| | On 19 Dec 2013, at 06:38, Dale K. Henrichs
| | <dale.henrichs at gemtalksystems.com> wrote:
| | 
| | > Just a touch more info:
| | > 
| | > ZnMultiThreadedServer>>listenLoop sents #repeat to a block ...
| | > 
| | > if `serverSocket isValid` returns false, the loop is exited but
| | > it
| | > launches another
| | > #listenLoop ... if we get consistent invalid serverSockets, this
| | > guy will spin out of control..
| | > 
| | > Inside the block, ZnMultiThreadedServer>>serveConnectionsOn: is
| | > called (right before the repeat).
| | > 
| | > In ZnMultiThreadedServer>>serveConnectionsOn: if
| | > #waitForAcceptFor:
| | > returns a non-nil response, we are off to the races with a forked
| | > process and a return to ZnMultiThreadedServer>>listenLoop which
| | > itself spins up a new call to
| | > ZnMultiThreadedServer>>serveConnectionsOn: ....
| | > 
| | > Soooo, it looks like SocketStreamSocket>>waitForAcceptFor: had
| | > better function correctly .... and serverSocket had better not go
| | > into terminal invalidity...
| | > 
| | > Some halts or breakpoints in and around these particular methods
| | > should yield some good information ... or point us in a more
| | > productive direction.
| | > 
| | > Dale
| | > 
| | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| | > To: "Johan Brichau" <johan at yesplan.be>
| | > Cc: glass at lists.gemtalksystems.com
| | > Sent: Wednesday, December 18, 2013 8:59:47 PM
| | > Subject: Re: [Glass] problem update: Zinc server with Seaside
| | > (3.1)
| | > 
| | > Johan,
| | > 
| | > I've collected more info on two processes[1] ... first the one
| | > that
| | > gets an "index out of range error" and the other process that was
| | > sitting in the process list waiting for more input from the
| | > socket, but the socket was closed (at leasted #isConnected is
| | > false).
| | > 
| | > I hit the out of memory earlier but didn't collect any meaningful
| | > info ... I'm assuming that the multi-threaded boy is hitting an
| | > error somewhere and spinning away like crazy ...
| | > 
| | > Need to catch the error in progress and perhaps find out from
| | > Sven
| | > if there is supposed to be a "deadman switch" on this guy to
| | > avoid
| | > infinite errors ..
| | > 
| | > Dale
| | > 
| | > [1]
| | > https://github.com/dalehenrich/todeHome/blob/master/home/seaside3.1.0/stacks
| | > 
| | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| | > To: "Johan Brichau" <johan at yesplan.be>
| | > Cc: glass at lists.gemtalksystems.com
| | > Sent: Wednesday, December 18, 2013 6:39:13 PM
| | > Subject: Re: [Glass] problem update: Zinc server with Seaside
| | > (3.1)
| | > 
| | > Here's the stack:
| | > 
| | > 1. OffsetError(AbstractException)>>_signalWith: @5 line 25
| | > 2. OffsetError(AbstractException)>>signal @2 line 47
| | > 3. ByteArray(Object)>>_error:args: @15 line 11
| | > 4. ByteArray(Object)>>_errorIndexOutOfRange: @2 line 6
| | > 5. ByteArray>>at: @4 line 13
| | > 6. SocketStream>>next @13 line 10
| | > 7. ZnLineReader>>processNext @5 line 4
| | > 8. ZnLineReader>>nextLine @3 line 3
| | > 9. ZnRequestLine>>readFrom: @3 line 3
| | > 10. ZnRequestLine class>>readFrom: @3 line 3
| | > 11. ZnRequest>>readHeaderFrom: @2 line 2
| | > 12. ZnRequest(ZnMessage)>>readBinaryFrom: @2 line 3
| | > 13. ZnRequest class(ZnMessage class)>>readBinaryFrom: @3 line 3
| | > 14. [] in
| | > ExecBlock1(ZnZincServerAdaptor)>>configureServerForBinaryReading
| | > @2 line 4
| | > 15. [] in
| | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
| | > @3 line 6
| | > 16. [] in
| | > ExecBlock0(ZnSingleThreadedServer)>>withMaximumEntitySizeDo: @2
| | > line 6
| | > 17. [] in ZnMaximumEntitySize class(DynamicVariable
| | > class)>>value:during: @3 line 9
| | > 18. ZnMaximumEntitySize class(ExecBlock)>>ensure: @2 line 12
| | > 19. ZnMaximumEntitySize class(DynamicVariable
| | > class)>>value:during:
| | > @6 line 10
| | > 20.
| | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>withMaximumEntitySizeDo:
| | > @5 line 5
| | > 21.
| | > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
| | > @2 line 6
| | > 22. [] in
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
| | > @2 line 4
| | > 23. ZnManagingMultiThreadedServer(ExecBlock)>>on:do: @3 line 42
| | > 24.
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
| | > @3 line 5
| | > 25.
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeOneRequestResponseOn:
| | > @2 line 7
| | > 26. [] in
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
| | > @2 line 10
| | > 27. [] in ZnCurrentServer class(DynamicVariable
| | > class)>>value:during: @3 line 9
| | > 28. ZnCurrentServer class(ExecBlock)>>ensure: @2 line 12
| | > 29. ZnCurrentServer class(DynamicVariable class)>>value:during:
| | > @6
| | > line 10
| | > 30.
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
| | > @4 line 8
| | > 31. [] in
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| | > @2 line 9
| | > 32. ZnManagingMultiThreadedServer(ExecBlock)>>ensure: @2 line 12
| | > 33. [] in
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| | > @2 line 10
| | > 34. [] in ExecBlock0(ExecBlock)>>ifCurtailed: @2 line 6
| | > 35. ExecBlock0(ExecBlock)>>ensure: @2 line 12
| | > 36. ZnManagingMultiThreadedServer(ExecBlock)>>ifCurtailed: @3
| | > line
| | > 8
| | > 37. [] in
| | > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| | > @2 line 11
| | > 38. GsProcess>>_start @7 line 16
| | > 39. UndefinedObject(GsNMethod class)>>_gsReturnToC @1 line 1
| | > 
| | > 
| | > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| | > To: "Johan Brichau" <johan at yesplan.be>
| | > Cc: glass at lists.gemtalksystems.com
| | > Sent: Wednesday, December 18, 2013 6:26:01 PM
| | > Subject: Re: [Glass] problem update: Zinc server with Seaside
| | > (3.1)
| | > 
| | > Johan,
| | > 
| | > I've finally started monkeying with this and the first thing I
| | > ran
| | > into (running this under tODE and trying to duplicate the
| | > GemTools
| | > error condition) was an error involving SocketStream and an
| | > apparent buffer overflow ... it looks to me that
| | > SocketStream>>next has no logic to refill the buffer once it is
| | > full ... at least I got a stack in which the ByteArray buffer was
| | > accessed at slot 4097 (1 larger than the buffer):
| | > 
| | > There are three other processes floating around as well ... I'll
| | > keep digging ...
| | > 
| | > Dale
| | > 
| | > 
| | > From: "Johan Brichau" <johan at yesplan.be>
| | > To: glass at lists.gemtalksystems.com
| | > Sent: Sunday, December 15, 2013 1:20:20 PM
| | > Subject: [Glass] problem update: Zinc server with Seaside (3.1)
| | > 
| | > Hi all,
| | > 
| | > I’m working on the Zinc server adaptor for Seaside 3.1 but I got
| | > stuck.
| | > 
| | > Launching the following from Gemtools brings up the server and
| | > makes it respond to requests nicely. Trying to do the same from a
| | > topaz session though, immediately throws an out-of-mem exception.
| | > 
| | > WAGsZincAdaptor startOn: 8383
| | > 
| | > If you break the blocking call in Gemtools and run the line above
| | > again, you also run out-of-mem after a couple of times (but only
| | > after _also_ connecting to the server from your web browser).
| | > First question about this is why that does not happen in gem
| | > tools?
| | > I tried turning the auto commit off but that did not change
| | > anything.
| | > 
| | > Following advice in [1], I started investigating from gemtools
| | > and
| | > got this for the byte sizes in temp obj memory on 75% full:
| | > 
| | > #'ByteArray'->22153440, #'GsMethodDictionary'->3073088,
| | > #'GsMethodLookupCache'->2224064, #'GsProcess'->562224,
| | > #'String'->415504, #'Array'->340168, #'SocketStream'->322680,
| | > #'ExecBlock'->201664, #'VariableContext'->175552,
| | > #'SocketStreamSocket'->150696,  …0
| | > 
| | > The process browser shows over 100 threads displayed as follows:
| | > 
| | > (priority=25) ready [oop= …. ]
| | > 
| | > I guess this is where Dale started having nightmares. The stack
| | > traces show the out-of-mem always happens in the
| | > ZnNetworkingUtils>>setSocketStreamParameters: method.
| | > 
| | > All of this is using the current Zinc version on github for
| | > gemstone 3.1 [2]  and my version of the Seaside 3.1 port [3]
| | > To try, load Seaside from my repo, load the ‘Zinc-Seaside’ group
| | > from the baseline and hit the above.
| | > 
| | > Any ideas are welcome
| | > 
| | > Johan
| | > 
| | > [1]
| | > http://gemstonesoup.wordpress.com/2008/11/19/gemstone-101-managing-out-of-memory-situations/
| | > [2] https://github.com/glassdb/zinc/tree/gemstone3.1
| | > [3] https://github.com/jbrichau/Seaside31
| | > 
| | > _______________________________________________
| | > Glass mailing list
| | > Glass at lists.gemtalksystems.com
| | > http://lists.gemtalksystems.com/mailman/listinfo/glass
| | 
| | 
| 