[Glass] problem update: Zinc server with Seaside (3.1)

Thu Dec 19 11:30:52 PST 2013

Johan ... I'm also digging more into this ... actually trying to get the logging api working to dump events to the object log ... I'm suspicious about a couple of things, but nothing showing up ...

Dale

----- Original Message -----
| From: "Johan Brichau" <johan at yesplan.be>
| To: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| Cc: glass at lists.gemtalksystems.com
| Sent: Thursday, December 19, 2013 7:46:37 AM
| Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| 
| Hi Dale,
| 
| Thanks for these pointers.
| I will look into it asap.
| 
| One thing I noticed is that after I merged the latest changes into
| Seaside3.1 [1], a lot less processes were spawning and it took more
| time before the system ran out of memory.
| But it might just have been a coincidence...
| 
| Johan
| 
| [1] https://github.com/glassdb/Seaside31/pull/5
| 
| On 19 Dec 2013, at 06:38, Dale K. Henrichs
| <dale.henrichs at gemtalksystems.com> wrote:
| 
| > Just a touch more info:
| > 
| > ZnMultiThreadedServer>>listenLoop sents #repeat to a block ...
| > 
| > if `serverSocket isValid` returns false, the loop is exited but it
| > launches another
| > #listenLoop ... if we get consistent invalid serverSockets, this
| > guy will spin out of control..
| > 
| > Inside the block, ZnMultiThreadedServer>>serveConnectionsOn: is
| > called (right before the repeat).
| > 
| > In ZnMultiThreadedServer>>serveConnectionsOn: if #waitForAcceptFor:
| > returns a non-nil response, we are off to the races with a forked
| > process and a return to ZnMultiThreadedServer>>listenLoop which
| > itself spins up a new call to
| > ZnMultiThreadedServer>>serveConnectionsOn: ....
| > 
| > Soooo, it looks like SocketStreamSocket>>waitForAcceptFor: had
| > better function correctly .... and serverSocket had better not go
| > into terminal invalidity...
| > 
| > Some halts or breakpoints in and around these particular methods
| > should yield some good information ... or point us in a more
| > productive direction.
| > 
| > Dale
| > 
| > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| > To: "Johan Brichau" <johan at yesplan.be>
| > Cc: glass at lists.gemtalksystems.com
| > Sent: Wednesday, December 18, 2013 8:59:47 PM
| > Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| > 
| > Johan,
| > 
| > I've collected more info on two processes[1] ... first the one that
| > gets an "index out of range error" and the other process that was
| > sitting in the process list waiting for more input from the
| > socket, but the socket was closed (at leasted #isConnected is
| > false).
| > 
| > I hit the out of memory earlier but didn't collect any meaningful
| > info ... I'm assuming that the multi-threaded boy is hitting an
| > error somewhere and spinning away like crazy ...
| > 
| > Need to catch the error in progress and perhaps find out from Sven
| > if there is supposed to be a "deadman switch" on this guy to avoid
| > infinite errors ..
| > 
| > Dale
| > 
| > [1]
| > https://github.com/dalehenrich/todeHome/blob/master/home/seaside3.1.0/stacks
| > 
| > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| > To: "Johan Brichau" <johan at yesplan.be>
| > Cc: glass at lists.gemtalksystems.com
| > Sent: Wednesday, December 18, 2013 6:39:13 PM
| > Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| > 
| > Here's the stack:
| > 
| > 1. OffsetError(AbstractException)>>_signalWith: @5 line 25
| > 2. OffsetError(AbstractException)>>signal @2 line 47
| > 3. ByteArray(Object)>>_error:args: @15 line 11
| > 4. ByteArray(Object)>>_errorIndexOutOfRange: @2 line 6
| > 5. ByteArray>>at: @4 line 13
| > 6. SocketStream>>next @13 line 10
| > 7. ZnLineReader>>processNext @5 line 4
| > 8. ZnLineReader>>nextLine @3 line 3
| > 9. ZnRequestLine>>readFrom: @3 line 3
| > 10. ZnRequestLine class>>readFrom: @3 line 3
| > 11. ZnRequest>>readHeaderFrom: @2 line 2
| > 12. ZnRequest(ZnMessage)>>readBinaryFrom: @2 line 3
| > 13. ZnRequest class(ZnMessage class)>>readBinaryFrom: @3 line 3
| > 14. [] in
| > ExecBlock1(ZnZincServerAdaptor)>>configureServerForBinaryReading
| > @2 line 4
| > 15. [] in
| > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
| > @3 line 6
| > 16. [] in
| > ExecBlock0(ZnSingleThreadedServer)>>withMaximumEntitySizeDo: @2
| > line 6
| > 17. [] in ZnMaximumEntitySize class(DynamicVariable
| > class)>>value:during: @3 line 9
| > 18. ZnMaximumEntitySize class(ExecBlock)>>ensure: @2 line 12
| > 19. ZnMaximumEntitySize class(DynamicVariable class)>>value:during:
| > @6 line 10
| > 20.
| > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>withMaximumEntitySizeDo:
| > @5 line 5
| > 21.
| > ZnManagingMultiThreadedServer(ZnSingleThreadedServer)>>readRequest:
| > @2 line 6
| > 22. [] in
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
| > @2 line 4
| > 23. ZnManagingMultiThreadedServer(ExecBlock)>>on:do: @3 line 42
| > 24.
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>readRequestSafely:
| > @3 line 5
| > 25.
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeOneRequestResponseOn:
| > @2 line 7
| > 26. [] in
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
| > @2 line 10
| > 27. [] in ZnCurrentServer class(DynamicVariable
| > class)>>value:during: @3 line 9
| > 28. ZnCurrentServer class(ExecBlock)>>ensure: @2 line 12
| > 29. ZnCurrentServer class(DynamicVariable class)>>value:during: @6
| > line 10
| > 30.
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>executeRequestResponseLoopOn:
| > @4 line 8
| > 31. [] in
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| > @2 line 9
| > 32. ZnManagingMultiThreadedServer(ExecBlock)>>ensure: @2 line 12
| > 33. [] in
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| > @2 line 10
| > 34. [] in ExecBlock0(ExecBlock)>>ifCurtailed: @2 line 6
| > 35. ExecBlock0(ExecBlock)>>ensure: @2 line 12
| > 36. ZnManagingMultiThreadedServer(ExecBlock)>>ifCurtailed: @3 line
| > 8
| > 37. [] in
| > ZnManagingMultiThreadedServer(ZnMultiThreadedServer)>>serveConnectionsOn:
| > @2 line 11
| > 38. GsProcess>>_start @7 line 16
| > 39. UndefinedObject(GsNMethod class)>>_gsReturnToC @1 line 1
| > 
| > 
| > From: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| > To: "Johan Brichau" <johan at yesplan.be>
| > Cc: glass at lists.gemtalksystems.com
| > Sent: Wednesday, December 18, 2013 6:26:01 PM
| > Subject: Re: [Glass] problem update: Zinc server with Seaside (3.1)
| > 
| > Johan,
| > 
| > I've finally started monkeying with this and the first thing I ran
| > into (running this under tODE and trying to duplicate the GemTools
| > error condition) was an error involving SocketStream and an
| > apparent buffer overflow ... it looks to me that
| > SocketStream>>next has no logic to refill the buffer once it is
| > full ... at least I got a stack in which the ByteArray buffer was
| > accessed at slot 4097 (1 larger than the buffer):
| > 
| > There are three other processes floating around as well ... I'll
| > keep digging ...
| > 
| > Dale
| > 
| > 
| > From: "Johan Brichau" <johan at yesplan.be>
| > To: glass at lists.gemtalksystems.com
| > Sent: Sunday, December 15, 2013 1:20:20 PM
| > Subject: [Glass] problem update: Zinc server with Seaside (3.1)
| > 
| > Hi all,
| > 
| > I’m working on the Zinc server adaptor for Seaside 3.1 but I got
| > stuck.
| > 
| > Launching the following from Gemtools brings up the server and
| > makes it respond to requests nicely. Trying to do the same from a
| > topaz session though, immediately throws an out-of-mem exception.
| > 
| > WAGsZincAdaptor startOn: 8383
| > 
| > If you break the blocking call in Gemtools and run the line above
| > again, you also run out-of-mem after a couple of times (but only
| > after _also_ connecting to the server from your web browser).
| > First question about this is why that does not happen in gem tools?
| > I tried turning the auto commit off but that did not change
| > anything.
| > 
| > Following advice in [1], I started investigating from gemtools and
| > got this for the byte sizes in temp obj memory on 75% full:
| > 
| > #'ByteArray'->22153440, #'GsMethodDictionary'->3073088,
| > #'GsMethodLookupCache'->2224064, #'GsProcess'->562224,
| > #'String'->415504, #'Array'->340168, #'SocketStream'->322680,
| > #'ExecBlock'->201664, #'VariableContext'->175552,
| > #'SocketStreamSocket'->150696,  …0
| > 
| > The process browser shows over 100 threads displayed as follows:
| > 
| > (priority=25) ready [oop= …. ]
| > 
| > I guess this is where Dale started having nightmares. The stack
| > traces show the out-of-mem always happens in the
| > ZnNetworkingUtils>>setSocketStreamParameters: method.
| > 
| > All of this is using the current Zinc version on github for
| > gemstone 3.1 [2]  and my version of the Seaside 3.1 port [3]
| > To try, load Seaside from my repo, load the ‘Zinc-Seaside’ group
| > from the baseline and hit the above.
| > 
| > Any ideas are welcome
| > 
| > Johan
| > 
| > [1]
| > http://gemstonesoup.wordpress.com/2008/11/19/gemstone-101-managing-out-of-memory-situations/
| > [2] https://github.com/glassdb/zinc/tree/gemstone3.1
| > [3] https://github.com/jbrichau/Seaside31
| > 
| > _______________________________________________
| > Glass mailing list
| > Glass at lists.gemtalksystems.com
| > http://lists.gemtalksystems.com/mailman/listinfo/glass
| 
| 