[Glass] Monit scripts for gemstone?

[Johan Brichau via Glass]

Hi Dale,

I’m a suspicious about the ’10 concurrent threads’ per gem.

I know this is what the code is supposed to do, but I always encounter a deadlock when a (seaside) request in process sends a request that is to be treated by the same seaside gem, but for a completely different application (and thus certainly not the same seaside session). I would expect that the gem accepts the request and is able to process it because 10 concurrent requests can be accepted, nevertheless I always get a timeout.

On the other hand, if you remember, we did hit and fix a bug with the semaphore back at ESUG in Edinburgh that clearly indicated that another request _can_ be processed while another one is sitting idle. However, there does not seem to be a guarantee in my experience. What I suspect (but you can probably tell me more :) is that the request in process not always yields to a process that would accept the incoming request.

Just yesterday I got at such a situation because I am fine-tuning the automated tests of our application. In that scenario, there is a single Swazoo (or Zinc) server running and a request in process is making a (Zinc) http call to the localhost for a different Seaside application (but thus served by the same gem). In a real usecase, this call is done in the service vm but for testing purposes this http call is made synchronously. However, I get a timeout for this second call. I hit this both in Gemstone 2.4 and 3.1

Johan

> On 19 Feb 2015, at 23:16, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> 
> 
> On 2/19/15 1:36 PM, Mariano Martinez Peck wrote:
>> 
>> 
>> On Thu, Feb 19, 2015 at 5:05 PM, Johan Brichau <johan at yesplan.be <mailto:johan at yesplan.be>> wrote:
>> If you want to allow longer requests, then make sure to set the timeout longer. 
>> GLASS is made to process only a single seaside request at a time and thus a request will get blocked until the previous one was processed. That includes the monit request.
>> So, what you say is correct: if a gem is busy processing requests, it will not answer to monit.
>> 
>> 
>> mmmmm are we 100% sure about this? I mean...at which level is the "lock" inside the gem to block requests? I thought it was at kind of at seaside session level. If monit request is a plain fast cgi request (outside seaside context) maybe the gem CAN answer because the lock happens further?
>> Let me CC Dale just in case ;) 
>> 
> Depending upon how deep the monit request goes into the request handling stack for FastCGI, yes.
> 
> There is a session lock that is applied once we've processed enough of the request to know what session we're dealing with. I think if the session is already locked we kick out and retry the http request (don't recall exactly).
> 
> Then down the stack a ways in GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock: we acquire a transaction mutex, and this is where we ensure that only one request is processed at a time (inside the transaction). If monit makes it this far, then it will wait for the transaction mutex to be released ...
> 
> For FastCGI, there is a gateway semaphore that only lets 10 concurrent threads to be forked for queuing up on the transaction mutex ... I found in testing that under high enough loads, the gem would blow up with an out of memory that was caused by too many threads sitting idle waiting for the transaction mutex ...
> 
> So monit could be blocking here, if there a number of outstanding requests ... of course there isn't much difference between waiting on the gateway semaphore and blocking on the transaction mutex ...
> 
> Dale

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150221/b7c352c3/attachment-0001.html>