[Glass] Monit scripts for gemstone?

[Dale Henrichs via Glass]

Sorry for the late response, I've been sick the last week or so and I'm 
just starting to dig out of my email hole:)


On 2/21/15 6:40 AM, Johan Brichau wrote:
> Hi Dale,
>
> I’m a suspicious about the ’10 concurrent threads’ per gem.
>
> I know this is what the code is supposed to do, but I always encounter 
> a deadlock when a (seaside) request in process sends a request that is 
> to be treated by the same seaside gem, but for a completely different 
> application (and thus certainly not the same seaside session). I would 
> expect that the gem accepts the request and is able to process it 
> because 10 concurrent requests can be accepted, nevertheless I always 
> get a timeout.
The 10 concurrent threads is a function of how many accept() calls are 
processed (gateway semaphore)... the gem is still only going to process 
1 request at a time (transaction mutex) so I guess I'd like to see some 
stacks with these errors ... is the timeout happening because of an 
accept() timeout or is the timout happening because the overall 
processing time has taken too lon?
>
> On the other hand, if you remember, we did hit and fix a bug with the 
> semaphore back at ESUG in Edinburgh that clearly indicated that 
> another request _can_ be processed while another one is sitting idle. 
> However, there does not seem to be a guarantee in my experience. What 
> I suspect (but you can probably tell me more :) is that the request in 
> process not always yields to a process that would accept the incoming 
> request.
I don't remember those details ... do you have a reference to the code 
we fixed?
>
> Just yesterday I got at such a situation because I am fine-tuning the 
> automated tests of our application. In that scenario, there is a 
> single Swazoo (or Zinc) server running and a request in process is 
> making a (Zinc) http call to the localhost for a different Seaside 
> application (but thus served by the same gem). In a real usecase, this 
> call is done in the service vm but for testing purposes this http call 
> is made synchronously. However, I get a timeout for this second call. 
> I hit this both in Gemstone 2.4 and 3.1
Okay ... you are being deadlocked by the transaction mutex ... with the 
transaction mutex there will only ever be on request being processes at 
any one time ... so with a single gem and making a client http call from 
within the single active seaside request (with the transaction mutex) 
you are guaranteed to get deadlock ...
>
> Johan
>
>> On 19 Feb 2015, at 23:16, Dale Henrichs 
>> <dale.henrichs at gemtalksystems.com 
>> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>
>>
>> On 2/19/15 1:36 PM, Mariano Martinez Peck wrote:
>>>
>>>
>>> On Thu, Feb 19, 2015 at 5:05 PM, Johan Brichau <johan at yesplan.be 
>>> <mailto:johan at yesplan.be>> wrote:
>>>
>>>     If you want to allow longer requests, then make sure to set the
>>>     timeout longer.
>>>     GLASS is made to process only a single seaside request at a time
>>>     and thus a request will get blocked until the previous one was
>>>     processed. That includes the monit request.
>>>     So, what you say is correct: if a gem is busy processing
>>>     requests, it will not answer to monit.
>>>
>>>
>>> mmmmm are we 100% sure about this? I mean...at which level is the 
>>> "lock" inside the gem to block requests? I thought it was at kind of 
>>> at seaside session level. If monit request is a plain fast cgi 
>>> request (outside seaside context) maybe the gem CAN answer because 
>>> the lock happens further?
>>> Let me CC Dale just in case ;)
>>>
>> Depending upon how deep the monit request goes into the request 
>> handling stack for FastCGI, yes.
>>
>> There is a session lock that is applied once we've processed enough 
>> of the request to know what session we're dealing with. I think if 
>> the session is already locked we kick out and retry the http request 
>> (don't recall exactly).
>>
>> Then down the stack a ways in 
>> GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock: we 
>> acquire a transaction mutex, and this is where we ensure that only 
>> one request is processed at a time (inside the transaction). If monit 
>> makes it this far, then it will wait for the transaction mutex to be 
>> released ...
>>
>> For FastCGI, there is a gateway semaphore that only lets 10 
>> concurrent threads to be forked for queuing up on the transaction 
>> mutex ... I found in testing that under high enough loads, the gem 
>> would blow up with an out of memory that was caused by too many 
>> threads sitting idle waiting for the transaction mutex ...
>>
>> So monit could be blocking here, if there a number of outstanding 
>> requests ... of course there isn't much difference between waiting on 
>> the gateway semaphore and blocking on the transaction mutex ...
>>
>> Dale
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150226/306c5b1a/attachment-0001.html>