[Glass] How to deal with timeouts, fastcgi and what is the expected behavior?

Fri Jul 18 08:54:37 PDT 2014

On Fri, Jul 18, 2014 at 12:08 PM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:

>
>
>
> On Fri, Jul 18, 2014 at 7:36 AM, Mariano Martinez Peck <
> marianopeck at gmail.com> wrote:
>
>> Hi guys,
>>
>> I have a strange situation with timeouts under nginx/fastCGI and I am not
>> sure what is expected. I am executing a Seaside WATask, where one of its
>> method takes (for sure) more then the timeout of nginx/fastCGI. My piece of
>> nginx setup is something like this:
>>
>> location @seasidemariano {
>>  include fastcgi_params;
>> fastcgi_param REQUEST_URI $uri?$args;
>>   fastcgi_pass seasidemariano;
>>  fastcgi_connect_timeout      180;
>>       fastcgi_send_timeout         180;
>>       fastcgi_read_timeout         180;
>>
>>   fastcgi_next_upstream error invalid_header timeout http_500;
>> }
>>
>>
>> So...as you can see I have a timeout of 180 and I tell to go to the next
>> upstream (gem) in any error, including timeout. Now...say I have this
>> method being executed and it takes more than 180 seconds. What happens is
>> that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But...
>> I have some questions:
>>
>> 1) what happens with that gem that was executing the task (the one that
>> took more than 180)? is the execution finished even if the nginx give a
>> timeout and pass the reuqest to the next gem? Or the gem execution is
>> aborted?  Why I ask? Because...I out a log to a file inside my method...and
>> it looks like if the method were called 3 times rather than 1. And from a
>> domain point of view.... it is not good that such a method is executed 3
>> times...
>>
>
> It does sound like nginx is redispatching the http request on timeout...
>
>>
>>
Exactly. And it should, as my configuration is:

fastcgi_next_upstream error invalid_header *timeout* http_500;

So yes...upon a gem timeout, nginx forwards the request to the next gem.

> 2) If I put a larger timeout...say 1500 ... it works correct..the method
>> is executed only once, no timeout. Same if I use swazoo. So it is something
>> to do with the timeouts and fastCGI for sure.
>>
>
> In general I try to avoid timeouts ... it seems that timeouts fire more
> often because the system is slow than for any other reason and the standard
> answer: increase the timeout ...
>
> So I guess I would wonder why the operation is taking so long ... if the
> operation is slow because the system is overloaded, then a longer timeout
> _is_ called for, but then what is a good value for the timeout ..
>
>
The operation takes long because I need to call a HTTPS api (using Zinc to
a local nginx tunnel) many times where I need to post a XML and I also get
a large XML response. The time it takes..depends on how many "items" have
been selected. So it is hard to estimate how much it would take.

> I guess the real question to ask is what is the purpose of the timeout?
>
>
If a gem went down, I would like nginx to forward request to the other
(available gems).

> You might want the gem itself to decide to terminate a request if it is
> "taking too long" then you don't need  a timeout at the nginx level?
>
>
>> 3) 3 times...why? It seems because I have 3 gems. I did an experiment,
>> and I set only 2 gems to nginx fastcgi. And yes, the method was executed
>> only 2 times rather than 3.
>>
>
> It does sound like nginx is sending the request again upon a timeout ...
> could that be?
>

Yes, it is that. But I don't know how to properly solve both things... be
able to have large timeout (like for this scenario), yet manage the
scenario of gems going down. Imagine I don't care and I put a timeout of 1
hour (to say something). Then.. imagine I have a gem down and then a web
user that connects to the site.  nginx might assign that gem that gem that
went down. Therefore... it would have the web user waiting in the browser
for 1 hour until nginx answers for a timeout....Is this correct?

The biggest issue is that nginx thinks the gem timeouted and then fordwads
to the next gem. However...the gem was not dead...it was simply too busy
with a time consuming request  ;) There is no way I can make the gem answer
to nginx "I am fine, don't worry, just busy, continue to next gem" hahaha ?

Probably the real real solution is the service VM as you and Paul pointed
out several times. But i didn't have time to take a look to it yet :(

>
>>
>> So....how do people normally deal with this? Of course, the immediate
>> workaround seems to increase the timeout...but it seems risky to me,
>> thinking that if for some reason (like GC running or whatever) one
>> particular request takes more than the timeout, then my "backend code"
>> could be run more than once...
>>
>> Thanks in advance,
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>

-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140718/02a72549/attachment-0001.html>