[Glass] How to remote debug exceptions caused by seaside rendering?
Dale Henrichs via Glass
glass at lists.gemtalksystems.com
Mon Apr 27 11:51:00 PDT 2015
Mariano,
I have submitted the bug and recorded the stack ... when I get a chance
I will dig into this mystery ... it really appears that the debug path
(#debug) is being skipped when debugging the continuation and it's not
clear why ... there is a bit of code involved so it's not
straightforward to resolve...
Looking again at the code, the following code snippet from
WARemoteDebuggingWalkbackErrorHandler>>open: might be causing some trouble:
self requestContext request isXmlHttpRequest
ifTrue: [
wb addContinuation.
(self requestContext responseGenerator internalError:
anException)
respond ]
if the internalError: code doesn't behave ...
Dale
On 04/27/2015 08:08 AM, Mariano Martinez Peck wrote:
>
>
> On Fri, Apr 24, 2015 at 8:49 PM, Dale Henrichs
> <dale.henrichs at gemtalksystems.com
> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>
> Well, in theory the following should work in GemTools:
>
> OTDebugger openProcess: (Object _objectForOop: processOOp)
>
> but I have to confess I haven't played with it a lot in GemTools.
> I just tried the following and it does not work:
>
> OTDebugger openProcess: ([(Delay forSeconds: 30) wait] fork)
>
>
> Ok, i does not work either here.
>
> I think that the problem stems from the fact that the opening of
> the debugger must be initiated from the client side of the gci in
> GemTools, because ..... that's just the way it's done ...
>
>
> Ok...
>
> Like I said you can `attach` or `debug` a live process in tODE
> (moral equivalent of the above), but then I rewrote the debugger
> from scratch for tODE and made the effort to get it working ...
> like when you attach to a continuation in tODE I just attach the
> debugger to the process without sending value:
>
> attachContinuation: objectLogEntry
> | process |
> process := objectLogEntry continuation.
> (TDDebugger new
> topez: topez;
> attachProcess: process;
> windowLabel: objectLogEntry labelString , ' @ ' ,
> objectLogEntry stampString;
> yourself) open
>
> whereas to debug a continuation you do the following (same as in
> GemTools):
>
> debugContinuation: objectLogEntry
> objectLogEntry continuation value: #'debug'
>
> and of course the above is what is getting you into trouble in
> GemStools.
>
>
> Ok, that's nice to know.
>
> Speaking of getting into trouble, I just read the code in
> WAGemStoneWalkbackErrorHandler>>open: and noticed at the very top
> these lines:
>
> open: anException
> | answer |
> self requestContext request isXmlHttpRequest
> ifTrue: [ ^ super open: anException ].
>
> and later on in this method there's code that does this:
>
> self session isNil
> ifFalse: [
> answer := self session presenter
>
> Which is presumably where you get into trouble ... I'm going to
> guess that for correctly debugging ajax continuations the request
> needs to answer true to isXmlHttpRequest
>
>
> I did some debugging and it seems by ajax requests are indeed
> answering true to such a message :(
>
> Thanks anyway.
>
> ...
>
> Dale
>
> On 04/24/2015 12:04 PM, Mariano Martinez Peck wrote:
>>
>>
>> On Fri, Apr 24, 2015 at 3:31 PM, Dale Henrichs
>> <dale.henrichs at gemtalksystems.com
>> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>
>> Glad to help ... ... the stack in frame 97 was an argument or
>> temp of frame97, so the process that the debugger opened on
>> was not the process that you wanted to debug from the object
>> log and you wouldn't have seen that stack in the debugger no
>> matter how many frames you were looking at at once ...
>>
>>
>> AHHHHH now I got it. Thanks Dale. I finally got that this was:
>>
>> [97] TransientRecursionLock >> critical: (envId 0)
>> aBlock: anExecBlock
>> proc: GsProcess(oop=*839224065*, status=debug, priority=4,
>> WARemoteDebuggingWalkbackErrorHandler >> open: (envId 0) @10 line
>> 1....
>>
>> Whatever is going wrong is related to the fact that the
>> process that was supposed to be debugged was not the one the
>> debugger was opened on and that's the initial focus for
>> solving this puzzle ...
>>
>> I've submitted a bug[1] to track this ... I do want to get to
>> the bottom of this, but I've got other fish to fry at the
>> moment:)
>>
>>
>> Thanks. Let me ask.... if I have the OOP of the GsProcess
>> (oop=*839224065), *then I can get the GsProcess instance.
>> Soo...until the issue is solved, maybe there is a way I can still
>> debug a live stack? Cannot I open a debugger over a certain
>> GsProcess instance?
>>
>>
>> Dale
>>
>> [1] https://github.com/GsDevKit/Seaside31/issues/73
>>
>>
>> On 04/24/2015 10:25 AM, Mariano Martinez Peck wrote:
>>>
>>>
>>> On Fri, Apr 24, 2015 at 2:01 PM, Dale Henrichs
>>> <dale.henrichs at gemtalksystems.com
>>> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>
>>> In frame 97 I came across this printString for a
>>> GsProcess and it looks like this guy might be the
>>> "droid" you are looking for:).
>>>
>>>
>>> Thanks Dale for the find. I was wondering why I didn't see
>>> it myself.. and the answer is that the debugger opened only
>>> shows a part of the stack. To attach you the stack I clicked
>>> on "copy stack". I thought it was the same I was seeing in
>>> the debugger..but it isn't. It's the real full stack.
>>> So at least I have the dead text stack to analyze. WTF that
>>> the debugger doesn't have a "full stack" button so that at
>>> least I can debug a live stack. I know I know ... i should
>>> use tODE :)
>>>
>>> From the looks of things it looks like the "normal"
>>> seaside object log continuation logic is being used, so
>>> I will have to dig deeper to understand what's happening
>>> ... if the following stack gets you going again, I might
>>> delay looking into this problem a bit, because I'm a bit
>>> buried at the moment ...
>>>
>>>
>>> Thanks Dale, the stack below does help me to solve this
>>> particular issue.
>>>
>>> Dale
>>>
>>> [97] TransientRecursionLock >> critical: (envId 0)
>>> aBlock: anExecBlock
>>> proc: GsProcess(oop=16012850177, status=debug,
>>> priority=4,
>>> WARemoteDebuggingWalkbackErrorHandler >> open: (envId 0)
>>> @10 line 10
>>> WAWalkbackErrorHandler >> handleDefault: (envId 0) @2 line 2
>>> WAErrorHandler >> handleError: (envId 0) @2 line 2
>>> WAErrorHandler >> handleGemStoneException: (envId 0) @4
>>> line 4
>>> WAGemStoneWalkbackErrorHandler >> handleException:
>>> (envId 0) @2 line 2
>>> [] in WAExceptionHandler >> handleExceptionsDuring:
>>> (envId 0) @2 line 5
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> [] in WAExceptionHandler >> handleExceptionsDuring:
>>> (envId 0) @2 line 8
>>> [] in ExecBlock >> on:do: (envId 0) @4 line 49
>>> AbstractException >> _executeHandler: (envId 0) @3 line 8
>>> AbstractException >> _signalWith: (envId 0) @1 line 1
>>> AbstractException >> signal (envId 0) @2 line 47
>>> Object >> doesNotUnderstand: (envId 0) @9 line 10
>>> Object >> _doesNotUnderstand:args:envId:reason: (envId
>>> 0) @7 line 12
>>> FaQuandlSF1StocksFundamentalsDBPopulator >>
>>> fillExtraInfoFromSeriesToSecurity: (envId 0) @16 line 12
>>> FaQuandlDirectDownloadDataAccessor >>
>>> getSF1SecurityForTicker:date: (envId 0) @11 line 23
>>> FaQuandl >> fill:withSecurityInfoWith:date: (envId 0) @7
>>> line 5
>>> FaQuandl >>
>>> newQuandlDataEntityByTickerSymbol:viewSampleDates:andAccessorSpec:
>>> (envId 0) @17 line 13
>>> FaQuandl >>
>>> newQuandlDataEntityByTickerSymbol:viewSampleDates:
>>> (envId 0) @2 line 2
>>> FaQuandl >>
>>> getQuandlDataProcessorProxyAt:withSampleDates:withAnalystSpec:
>>> (envId 0) @9 line 9
>>> FaQuandl >>
>>> getQuandlDataProcessorProxyOnAnnualDataAt:withSampleDates:
>>> (envId 0) @2 line 3
>>> FaFnDataResourceRetriever >> dataSource:at: (envId 0)
>>> @17 line 15
>>> [] in FaFnDataResourceRetriever >>
>>> processorCreateFor:sourceSpec:andAnalystConfigSpec:
>>> (envId 0) @5 line 19
>>> SequenceableCollection >> collect: (envId 0) @9 line 16
>>> FaFnDataResourceRetriever >>
>>> processorCreateFor:sourceSpec:andAnalystConfigSpec:
>>> (envId 0) @12 line 17
>>> FaFnDataResourceRetriever >>
>>> processorAt:sourceSpec:usingAnalystConfig: (envId 0) @2
>>> line 8
>>> FaFnDataAccessor >>
>>> processorAt:sourceSpec:usingAnalystConfig: (envId 0) @3
>>> line 2
>>> FaFnDataReportGeneratorV2 >>
>>> processorAt:sourceSpec:usingAnalystConfig: (envId 0) @3
>>> line 7
>>> DpNewFnSeriesProcessorReportQuery >> processor (envId 0)
>>> @22 line 10
>>> DpNewFnSeriesProcessorReportQuery >> getReport (envId 0)
>>> @4 line 6
>>> [] in DpNewFnSeriesProcessorReportQuery >>
>>> renderButtonsOn: (envId 0) @4 line 13
>>> [] in JSObject >> script:on: (envId 0) @8 line 12
>>> [] in WARenderContext >> document:during: (envId 0) @3
>>> line 6
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WARenderContext >> document:during: (envId 0) @3 line 7
>>> JSObject >> script:on: (envId 0) @7 line 8
>>> [] in JQAjax >> script: (envId 0) @10 line 10
>>> WARequestContext >> respond: (envId 0) @3 line 4
>>> [] in JQAjax >> respond: (envId 0) @3 line 4
>>> JQAjax >> processCallback (envId 0) @4 line 3
>>> [] in JQAjax >> enableCallbacks (envId 0) @2 line 6
>>> ExecBlock >> valueWithPossibleArguments: (envId 0) @6 line 4
>>> JSAjaxCallback >> evaluateWithArgument: (envId 0) @5 line 3
>>> WACallback >> evaluateWithFieldValues: (envId 0) @4 line 2
>>> [] in WACallbackRegistry >> handle: (envId 0) @4 line 10
>>> Collection >> do: (envId 0) @5 line 10
>>> WACallbackRegistry >> handle: (envId 0) @9 line 9
>>> WACallbackProcessingActionContinuation >>
>>> basicPerformAction (envId 0) @6 line 3
>>> [] in WAActionPhaseContinuation >> performAction (envId
>>> 0) @2 line 2
>>> ExecBlock >> onException:do: (envId 0) @2 line 66
>>> ExecBlock >> on:do: (envId 0) @5 line 47
>>> WAExceptionHandler >> handleExceptionsDuring: (envId 0)
>>> @2 line 3
>>> [] in WARenderLoopContinuation >>
>>> withNotificationHandlerDo: (envId 0) @2 line 20
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WARenderLoopContinuation >> withNotificationHandlerDo:
>>> (envId 0) @8 line 21
>>> WAActionPhaseContinuation >> performAction (envId 0) @2
>>> line 2
>>> [] in WACallbackProcessingActionContinuation >>
>>> performAction (envId 0) @2 line 3
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WACallbackProcessingActionContinuation >> performAction
>>> (envId 0) @2 line 3
>>> DpCallbackProcessingActionContinuation >> performAction
>>> (envId 0) @5 line 4
>>> WAActionPhaseContinuation >> handleFiltered: (envId 0)
>>> @2 line 2
>>> [] in WARequestHandler >> handle: (envId 0) @3 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> [] in WARequestContext >> push:during: (envId 0) @2 line 5
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WARequestContext >> push:during: (envId 0) @3 line 6
>>> WARequestHandler >> handle: (envId 0) @2 line 4
>>> [] in WASessionContinuation >> handle: (envId 0) @2 line 5
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WASessionContinuation >> withUnregisteredHandlerDo:
>>> (envId 0) @2 line 3
>>> WASessionContinuation >> handle: (envId 0) @4 line 5
>>> WASession >> handleFiltered: (envId 0) @20 line 20
>>> [] in WARequestHandler >> handle: (envId 0) @3 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> [] in WARequestContext >> push:during: (envId 0) @2 line 5
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WARequestContext >> push:during: (envId 0) @3 line 6
>>> WARequestHandler >> handle: (envId 0) @2 line 4
>>> WASession >> handle: (envId 0) @10 line 11
>>> WARegistry >> dispatch:to:key: (envId 0) @4 line 6
>>> WARegistry >> handleKeyed:with:context: (envId 0) @2 line 5
>>> WARegistry >> handleFiltered: (envId 0) @15 line 13
>>> WAApplication >> handleFiltered: (envId 0) @10 line 8
>>> WARequestFilter >> handleFiltered: (envId 0) @3 line 4
>>> [] in FaCurrentUserContextInformationFilter >>
>>> handleFiltered: (envId 0) @2 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> FaCurrentUserContextInformationFilter >> handleFiltered:
>>> (envId 0) @3 line 3
>>> WARequestFilter >> handleFiltered: (envId 0) @3 line 4
>>> [] in WAExceptionFilter >> handleFiltered: (envId 0) @2
>>> line 7
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> [] in WAExceptionFilter >> handleFiltered: (envId 0) @2
>>> line 6
>>> ExecBlock >> onException:do: (envId 0) @2 line 66
>>> ExecBlock >> on:do: (envId 0) @5 line 47
>>> WAExceptionHandler >> handleExceptionsDuring: (envId 0)
>>> @2 line 3
>>> WAExceptionFilter >> handleFiltered: (envId 0) @5 line 4
>>> [] in WARequestHandler >> handle: (envId 0) @3 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> [] in WARequestContext >> push:during: (envId 0) @2 line 5
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WARequestContext >> push:during: (envId 0) @3 line 6
>>> WARequestHandler >> handle: (envId 0) @2 line 4
>>> WADispatcher >> handleFiltered:named: (envId 0) @3 line 5
>>> WADispatcher >> handleFiltered: (envId 0) @8 line 6
>>> [] in WARequestHandler >> handle: (envId 0) @3 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WADynamicVariable class >> use:during: (envId 0) @2 line 4
>>> [] in WARequestContext >> push:during: (envId 0) @2 line 5
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WARequestContext >> push:during: (envId 0) @3 line 6
>>> WARequestHandler >> handle: (envId 0) @2 line 4
>>> [] in WAServerAdaptor >> handleRequest: (envId 0) @3 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WAServerAdaptor >> handleRequest: (envId 0) @2 line 5
>>> WAServerAdaptor >> handle: (envId 0) @2 line 4
>>> [] in WAServerAdaptor >> process: (envId 0) @2 line 6
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> WAServerAdaptor >> process: (envId 0) @4 line 7
>>> [] in WAFastCGIAdaptor >> process: (envId 0) @2 line 6
>>> [] in GRGemStonePlatform >>
>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @2
>>> line 11
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> [] in GRGemStonePlatform >>
>>> seasideProcessRequestWithRetry:resultBlock: (envId 0)
>>> @11 line 12
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> TransientRecursionLock >> critical: (envId 0) @11 line 12
>>> GRGemStonePlatform >>
>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @3
>>> line 5
>>> [] in GRGemStonePlatform >>
>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6
>>> line 9
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> GRGemStonePlatform >>
>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @2
>>> line 17
>>> WAFastCGIAdaptor >> process: (envId 0) @3 line 4
>>> [] in WAFastCGIAdaptor >> answerResponderRole: (envId
>>> 0) @2 line 4
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>>> FSResponderRole >> answer (envId 0) @3 line 4
>>> FSRole >> handleConnection (envId 0) @3 line 5
>>> FSConnection >> unsafeServe (envId 0) @5 line 8
>>> [] in FSConnection >> safeServe (envId 0) @2 line 8
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> [] in FSConnection >> safeServe (envId 0) @2 line 9
>>> ExecBlock >> on:do: (envId 0) @3 line 42
>>> [] in FSConnection >> safeServe (envId 0) @2 line 12
>>> ExecBlock >> ensure: (envId 0) @2 line 12
>>> FSConnection >> safeServe (envId 0) @2 line 15
>>> FSConnection >> serve (envId 0) @2 line 4
>>> [] in FSSocketServer >> listen: (envId 0) @3 line 15
>>> GsProcess >> _start (envId 0) @7 line 16
>>> GsNMethod class >> _gsReturnToC (envId 0) @1 line 1
>>> )
>>> self: aTransientRecursionLock
>>> .t1: anExecBlock
>>> .t2: anExecBlock
>>> receiver: aTransientRecursionLock
>>>
>>>
>>>
>>> On 04/24/2015 09:48 AM, Mariano Martinez Peck wrote:
>>>>
>>>> On Fri, Apr 24, 2015 at 1:40 PM, Dale Henrichs via
>>>> Glass <glass at lists.gemtalksystems.com
>>>> <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>>
>>>> Mariano,
>>>>
>>>> Debugging the ajax calls is a bit problematic
>>>> because the ajax response is in XML and arbitrary
>>>> xml (which is what is done for the html debugger)
>>>> is not an acceptable response ...
>>>>
>>>> Anyway ... I would like to see more of the error
>>>> stack specifically the first couple of frames of
>>>> the process.
>>>>
>>>> When the debugger is being opened on an error
>>>> continuation, we basically `execute` the
>>>> continuation using #debug as the value. The error
>>>> handling code inserts code that opens a debugger
>>>> when the value is #debug and voila the debugger is
>>>> opened.
>>>>
>>>> In your particular case it LOOKS like the
>>>> continuation was put into the object log without
>>>> the special debug code and the error you are
>>>> getting is the result of `executing` the
>>>> continuation without the benefit of the proper
>>>> session handlers wrapped around...
>>>>
>>>> BUT, I need to see the full stack or at the least
>>>> the bottom of the stack to confirm ...
>>>>
>>>>
>>>> Ok, thanks for the explanation. I attach the full stack.
>>>>
>>>> Finally, could you point me to the code where the
>>>> ajax handler resides? It's been a while since I've
>>>> been in that part of the code and frankly I recall
>>>> that there was a problem with error handling for
>>>> ajax, but I don't recall actually doing anything
>>>> about it ... so a pointer will help ... if I don't
>>>> find what I think I am looking for:)
>>>>
>>>>
>>>> I invoke my ajax handler this way:
>>>>
>>>>
>>>> html document
>>>> addLoadScript:
>>>> (html jQuery document
>>>> onAjaxError: (self ajaxErrorHandler asFunction:
>>>> #('event' 'jqxhr' 'settings' 'exception'))).
>>>>
>>>>
>>>> And #ajaxErrorHandler looks like this:
>>>>
>>>>
>>>> ajaxErrorHandler
>>>> ^ ' if (jqxhr.status == 403) {
>>>> alert("For security reasons we sign people out during
>>>> periods of inactivity. Please sign in again.");
>>>> window.location.href =
>>>> settings.url.split("?")[0].replace("help","");
>>>> } else {
>>>>
>>>> // This is on purpose because sometimes with TinyMCE we
>>>> would get status 0 and empty error...when there was no
>>>> error
>>>> // The reason is explained in:
>>>> http://bartwullems.blogspot.com.ar/2012/02/ajax-request-returns-status-0.html
>>>> if (jqxhr.readyState == 0 || jqxhr.status == 0) {
>>>> return; //Skip this error
>>>> };
>>>>
>>>> // Lets write to console all error info possbile
>>>> var requestResponse = {
>>>> url: settings.url,
>>>> method: settings.type,
>>>> data: settings.data,
>>>> httpStatus: jqxhr.status,
>>>> error: exception || jqxhr.statusText,
>>>> data: settings.data
>>>> };
>>>>
>>>> console.error(requestResponse);
>>>>
>>>> alert("This program just broke. You can either try
>>>> again, sign out and sign in and try again, or contact
>>>> us about error: " + exception);
>>>>
>>>> }'
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Dale
>>>>
>>>>
>>>> On 04/24/2015 08:45 AM, Mariano Martinez Peck via
>>>> Glass wrote:
>>>>> Hi guys,
>>>>>
>>>>> I think that at some point this worked for me...
>>>>> but it doesn't anymore... I have a seaside app
>>>>> and it is using the
>>>>> WAGemsToneWalkbackErrorHandler. I have an ajax
>>>>> request that it seems to be causing a walkback.
>>>>> Since it is an ajax call, I do not even receive
>>>>> the walkback page where it shows the stack and I
>>>>> have the buttons "remote debug", "full stack" ,
>>>>> etc. Instead.... I have a new entry in the Object
>>>>> Log. In other words, in GemTools I go to "Debug"
>>>>> and I see the exception there. THERE in the list,
>>>>> I can see the real error..for example
>>>>> "MessageNotUnderstood ocurred (error 2010), a Set
>>>>> does not undersntand #'on:'". However, when I
>>>>> click in the exception so that to open a debugger,
>>>>> the debugger cannot open the stack because it
>>>>> always fails to get the Seaside context (#session
>>>>> answers nil and therefore I get another new MNU
>>>>> about this). See attached screenshot.
>>>>>
>>>>> So... the questions is:
>>>>>
>>>>> 1) In the ideal world, I would like to debug it.
>>>>> 2) If I am not in the ideal world, then at least I
>>>>> would like to see the stack of the exception.
>>>>> Right now I cannot see ANYTHING... all I know is
>>>>> Set dnu #on: so yeah, probably a class I don't
>>>>> have a in GemStone that is getting a class side #on:
>>>>>
>>>>> Any idea how can I make this to work?
>>>>>
>>>>> ps: I have native code enabled. This is in a
>>>>> CentOS machine. I am using latest GemStone 3.1 and
>>>>> latest Seaside.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mariano
>>>>> http://marianopeck.wordpress.com
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Glass mailing list
>>>>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> <mailto:Glass at lists.gemtalksystems.com>
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>
>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150427/ebed3fbb/attachment-0001.html>
More information about the Glass
mailing list