[Glass] Again, Corrup Error preventing debugging real seaside exception
Dale Henrichs via Glass
glass at lists.gemtalksystems.com
Wed Feb 1 11:26:33 PST 2017
Darn, the ExecBlock isn't persistent then ... well, the next best thing
is to get the #sourceString of the home method for the block and/or the
#blockSource of the block. So we'll add a bit more logging information
to ExecBlock>>selfValue:
GsFile gciLogServer: 'Bug46661 ExecBlock oop: ', self asOop
printString.
GsFile gciLogServer: 'Bug46661 ExecBlock blockSource: ', self
blockSource.
GsFile gciLogServer: 'Bug46661 ExecBlock homeMethod inClass:
', self method homeMethod inClass printString.
GsFile gciLogServer: 'Bug46661 ExecBlock homeMethod selector:
', self method homeMethod selector asString.
GsFile gciLogServer: 'Bug46661 ExecBlock homeMethod source:
', self method homeMethod sourceString.
With regards to additional corrupObj errors, we are hitting the patch,
so there must be more than one failure mode involved. Can you get us the
stack from the other corruptobj errors? The stack should be slightly
different.
Dale
On 02/01/2017 10:47 AM, Mariano Martinez Peck wrote:
>
>
> On Wed, Feb 1, 2017 at 3:30 PM, Dale Henrichs
> <dale.henrichs at gemtalksystems.com
> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>
> Mariano,
>
> This is a GemStone bug (internal bug #46661 "error 2110 from
> ExecBlock >>selfValue").
>
> We'd like to be able to disassemble the home method of the
> ExecBlock that is causing the error, so that we can characterize
> (and fix) the bug properly. To that we need to get the oop of the
> ExecBlock from frame 9 of the stack ... given the oop of the
> ExecBlock, we'll be able to disassemble the home method and
> perhaps understand the root cause ...
>
> Probably the best way to get the oop of the ExecBlock is to
> install a modified version of the proposed patch for the bug and
> log the oop of the receiver:
>
> login as SystemUser and patch ExecBlock>>selfValue with the following:
>
> method: ExecBlock
> selfValue
> "return self in an active copy , or nil if self is not referenced."
> | theVc ofs |
> theVc := staticLink .
> theVc ~~ nil ifTrue:[ | vcOfs |
> vcOfs := self selfOffsetInVC .
> vcOfs > 2"VAR_CONTEXT_NAMED_SIZE" ifTrue:[
> vcOfs > theVc basicSize ifTrue:[
>
> GsFile gciLogServer: 'Bug46661 ExecBlock oop: ', self
> asOop printString.
>
> ^ nil "workaround bug 46661 <http://kermit/bug?bug=46661>"].
> ^ theVc _primitiveAt: vcOfs
> ].
> ].
> ofs := self _selfOffsetInSelf .
> ofs ~~ 0 ifTrue:[
> ofs := ofs - 1 .
> ^ self _primitiveAt: ofs .
> ].
> ^ nil
> %
>
> When you get the oop of the ExecBlock, use the Oop in the
> following and remove the gciLogServer: from the patch.
>
> Here's sample topaz code that you can run, once you get the oop of
> the exec block (our sample is based on getting the oop of the
> default sort block for a SortCollection.
>
> topaz 1> run
> SortedCollection new _defaultBlock
> %
> [20400641 sz:0 ExecBlock2] anExecBlock2
> topaz 1> disa @20400641
> block for home method 19569409, SortedCollection >> _defaultBlock
> method:19568385 sourceString:20808705 numArgs:2, public, No literals
> Step/Source IP Opcode Description (IP in bytes from &obj.hdr)
> ---/------ --- -------------- -----------
> 1/ 101 56 CHKInt
> 58 push_STK_s1 s1=24 bytes(3 words)
> 64 push_STK_s1 s1=16 bytes(2 words)
> 2/ 112 72 SEND sel=Symbol(oop 2366721) #'<=' env:0, 1 args,
> sendCache.u2=16
> 3/ 116 80 return_tos_u1 u1=32 bytes(4 words)
>
> topaz 1> disa @19569409
> SortedCollection >> _defaultBlock
> method:19569409 sourceString:20808705 numArgs:0, public, literals at IP 72
>
> Step/Source IP Opcode Description (IP in bytes from &obj.hdr)
> ---/------ --- -------------- -----------
> 1/ 1 56 InterpCHKInt
> 58 push_SBLOCK_u1 u1=14 (zlitIdx 6 Block oop 20400641
> (MethodId 19568385 ))
> 2/ 98 64 return_tos_u1 u1=16 bytes(2 words)
>
>
>
>
> Hi Dale,
>
>
> [debrisdemo at quuveserver1 logs]$ grep "Bug46661" *.log
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> gemnetobject8706.log:Bug46661 ExecBlock oop: 2425564417
> [debrisdemo at quuveserver1 logs]$
>
>
>
> topaz 2> disa @2425564417
> ERROR 2101 , a InternalError occurred (error 2101), The object with
> object ID 2425564417 does not exist. (InternalError)
> topaz 2>
>
>
> So...maybe it is a none persistent object?
>
> BTW should the workaround have fixed the original issue? Because I am
> still getting the Corrupt error even with the patched selfValue.
>
> Thanks!
>
>
>
>
> On 02/01/2017 10:00 AM, Dale Henrichs wrote:
>>
>> This looks like a smoking gun ... thanks ... I'll let you know if
>> we need more information.
>>
>>
>> On 02/01/2017 09:47 AM, Mariano Martinez Peck wrote:
>>>
>>>
>>> On Wed, Feb 1, 2017 at 2:42 PM, Dale Henrichs via Glass
>>> <glass at lists.gemtalksystems.com
>>> <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>
>>> Which version of GemStone?
>>>
>>>
>>> 3.3.3
>>>
>>> This looks to be some sort of error in trying to create a
>>> continuation in WAGemStoneWalkbackErrorHandler>>open: or
>>> WARemoteDebuggingWalkbackErrorHandler>>open:
>>>
>>> And looks to be similar to the error that Paul has hit in 3.3.1
>>>
>>>
>>> So...What I did is that in #open: I added code to write the
>>> ORIGINAL exception into the gem log:
>>>
>>> Transcript
>>> show:
>>> (String
>>> streamContents: [ :aStream |
>>> FACompatibilityUtils current printExceptionStack: err toLevel:
>>> 250 on: aStream ]).
>>>
>>>
>>>
>>> With that, I got the following information (please note the
>>> parts in bold).
>>>
>>>
>>>
>>> --transcript--'New style SigAbort hanlder'
>>> FastCGI Server started on port 40212
>>> --transcript--'Starting Gem for adaptor name: FastCGI with
>>> adaptor class: WAFastCGIAdaptor on port: 40212'
>>> --transcript--'Error stack for creating walkback: '
>>> --transcript--'1 FAGemStoneCompatibility class >>
>>> printExceptionStack:toLevel:on: @2 line 2 [GsNMethod 37854443265]
>>> 2 [] in ExecBlock1 (DpQuuveProductionWalkbackErrorHandler) >>
>>> open: @81 line 66 [GsNMethod 52025345]
>>> 3 String class (SequenceableCollection class) >>
>>> new:streamContents: @6 line 5 [GsNMethod 19668114433]
>>> 4 String class (SequenceableCollection class) >> streamContents:
>>> @2 line 3 [GsNMethod 19668113921]
>>> 5 [] in ExecBlock1 (DpQuuveProductionWalkbackErrorHandler) >>
>>> open: @71 line 66 [GsNMethod 52027649]
>>> 6 OutOfRange (AbstractException) >> _executeHandler: @4 line 8
>>> [GsNMethod 19134335489]
>>> 7 OutOfRange (AbstractException) >> _signalFromPrimitive: @1
>>> line 1 [GsNMethod 19134325505]
>>> 8 VariableContext (Object) >> _primitiveAt: @1 line 1
>>> [GsNMethod 19131848449]
>>> *9 ExecBlock0 (ExecBlock) >> selfValue @9 line 9 [GsNMethod
>>> 19132408321]*
>>> *10 GsProcess >> _frameContentsAt: @55 line 70 [GsNMethod
>>> 19133016321]*
>>> *11 GsContext >> continuation:level: @4 line 4 [GsNMethod
>>> 35975919105]*
>>> *12 GsContext class >> fromContinuation:atLevel: @3 line 3
>>> [GsNMethod 35975921409]*
>>> 13 GsContext >> sender @8 line 4 [GsNMethod 35975918593]
>>> 14 DpQuuveProductionWalkback (WAGemStoneWalkback) >>
>>> currentContextForContinuation: @12 line 6 [GsNMethod 35981465089]
>>> 15 DpQuuveProductionWalkback (WAGemStoneWalkback) >>
>>> currentContextForWalkback @3 line 2 [GsNMethod 35981465857]
>>> 16 DpQuuveProductionWalkback (WAGemStoneWalkback) >>
>>> initializeWithException: @3 line 3 [GsNMethod 35981466369]
>>> 17 DpQuuveProductionWalkback >>
>>> initializeWithException:continuation: @2 line 2 [GsNMethod
>>> 56277705729]
>>> 18 DpQuuveProductionWalkback class (WARemoteDebuggingWalkback
>>> class) >> exception:continuation: @3 line 2 [GsNMethod 35981461761]
>>> 19 [] in ExecBlock0 (DpQuuveProductionWalkbackErrorHandler) >>
>>> open: @43 line 38 [GsNMethod 52027905]
>>> 20 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>> 19132399361]
>>> 21 DpQuuveProductionWalkbackErrorHandler >> open: @15 line 50
>>> [GsNMethod 52034305]
>>> 22 DpQuuveProductionWalkbackErrorHandler
>>> (WAWalkbackErrorHandler) >> handleDefault: @2 line 2 [GsNMethod
>>> 35981553921]
>>> 23 DpQuuveProductionWalkbackErrorHandler (WAErrorHandler) >>
>>> handleError: @2 line 2 [GsNMethod 35977035521]
>>> 24 DpQuuveProductionWalkbackErrorHandler (WAErrorHandler) >>
>>> handleGemStoneException: @5 line 4 [GsNMethod 35976973313]
>>> 25 DpQuuveProductionWalkbackErrorHandler
>>> (WAGemStoneWalkbackErrorHandler) >> handleException: @2 line 2
>>> [GsNMethod 35981454337]
>>> 26 [] in ExecBlock0 (WAExceptionHandler) >>
>>> handleExceptionsDuring: @11 line 5 [GsNMethod 37840329473]
>>> 27 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>> 19132399361]
>>> 28 [] in ExecBlock1 (WAExceptionHandler) >>
>>> handleExceptionsDuring: @7 line 8 [GsNMethod 36035181057]
>>> 29 [] in ExecBlock1 (ExecBlock) >> on:do: @11 line 51
>>> [GsNMethod 19431386881]
>>> 30 InternalError (AbstractException) >> _executeOuterHandler: @4
>>> line 7 [GsNMethod 19134334977]
>>> 31 InternalError (AbstractException) >> _pass:with: @5 line 13
>>> [GsNMethod 19134331905]
>>> 32 InternalError (AbstractException) >> pass @2 line 14
>>> [GsNMethod 19134331137]
>>> 33 [] in ExecBlock1 (FaApplicationContextManager) >>
>>> inContext:do: @32 line 12 [GsNMethod 50887641345]
>>> 34 InternalError (AbstractException) >> _executeOuterHandler: @4
>>> line 7 [GsNMethod 19134334977]
>>> 35 InternalError (AbstractException) >> _pass:with: @5 line 13
>>> [GsNMethod 19134331905]
>>> 36 InternalError (AbstractException) >> pass @2 line 14
>>> [GsNMethod 19134331137]
>>> 37 [] in ExecBlock1 (FaBasicApplicationContext) >> renderOn:
>>> @23 line 11 [GsNMethod 56210914817]
>>> 38 InternalError (AbstractException) >> _executeHandler: @4 line
>>> 8 [GsNMethod 19134335489]
>>> 39 InternalError (AbstractException) >> _signalFromPrimitive: @1
>>> line 1 [GsNMethod 19134325505]
>>> 40 Utf8 (Object) >> _onStackAsSelfBelow: @1 line 1 [GsNMethod
>>> 19131797505]
>>> 41 [] in ExecBlock0 (Object) >> changeClassTo: @9 line 26
>>> [GsNMethod 19432195073]
>>> 42 ExecBlock0 (ExecBlock) >> ensure: @2 line 12 [GsNMethod
>>> 19132413953]
>>> 43 Utf8 (Object) >> changeClassTo: @5 line 30 [GsNMethod
>>> 19131817473]
>>> 44 GRUtf8CodecStream >> nextPutAll: @10 line 8 [GsNMethod
>>> 29463089665]
>>> 45 WAHtmlDocument (WADocument) >> nextPutAll: @2 line 2
>>> [GsNMethod 35976968705]
>>> 46 [] in ExecBlock2 (WAHtmlAttributes) >> encodeOn: @10 line 6
>>> [GsNMethod 36035602689]
>>> 47 WAHtmlAttributes (GRSmallDictionary) >> keysAndValuesDo: @6
>>> line 2 [GsNMethod 19745996545]
>>> 48 WAHtmlAttributes >> encodeOn: @3 line 4 [GsNMethod 26035337729]
>>> 49 WAHtmlDocument (WAXmlDocument) >> openTag:attributes:closed:
>>> @4 line 5 [GsNMethod 35962598145]
>>>
>>>
>>>
>>>
>>> That's the info I could get for the moment. Will continue
>>> trying to find more.
>>>
>>>
>>>
>>>
>>>
>>> Dale
>>>
>>> On 02/01/2017 09:09 AM, Mariano Martinez Peck via Glass wrote:
>>>> Hi
>>>>
>>>> I am clicking a button on my Seaside app and I get the
>>>> attached screenshot. It looks like there is an original problem
>>>>
>>>> 'Error creating WAWalkback: a OutOfRange occurred (error
>>>> 2110), for a VariableContext should be between 4 and 5'
>>>>
>>>> Which is then lost when trying to log the exception.
>>>>
>>>> Below is the gem log.
>>>>
>>>> Right now I would try to change
>>>> #internalServerErrorMessage: to see if I can prevent the
>>>> logError so that at least I am able to get the original
>>>> exception so that I can debug it and know what it is about....
>>>>
>>>>
>>>> -------
>>>>
>>>>
>>>>
>>>>
>>>> --transcript--'New style SigAbort hanlder'
>>>> FastCGI Server started on port 40211
>>>> --transcript--'Starting Gem for adaptor name: FastCGI with
>>>> adaptor class: WAFastCGIAdaptor on port: 40211'
>>>> --transcript--'Error creating WAWalkback: a OutOfRange
>>>> occurred (error 2110), for a VariableContext should be
>>>> between 4 and 5'
>>>> ----------- Internal FASTCGI LOG ENTRY: anArray-----------
>>>> ----------- Internal FASTCGI ERROR Encountered:
>>>> 2017-02-01T10:18:31.43275594711304-05:00
>>>> a TransactionError occurred (error 2249), Further commits
>>>> have been disabled for this session because: 'CorruptObj,
>>>> fetch past end'. This session must logout.
>>>> 1 GRGemStonePlatform >> logError:title:shouldCommit: @3
>>>> line 4 [GsNMethod 19837057537]
>>>> 2 GRGemStonePlatform >> logError:title: @2 line 3
>>>> [GsNMethod 19837063681]
>>>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: @21 line
>>>> 14 [GsNMethod 37855252481]
>>>> 4 [] in ExecBlock1 (GRGemStonePlatform) >>
>>>> seasideProcessRequest:adaptor:resultBlock: @32 line 15
>>>> [GsNMethod 36040141313]
>>>> 5 TransactionError (AbstractException) >>
>>>> _executeOuterHandler: @4 line 7 [GsNMethod 19134334977]
>>>> 6 TransactionError (AbstractException) >> _pass:with: @5
>>>> line 13 [GsNMethod 19134331905]
>>>> 7 TransactionError (AbstractException) >> pass @2 line 14
>>>> [GsNMethod 19134331137]
>>>> 8 [] in ExecBlock1 (System class) >> _localCommit: @24
>>>> line 34 [GsNMethod 19431909377]
>>>> 9 TransactionError (AbstractException) >> _executeHandler:
>>>> @4 line 8 [GsNMethod 19134335489]
>>>> 10 TransactionError (AbstractException) >>
>>>> _signalFromPrimitive: @1 line 1 [GsNMethod 19134325505]
>>>> 11 System class >> _primitiveCommit: @1 line 1 [GsNMethod
>>>> 19133373185]
>>>> 12 System class >> __commit: @2 line 8 [GsNMethod 19133359617]
>>>> 13 [] in ExecBlock0 (System class) >> _localCommit: @21
>>>> line 30 [GsNMethod 19431909633]
>>>> 14 ExecBlock0 (ExecBlock) >> onException:do: @2 line 66
>>>> [GsNMethod 19132400129]
>>>> 15 System class >> _localCommit: @9 line 31 [GsNMethod
>>>> 19133359873]
>>>> 16 SessionMethodTransactionBoundaryPolicy
>>>> (TransactionBoundaryDefaultPolicy) >> commit: @3 line 3
>>>> [GsNMethod 19470788865]
>>>> 17 System class >> _commit: @8 line 16 [GsNMethod 19133360129]
>>>> 18 System class >> commitTransaction @5 line 7 [GsNMethod
>>>> 19133369089]
>>>> 19 System class >> _commitPrintingDiagnostics @3 line 8
>>>> [GsNMethod 19133283585]
>>>> 20 SystemCommitTransaction >> defaultAction @2 line 3
>>>> [GsNMethod 19832138497]
>>>> 21 SystemCommitTransaction (AbstractException) >>
>>>> _signalWith: @6 line 25 [GsNMethod 19134336001]
>>>> 22 SystemCommitTransaction class (AbstractException class)
>>>> >> signal @3 line 5 [GsNMethod 19134317825]
>>>> 23 GRGemStonePlatform >> doCommitTransaction @4 line 3
>>>> [GsNMethod 19837059329]
>>>> 24 [] in ExecBlock0 (GRGemStonePlatform) >>
>>>> seasideProcessRequestWithRetry:resultBlock: @30 line 32
>>>> [GsNMethod 36040140801]
>>>> 25 ExecBlock0 (ExecBlock) >> ensure: @2 line 12 [GsNMethod
>>>> 19132413953]
>>>> 26 TransientRecursionLock >> critical: @12 line 12
>>>> [GsNMethod 30973507329]
>>>> 27 GRGemStonePlatform >>
>>>> seasideProcessRequestWithRetry:resultBlock: @4 line 6
>>>> [GsNMethod 35976432641]
>>>> 28 [] in ExecBlock1 (GRGemStonePlatform) >>
>>>> seasideProcessRequest:adaptor:resultBlock: @18 line 6
>>>> [GsNMethod 37840320257]
>>>> 29 Array (Collection) >> do: @6 line 10 [GsNMethod
>>>> 19132528897]
>>>> 30 [] in ExecBlock0 (GRGemStonePlatform) >>
>>>> seasideProcessRequest:adaptor:resultBlock: @8 line 5
>>>> [GsNMethod 36040141569]
>>>> 31 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>>> 19132399361]
>>>> 32 GRGemStonePlatform >>
>>>> seasideProcessRequest:adaptor:resultBlock: @3 line 12
>>>> [GsNMethod 35976541697]
>>>> 33 WAFastCGIAdaptor >> process: @4 line 4 [GsNMethod
>>>> 37855252225]
>>>> 34 [] in ExecBlock0 (WAFastCGIAdaptor) >>
>>>> answerResponderRole: @8 line 4 [GsNMethod 50547927553]
>>>> 35 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>>> 19132399361]
>>>> 36 WAFastCGIAdaptor >> answerResponderRole: @3 line 5
>>>> [GsNMethod 37855252993]
>>>> 37 FSResponderRole >> answer @3 line 4 [GsNMethod 37855475457]
>>>> 38 FSResponderRole (FSRole) >> handleConnection @3 line 5
>>>> [GsNMethod 37855487745]
>>>> 39 FSConnection >> unsafeServe @5 line 8 [GsNMethod
>>>> 37855375105]
>>>> 40 [] in ExecBlock0 (FSConnection) >> safeServe @15 line 8
>>>> [GsNMethod 56276117761]
>>>> 41 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>>> 19132399361]
>>>> 42 [] in ExecBlock0 (FSConnection) >> safeServe @11 line 9
>>>> [GsNMethod 56211351041]
>>>> 43 ExecBlock0 (ExecBlock) >> on:do: @3 line 44 [GsNMethod
>>>> 19132399361]
>>>> 44 [] in ExecBlock0 (FSConnection) >> safeServe @7 line 12
>>>> [GsNMethod 50842477057]
>>>> 45 ExecBlock0 (ExecBlock) >> ensure: @2 line 12 [GsNMethod
>>>> 19132413953]
>>>> 46 FSConnection >> safeServe @3 line 15 [GsNMethod
>>>> 37855377153]
>>>> 47 FSConnection >> serve @2 line 4 [GsNMethod 37855376641]
>>>> 48 [] in ExecBlock (FSSocketServer) >> listen: @15 line 15
>>>> [GsNMethod 50843215361]
>>>> 49 ExecBlock >> valueWithArguments: @1 line 1 [GsNMethod
>>>> 19132413697]
>>>> 50 GsProcess >> _start @8 line 16 [GsNMethod 19132995841]
>>>> 51 <Reenter marker>
>>>> -----------
>>>> --transcript--'handled sigabort:
>>>> 2017-02-01T10:28:51.61530303955078-05:00'
>>>> GemStone signal handler: signal 15 (SIGTERM), received from
>>>> process 1 userId 0
>>>> si_code: 0 at 1485964384, SI_USER, signal from kill(2),
>>>> sigsend(2), raise(3C) or abort(3C)
>>>> [debrisdemo at quuveserver1 logs]$
>>>>
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>> <http://marianopeck.wordpress.com>
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> <mailto:Glass at lists.gemtalksystems.com>
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>> <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>> _______________________________________________ Glass
>>> mailing list Glass at lists.gemtalksystems.com
>>> <mailto:Glass at lists.gemtalksystems.com>
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>>
>>> --
>>> Mariano http://marianopeck.wordpress.com
>>> <http://marianopeck.wordpress.com>
>
> --
> Mariano http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20170201/863962ef/attachment-0001.html>
More information about the Glass
mailing list