[Glass] Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
Dale Henrichs
dale.henrichs at gemtalksystems.com
Tue May 27 12:38:06 PDT 2014
... and opened a GemStone Seaside3.1 issue[1]...
Dale
[1] https://github.com/glassdb/Seaside31/issues/20
On Tue, May 27, 2014 at 12:18 PM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:
> I forgot to mention that internally we are tracking this as bug 44291...
>
> Dale
>
>
> On Tue, May 27, 2014 at 12:16 PM, Dale Henrichs <
> dale.henrichs at gemtalksystems.com> wrote:
>
>> Okay, I've talked to engineering and they agree that the corruptObj error
>> in this case is too extreme. We are in the final stages of a 3.2.1 release,
>> so it may be too late to include a fix in 3.2.1, but it will most likely be
>> included in a 3.2.2, but we don't have a schedule for that.
>>
>> Until then I think that the following patch
>> to GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: is the
>> best we can do:
>>
>>
>> ----- Method:
>> GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: (in category
>> '*seaside-gemstone-core') -----seasideProcessRequest: aNativeRequest
>> adaptor: adaptor resultBlock: resultBlock
>> | result |
>> [
>> | retryCount result count |
>> retryCount := self retryCount.
>> count := 1.
>> retryCount
>> timesRepeat: [
>> (result := self
>> seasideProcessRequestWithRetry: aNativeRequest
>> resultBlock: resultBlock) ~~ nil
>> ifTrue: [ ^ result ].
>> System _sessionCacheStatAt: 2 put: (System _sessionCacheStatAt:
>> 2) + 1. "requests retried"
>> (Delay forMilliseconds: self retryDelay * count) wait.
>> count := count * 10 ]. "exceeded retry limit"
>> ^ adaptor
>> internalServerErrorMessage:
>> 'Too many retries: ' , (retryCount + 1) printString ]
>> on: Error
>> do: [ :ex |
>> self doAbortTransaction.
>> + result := adaptor internalServerErrorMessage: ex description.
>> + ex gsNumber == 2261
>> + ifTrue:
>> + [
>> + "see discussion in:
>> http://forum.world.st/Glass-Further-commits-have-been-disabled-for-this-session-because-CorruptObj-error-This-session-must-td4760409.html
>> + "
>> + Transcript
>> + cr;
>> + show:
>> + 'Session terminating due to CorruptObj error ...
>> session must logout'.
>> + System logout ]
>> + ex
>> + return: nil "Do an explicit return. Because of the abort
>> above, the default action for an exception (resume) is set. see bug39246."
>> ].
>> - result := adaptor internalServerErrorMessage: ex description. "Do
>> an explicit return. Because of the abort above, the default action for an
>> exception (resume) is set. see bug39246."
>> - ex return: nil ].
>> ^ result
>>
>> If you use this patch, make sure that you have a system in place for
>> automatically restarting gems that get fatal errors. You should be using
>> monit or daaemontools in production since gems may crash occasionally and
>> they need to be restarted. Other than the extra time/cpu/disk for
>> restarting a gem, stopping/restarting gems is not a fatal problem as no
>> persistent data is lost or corrupted ... see my post on "GLASS 101:
>> Disposable Gems, Durable Data"[1].
>>
>> I'm on the fence about including this patch in the standard release of
>> Seaside for GemStone ... on the one hand if you get this particular error
>> (or any other "commit prohibiting error" the gem is effectively out of
>> service and needs to be restarted...
>>
>> An alternative to logout would be to update a field in SessionTemps that
>> indicates the the gem needs to be restarted ... that session state can be
>> queried by a monit http request (presumably) ... so at the end of the day
>> we should probably invent some sort of gem termination policy that can be
>> controlled at the application level...
>>
>> Dale
>>
>>
>> [1]
>> http://gemstonesoup.wordpress.com/2008/03/08/glass-101-disposable-gems-durable-data/
>>
>>
>>
>> On Mon, May 26, 2014 at 10:33 AM, Dale Henrichs <
>> dale.henrichs at gemtalksystems.com> wrote:
>>
>>> Johan,
>>>
>>> I think that a corruptObj error in this case may be a bit extreme. I
>>> have asked engineering if there is a "session safe" method for finding the
>>> obviously invalid utf8 ... Today is a holiday, so I may not get a response
>>> until tomorrow.
>>>
>>> I hope that crashed vms is an acceptable outcome of the security audit?
>>>
>>> The corruptObj error is our response to the error thrown by ICU ... the
>>> commit prohibiting error normally is thrown when we think that "memory
>>> stomping" has occurred and we want to avoid persisting potentially corrupt
>>> objects ... presumably the security audit folks knew that mishandling this
>>> particular ICU error condition could lead to a security breach:)
>>>
>>> Dale
>>>
>>>
>>> On Mon, May 26, 2014 at 9:44 AM, Johan Brichau <johan at yesplan.be> wrote:
>>>
>>>> Hi,
>>>>
>>>> Today, we had a security audit on a Seaside 3.0.10 application running
>>>> in a GS 3.1.0.5 stone with FastCGI behind nginx.
>>>>
>>>> I have no idea what exactly they did to obtain this, but the system
>>>> went unresponsive after the following error until I restarted the gems.
>>>> - a InternalError occurred (error 2261), The object with object ID
>>>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>>>> - a TransactionError occurred (error 2249), Further commits have been
>>>> disabled for this session because: 'CorruptObj error'. This session must
>>>> logout.
>>>> - repeat last one
>>>>
>>>> The stacks from the gem log are attached.
>>>>
>>>> I am trying to trace what happened. Any clues?
>>>>
>>>> Johan
>>>>
>>>> ----------- Internal FASTCGI ERROR Encountered:
>>>> 2014-05-26T13:37:41.31612992286682+02:00
>>>> a InternalError occurred (error 2261), The object with object ID
>>>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>>>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line
>>>> 4 [GsNMethod 7469480705]
>>>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod
>>>> 7469494785]
>>>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14
>>>> [GsNMethod 9828254465]
>>>> 4 [] in GRGemStonePlatform >>
>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod
>>>> 10506396161]
>>>> 5 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod
>>>> 1394121473]
>>>> 6 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>>>> [GsNMethod 1360964097]
>>>> 7 String >> decodeFromUTF8 (envId 0) @1 line 1 [GsNMethod 1064104705]
>>>> 8 UTF8PrimitiveEncoding >> decode: (envId 0) @2 line 4 [GsNMethod
>>>> 7470277121]
>>>> 9 GRUtf8GemStoneCodec >> decode: (envId 0) @3 line 3 [GsNMethod
>>>> 7468617985]
>>>> 10 [] in WAUrl >> decodedWith: (envId 0) @3 line 17 [GsNMethod
>>>> 9230396417]
>>>> 11 SequenceableCollection >> collect: (envId 0) @9 line 16 [GsNMethod
>>>> 1064183041]
>>>> 12 WAUrl >> decodedWith: (envId 0) @22 line 17 [GsNMethod 8789933313]
>>>> 13 WAFastCGIRequestConverter >> requestUrlFor: (envId 0) @6 line 4
>>>> [GsNMethod 9828231425]
>>>> 14 WAServerAdaptor >> requestFor: (envId 0) @3 line 6 [GsNMethod
>>>> 8790261761]
>>>> 15 WAFastCGIRequestConverter >> requestFor: (envId 0) @12 line 7
>>>> [GsNMethod 9828215809]
>>>> 16 WAFastCGIAdaptor >> requestFor: (envId 0) @2 line 4 [GsNMethod
>>>> 9828250881]
>>>> 17 WAServerAdaptor >> contextFor: (envId 0) @2 line 5 [GsNMethod
>>>> 8790264577]
>>>> 18 WAServerAdaptor >> process: (envId 0) @2 line 5 [GsNMethod
>>>> 8790258433]
>>>> 19 [] in WAFastCGIAdaptor >> process: (envId 0) @2 line 6 [GsNMethod
>>>> 8794996737]
>>>> 20 [] in GRGemStonePlatform >>
>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @2 line 13
>>>> [GsNMethod 10501067521]
>>>> 21 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 22 [] in GRGemStonePlatform >>
>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @14 line 14
>>>> [GsNMethod 10506395905]
>>>> 23 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>> 24 TransientRecursionLock >> critical: (envId 0) @11 line 12
>>>> [GsNMethod 6527748609]
>>>> 25 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>>>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>>>> 26 [] in GRGemStonePlatform >>
>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>>>> 10506396417]
>>>> 27 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 28 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>>>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>>>> 29 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>>>> 9828418049]
>>>> 30 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>>>> [GsNMethod 8795113729]
>>>> 31 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 32 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>>>> [GsNMethod 9828248321]
>>>> 33 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
>>>> 34 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod
>>>> 8854243329]
>>>> 35 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod
>>>> 8853951745]
>>>> 36 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>>>> 9557561601]
>>>> 37 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>>>> 9322044673]
>>>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>>>> 9258731777]
>>>> 41 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>> 42 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod
>>>> 8853958913]
>>>> 43 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>>>> 44 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>>>> 9261209601]
>>>> 45 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>>>> 46 <Reenter marker>
>>>> -----------
>>>> ----------- Internal FASTCGI LOG ENTRY: anArray-----------
>>>> ----------- Internal FASTCGI ERROR Encountered:
>>>> 2014-05-26T13:37:41.37823009490967+02:00
>>>> a TransactionError occurred (error 2249), Further commits have been
>>>> disabled for this session because: 'CorruptObj error'. This session must
>>>> logout.
>>>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line
>>>> 4 [GsNMethod 7469480705]
>>>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod
>>>> 7469494785]
>>>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14
>>>> [GsNMethod 9828254465]
>>>> 4 [] in GRGemStonePlatform >>
>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod
>>>> 10506396161]
>>>> 5 AbstractException >> _executeOuterHandler: (envId 0) @3 line 7
>>>> [GsNMethod 1394117633]
>>>> 6 AbstractException >> _pass:with: (envId 0) @4 line 13 [GsNMethod
>>>> 1393936385]
>>>> 7 AbstractException >> pass (envId 0) @2 line 14 [GsNMethod 1393935361]
>>>> 8 [] in System class >> _localCommit: (envId 0) @2 line 34 [GsNMethod
>>>> 5944183041]
>>>> 9 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod
>>>> 1394121473]
>>>> 10 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>>>> [GsNMethod 1360964097]
>>>> 11 System class >> _primitiveCommit: (envId 0) @1 line 1 [GsNMethod
>>>> 1789434881]
>>>> 12 System class >> __commit: (envId 0) @2 line 8 [GsNMethod 1789345025]
>>>> 13 [] in System class >> _localCommit: (envId 0) @2 line 30
>>>> [GsNMethod 5944183297]
>>>> 14 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod
>>>> 1064628225]
>>>> 15 System class >> _localCommit: (envId 0) @8 line 31 [GsNMethod
>>>> 1789345281]
>>>> 16 TransactionBoundaryDefaultPolicy >> commit: (envId 0) @2 line 3
>>>> [GsNMethod 5986577665]
>>>> 17 System class >> _commit: (envId 0) @7 line 16 [GsNMethod 1789345537]
>>>> 18 System class >> commitTransaction (envId 0) @5 line 7 [GsNMethod
>>>> 1789402113]
>>>> 19 System class >> _commitPrintingDiagnostics (envId 0) @2 line 8
>>>> [GsNMethod 1700522241]
>>>> 20 SystemCommitTransaction >> defaultAction (envId 0) @2 line 3
>>>> [GsNMethod 7468825857]
>>>> 21 AbstractException >> _signalWith: (envId 0) @5 line 25 [GsNMethod
>>>> 1394122241]
>>>> 22 AbstractException class >> signal (envId 0) @3 line 5 [GsNMethod
>>>> 1172775681]
>>>> 23 GRGemStonePlatform >> doCommitTransaction (envId 0) @4 line 3
>>>> [GsNMethod 7469481473]
>>>> 24 [] in GRGemStonePlatform >>
>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @47 line 54
>>>> [GsNMethod 10506395905]
>>>> 25 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>> 26 TransientRecursionLock >> critical: (envId 0) @11 line 12
>>>> [GsNMethod 6527748609]
>>>> 27 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>>>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>>>> 28 [] in GRGemStonePlatform >>
>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>>>> 10506396417]
>>>> 29 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 30 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>>>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>>>> 31 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>>>> 9828418049]
>>>> 32 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>>>> [GsNMethod 8795113729]
>>>> 33 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 34 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>>>> [GsNMethod 9828248321]
>>>> 35 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
>>>> 36 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod
>>>> 8854243329]
>>>> 37 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod
>>>> 8853951745]
>>>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>>>> 9557561601]
>>>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>>>> 9322044673]
>>>> 41 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>> 42 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>>>> 9258731777]
>>>> 43 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>> 44 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod
>>>> 8853958913]
>>>> 45 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>>>> 46 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>>>> 9261209601]
>>>> 47 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>>>> 48 <Reenter marker>
>>>> -----------
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140527/952132be/attachment-0001.html>
More information about the Glass
mailing list