[Glass] Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
Dale Henrichs
dale.henrichs at gemtalksystems.com
Tue May 27 12:54:40 PDT 2014
... and it turns that the particular case that was causing the corruptObj
error in 3.1.0.5 produces an ArgumentError in 3.2 ... there are some paths
in the 3.2 code base that could lead to corruptObj errors and THOSE will be
fixed in 3.2.x. If we do a 3.1.0.7 release, the bug will be fixed there.
Right now there is no schedule for a 3.1.0.7 release.
Dale
On Tue, May 27, 2014 at 12:38 PM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:
> ... and opened a GemStone Seaside3.1 issue[1]...
>
> Dale
>
> [1] https://github.com/glassdb/Seaside31/issues/20
>
>
> On Tue, May 27, 2014 at 12:18 PM, Dale Henrichs <
> dale.henrichs at gemtalksystems.com> wrote:
>
>> I forgot to mention that internally we are tracking this as bug 44291...
>>
>> Dale
>>
>>
>> On Tue, May 27, 2014 at 12:16 PM, Dale Henrichs <
>> dale.henrichs at gemtalksystems.com> wrote:
>>
>>> Okay, I've talked to engineering and they agree that the corruptObj
>>> error in this case is too extreme. We are in the final stages of a 3.2.1
>>> release, so it may be too late to include a fix in 3.2.1, but it will most
>>> likely be included in a 3.2.2, but we don't have a schedule for that.
>>>
>>> Until then I think that the following patch
>>> to GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: is the
>>> best we can do:
>>>
>>>
>>> ----- Method:
>>> GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: (in category
>>> '*seaside-gemstone-core') -----seasideProcessRequest: aNativeRequest
>>> adaptor: adaptor resultBlock: resultBlock
>>> | result |
>>> [
>>> | retryCount result count |
>>> retryCount := self retryCount.
>>> count := 1.
>>> retryCount
>>> timesRepeat: [
>>> (result := self
>>> seasideProcessRequestWithRetry: aNativeRequest
>>> resultBlock: resultBlock) ~~ nil
>>> ifTrue: [ ^ result ].
>>> System _sessionCacheStatAt: 2 put: (System _sessionCacheStatAt:
>>> 2) + 1. "requests retried"
>>> (Delay forMilliseconds: self retryDelay * count) wait.
>>> count := count * 10 ]. "exceeded retry limit"
>>> ^ adaptor
>>> internalServerErrorMessage:
>>> 'Too many retries: ' , (retryCount + 1) printString ]
>>> on: Error
>>> do: [ :ex |
>>> self doAbortTransaction.
>>> + result := adaptor internalServerErrorMessage: ex description.
>>> + ex gsNumber == 2261
>>> + ifTrue:
>>> + [
>>> + "see discussion in:
>>> http://forum.world.st/Glass-Further-commits-have-been-disabled-for-this-session-because-CorruptObj-error-This-session-must-td4760409.html
>>> + "
>>> + Transcript
>>> + cr;
>>> + show:
>>> + 'Session terminating due to CorruptObj error ...
>>> session must logout'.
>>> + System logout ]
>>> + ex
>>> + return: nil "Do an explicit return. Because of the abort
>>> above, the default action for an exception (resume) is set. see bug39246."
>>> ].
>>> - result := adaptor internalServerErrorMessage: ex description. "Do
>>> an explicit return. Because of the abort above, the default action for an
>>> exception (resume) is set. see bug39246."
>>> - ex return: nil ].
>>> ^ result
>>>
>>> If you use this patch, make sure that you have a system in place for
>>> automatically restarting gems that get fatal errors. You should be using
>>> monit or daaemontools in production since gems may crash occasionally and
>>> they need to be restarted. Other than the extra time/cpu/disk for
>>> restarting a gem, stopping/restarting gems is not a fatal problem as no
>>> persistent data is lost or corrupted ... see my post on "GLASS 101:
>>> Disposable Gems, Durable Data"[1].
>>>
>>> I'm on the fence about including this patch in the standard release of
>>> Seaside for GemStone ... on the one hand if you get this particular error
>>> (or any other "commit prohibiting error" the gem is effectively out of
>>> service and needs to be restarted...
>>>
>>> An alternative to logout would be to update a field in SessionTemps that
>>> indicates the the gem needs to be restarted ... that session state can be
>>> queried by a monit http request (presumably) ... so at the end of the day
>>> we should probably invent some sort of gem termination policy that can be
>>> controlled at the application level...
>>>
>>> Dale
>>>
>>>
>>> [1]
>>> http://gemstonesoup.wordpress.com/2008/03/08/glass-101-disposable-gems-durable-data/
>>>
>>>
>>>
>>> On Mon, May 26, 2014 at 10:33 AM, Dale Henrichs <
>>> dale.henrichs at gemtalksystems.com> wrote:
>>>
>>>> Johan,
>>>>
>>>> I think that a corruptObj error in this case may be a bit extreme. I
>>>> have asked engineering if there is a "session safe" method for finding the
>>>> obviously invalid utf8 ... Today is a holiday, so I may not get a response
>>>> until tomorrow.
>>>>
>>>> I hope that crashed vms is an acceptable outcome of the security audit?
>>>>
>>>> The corruptObj error is our response to the error thrown by ICU ... the
>>>> commit prohibiting error normally is thrown when we think that "memory
>>>> stomping" has occurred and we want to avoid persisting potentially corrupt
>>>> objects ... presumably the security audit folks knew that mishandling this
>>>> particular ICU error condition could lead to a security breach:)
>>>>
>>>> Dale
>>>>
>>>>
>>>> On Mon, May 26, 2014 at 9:44 AM, Johan Brichau <johan at yesplan.be>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Today, we had a security audit on a Seaside 3.0.10 application running
>>>>> in a GS 3.1.0.5 stone with FastCGI behind nginx.
>>>>>
>>>>> I have no idea what exactly they did to obtain this, but the system
>>>>> went unresponsive after the following error until I restarted the gems.
>>>>> - a InternalError occurred (error 2261), The object with object ID
>>>>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>>>>> - a TransactionError occurred (error 2249), Further commits have been
>>>>> disabled for this session because: 'CorruptObj error'. This session must
>>>>> logout.
>>>>> - repeat last one
>>>>>
>>>>> The stacks from the gem log are attached.
>>>>>
>>>>> I am trying to trace what happened. Any clues?
>>>>>
>>>>> Johan
>>>>>
>>>>> ----------- Internal FASTCGI ERROR Encountered:
>>>>> 2014-05-26T13:37:41.31612992286682+02:00
>>>>> a InternalError occurred (error 2261), The object with object ID
>>>>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>>>>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line
>>>>> 4 [GsNMethod 7469480705]
>>>>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3
>>>>> [GsNMethod 7469494785]
>>>>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line
>>>>> 14 [GsNMethod 9828254465]
>>>>> 4 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod
>>>>> 10506396161]
>>>>> 5 AbstractException >> _executeHandler: (envId 0) @3 line 8
>>>>> [GsNMethod 1394121473]
>>>>> 6 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>>>>> [GsNMethod 1360964097]
>>>>> 7 String >> decodeFromUTF8 (envId 0) @1 line 1 [GsNMethod 1064104705]
>>>>> 8 UTF8PrimitiveEncoding >> decode: (envId 0) @2 line 4 [GsNMethod
>>>>> 7470277121]
>>>>> 9 GRUtf8GemStoneCodec >> decode: (envId 0) @3 line 3 [GsNMethod
>>>>> 7468617985]
>>>>> 10 [] in WAUrl >> decodedWith: (envId 0) @3 line 17 [GsNMethod
>>>>> 9230396417]
>>>>> 11 SequenceableCollection >> collect: (envId 0) @9 line 16 [GsNMethod
>>>>> 1064183041]
>>>>> 12 WAUrl >> decodedWith: (envId 0) @22 line 17 [GsNMethod 8789933313]
>>>>> 13 WAFastCGIRequestConverter >> requestUrlFor: (envId 0) @6 line 4
>>>>> [GsNMethod 9828231425]
>>>>> 14 WAServerAdaptor >> requestFor: (envId 0) @3 line 6 [GsNMethod
>>>>> 8790261761]
>>>>> 15 WAFastCGIRequestConverter >> requestFor: (envId 0) @12 line 7
>>>>> [GsNMethod 9828215809]
>>>>> 16 WAFastCGIAdaptor >> requestFor: (envId 0) @2 line 4 [GsNMethod
>>>>> 9828250881]
>>>>> 17 WAServerAdaptor >> contextFor: (envId 0) @2 line 5 [GsNMethod
>>>>> 8790264577]
>>>>> 18 WAServerAdaptor >> process: (envId 0) @2 line 5 [GsNMethod
>>>>> 8790258433]
>>>>> 19 [] in WAFastCGIAdaptor >> process: (envId 0) @2 line 6 [GsNMethod
>>>>> 8794996737]
>>>>> 20 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @2 line 13
>>>>> [GsNMethod 10501067521]
>>>>> 21 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 22 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @14 line 14
>>>>> [GsNMethod 10506395905]
>>>>> 23 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>>> 24 TransientRecursionLock >> critical: (envId 0) @11 line 12
>>>>> [GsNMethod 6527748609]
>>>>> 25 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>>>>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>>>>> 26 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>>>>> 10506396417]
>>>>> 27 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 28 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>>>>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>>>>> 29 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>>>>> 9828418049]
>>>>> 30 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>>>>> [GsNMethod 8795113729]
>>>>> 31 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 32 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>>>>> [GsNMethod 9828248321]
>>>>> 33 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod
>>>>> 8854227457]
>>>>> 34 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod
>>>>> 8854243329]
>>>>> 35 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod
>>>>> 8853951745]
>>>>> 36 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>>>>> 9557561601]
>>>>> 37 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>>>>> 9322044673]
>>>>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>>>>> 9258731777]
>>>>> 41 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>>> 42 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod
>>>>> 8853958913]
>>>>> 43 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>>>>> 44 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>>>>> 9261209601]
>>>>> 45 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>>>>> 46 <Reenter marker>
>>>>> -----------
>>>>> ----------- Internal FASTCGI LOG ENTRY: anArray-----------
>>>>> ----------- Internal FASTCGI ERROR Encountered:
>>>>> 2014-05-26T13:37:41.37823009490967+02:00
>>>>> a TransactionError occurred (error 2249), Further commits have been
>>>>> disabled for this session because: 'CorruptObj error'. This session must
>>>>> logout.
>>>>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line
>>>>> 4 [GsNMethod 7469480705]
>>>>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3
>>>>> [GsNMethod 7469494785]
>>>>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line
>>>>> 14 [GsNMethod 9828254465]
>>>>> 4 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod
>>>>> 10506396161]
>>>>> 5 AbstractException >> _executeOuterHandler: (envId 0) @3 line 7
>>>>> [GsNMethod 1394117633]
>>>>> 6 AbstractException >> _pass:with: (envId 0) @4 line 13 [GsNMethod
>>>>> 1393936385]
>>>>> 7 AbstractException >> pass (envId 0) @2 line 14 [GsNMethod
>>>>> 1393935361]
>>>>> 8 [] in System class >> _localCommit: (envId 0) @2 line 34
>>>>> [GsNMethod 5944183041]
>>>>> 9 AbstractException >> _executeHandler: (envId 0) @3 line 8
>>>>> [GsNMethod 1394121473]
>>>>> 10 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>>>>> [GsNMethod 1360964097]
>>>>> 11 System class >> _primitiveCommit: (envId 0) @1 line 1 [GsNMethod
>>>>> 1789434881]
>>>>> 12 System class >> __commit: (envId 0) @2 line 8 [GsNMethod
>>>>> 1789345025]
>>>>> 13 [] in System class >> _localCommit: (envId 0) @2 line 30
>>>>> [GsNMethod 5944183297]
>>>>> 14 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod
>>>>> 1064628225]
>>>>> 15 System class >> _localCommit: (envId 0) @8 line 31 [GsNMethod
>>>>> 1789345281]
>>>>> 16 TransactionBoundaryDefaultPolicy >> commit: (envId 0) @2 line 3
>>>>> [GsNMethod 5986577665]
>>>>> 17 System class >> _commit: (envId 0) @7 line 16 [GsNMethod
>>>>> 1789345537]
>>>>> 18 System class >> commitTransaction (envId 0) @5 line 7 [GsNMethod
>>>>> 1789402113]
>>>>> 19 System class >> _commitPrintingDiagnostics (envId 0) @2 line 8
>>>>> [GsNMethod 1700522241]
>>>>> 20 SystemCommitTransaction >> defaultAction (envId 0) @2 line 3
>>>>> [GsNMethod 7468825857]
>>>>> 21 AbstractException >> _signalWith: (envId 0) @5 line 25 [GsNMethod
>>>>> 1394122241]
>>>>> 22 AbstractException class >> signal (envId 0) @3 line 5 [GsNMethod
>>>>> 1172775681]
>>>>> 23 GRGemStonePlatform >> doCommitTransaction (envId 0) @4 line 3
>>>>> [GsNMethod 7469481473]
>>>>> 24 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @47 line 54
>>>>> [GsNMethod 10506395905]
>>>>> 25 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>>> 26 TransientRecursionLock >> critical: (envId 0) @11 line 12
>>>>> [GsNMethod 6527748609]
>>>>> 27 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>>>>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>>>>> 28 [] in GRGemStonePlatform >>
>>>>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>>>>> 10506396417]
>>>>> 29 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 30 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>>>>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>>>>> 31 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>>>>> 9828418049]
>>>>> 32 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>>>>> [GsNMethod 8795113729]
>>>>> 33 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 34 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>>>>> [GsNMethod 9828248321]
>>>>> 35 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod
>>>>> 8854227457]
>>>>> 36 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod
>>>>> 8854243329]
>>>>> 37 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod
>>>>> 8853951745]
>>>>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>>>>> 9557561601]
>>>>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>>>>> 9322044673]
>>>>> 41 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>>>>> 42 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>>>>> 9258731777]
>>>>> 43 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>>>>> 44 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod
>>>>> 8853958913]
>>>>> 45 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>>>>> 46 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>>>>> 9261209601]
>>>>> 47 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>>>>> 48 <Reenter marker>
>>>>> -----------
>>>>> _______________________________________________
>>>>> Glass mailing list
>>>>> Glass at lists.gemtalksystems.com
>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140527/0f8cc5f5/attachment-0001.html>
More information about the Glass
mailing list