[Glass] Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
Dale Henrichs
dale.henrichs at gemtalksystems.com
Tue May 27 12:16:42 PDT 2014
Okay, I've talked to engineering and they agree that the corruptObj error
in this case is too extreme. We are in the final stages of a 3.2.1 release,
so it may be too late to include a fix in 3.2.1, but it will most likely be
included in a 3.2.2, but we don't have a schedule for that.
Until then I think that the following patch
to GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: is the
best we can do:
----- Method:
GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: (in category
'*seaside-gemstone-core') -----seasideProcessRequest: aNativeRequest
adaptor: adaptor resultBlock: resultBlock
| result |
[
| retryCount result count |
retryCount := self retryCount.
count := 1.
retryCount
timesRepeat: [
(result := self
seasideProcessRequestWithRetry: aNativeRequest
resultBlock: resultBlock) ~~ nil
ifTrue: [ ^ result ].
System _sessionCacheStatAt: 2 put: (System _sessionCacheStatAt: 2)
+ 1. "requests retried"
(Delay forMilliseconds: self retryDelay * count) wait.
count := count * 10 ]. "exceeded retry limit"
^ adaptor
internalServerErrorMessage:
'Too many retries: ' , (retryCount + 1) printString ]
on: Error
do: [ :ex |
self doAbortTransaction.
+ result := adaptor internalServerErrorMessage: ex description.
+ ex gsNumber == 2261
+ ifTrue:
+ [
+ "see discussion in:
http://forum.world.st/Glass-Further-commits-have-been-disabled-for-this-session-because-CorruptObj-error-This-session-must-td4760409.html
+ "
+ Transcript
+ cr;
+ show:
+ 'Session terminating due to CorruptObj error ... session
must logout'.
+ System logout ]
+ ex
+ return: nil "Do an explicit return. Because of the abort above,
the default action for an exception (resume) is set. see bug39246." ].
- result := adaptor internalServerErrorMessage: ex description. "Do
an explicit return. Because of the abort above, the default action for an
exception (resume) is set. see bug39246."
- ex return: nil ].
^ result
If you use this patch, make sure that you have a system in place for
automatically restarting gems that get fatal errors. You should be using
monit or daaemontools in production since gems may crash occasionally and
they need to be restarted. Other than the extra time/cpu/disk for
restarting a gem, stopping/restarting gems is not a fatal problem as no
persistent data is lost or corrupted ... see my post on "GLASS 101:
Disposable Gems, Durable Data"[1].
I'm on the fence about including this patch in the standard release of
Seaside for GemStone ... on the one hand if you get this particular error
(or any other "commit prohibiting error" the gem is effectively out of
service and needs to be restarted...
An alternative to logout would be to update a field in SessionTemps that
indicates the the gem needs to be restarted ... that session state can be
queried by a monit http request (presumably) ... so at the end of the day
we should probably invent some sort of gem termination policy that can be
controlled at the application level...
Dale
[1]
http://gemstonesoup.wordpress.com/2008/03/08/glass-101-disposable-gems-durable-data/
On Mon, May 26, 2014 at 10:33 AM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:
> Johan,
>
> I think that a corruptObj error in this case may be a bit extreme. I have
> asked engineering if there is a "session safe" method for finding the
> obviously invalid utf8 ... Today is a holiday, so I may not get a response
> until tomorrow.
>
> I hope that crashed vms is an acceptable outcome of the security audit?
>
> The corruptObj error is our response to the error thrown by ICU ... the
> commit prohibiting error normally is thrown when we think that "memory
> stomping" has occurred and we want to avoid persisting potentially corrupt
> objects ... presumably the security audit folks knew that mishandling this
> particular ICU error condition could lead to a security breach:)
>
> Dale
>
>
> On Mon, May 26, 2014 at 9:44 AM, Johan Brichau <johan at yesplan.be> wrote:
>
>> Hi,
>>
>> Today, we had a security audit on a Seaside 3.0.10 application running in
>> a GS 3.1.0.5 stone with FastCGI behind nginx.
>>
>> I have no idea what exactly they did to obtain this, but the system went
>> unresponsive after the following error until I restarted the gems.
>> - a InternalError occurred (error 2261), The object with object ID
>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>> - a TransactionError occurred (error 2249), Further commits have been
>> disabled for this session because: 'CorruptObj error'. This session must
>> logout.
>> - repeat last one
>>
>> The stacks from the gem log are attached.
>>
>> I am trying to trace what happened. Any clues?
>>
>> Johan
>>
>> ----------- Internal FASTCGI ERROR Encountered:
>> 2014-05-26T13:37:41.31612992286682+02:00
>> a InternalError occurred (error 2261), The object with object ID
>> 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4
>> [GsNMethod 7469480705]
>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod
>> 7469494785]
>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14
>> [GsNMethod 9828254465]
>> 4 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>> (envId 0) @4 line 26 [GsNMethod 10506396161]
>> 5 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod
>> 1394121473]
>> 6 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>> [GsNMethod 1360964097]
>> 7 String >> decodeFromUTF8 (envId 0) @1 line 1 [GsNMethod 1064104705]
>> 8 UTF8PrimitiveEncoding >> decode: (envId 0) @2 line 4 [GsNMethod
>> 7470277121]
>> 9 GRUtf8GemStoneCodec >> decode: (envId 0) @3 line 3 [GsNMethod
>> 7468617985]
>> 10 [] in WAUrl >> decodedWith: (envId 0) @3 line 17 [GsNMethod
>> 9230396417]
>> 11 SequenceableCollection >> collect: (envId 0) @9 line 16 [GsNMethod
>> 1064183041]
>> 12 WAUrl >> decodedWith: (envId 0) @22 line 17 [GsNMethod 8789933313]
>> 13 WAFastCGIRequestConverter >> requestUrlFor: (envId 0) @6 line 4
>> [GsNMethod 9828231425]
>> 14 WAServerAdaptor >> requestFor: (envId 0) @3 line 6 [GsNMethod
>> 8790261761]
>> 15 WAFastCGIRequestConverter >> requestFor: (envId 0) @12 line 7
>> [GsNMethod 9828215809]
>> 16 WAFastCGIAdaptor >> requestFor: (envId 0) @2 line 4 [GsNMethod
>> 9828250881]
>> 17 WAServerAdaptor >> contextFor: (envId 0) @2 line 5 [GsNMethod
>> 8790264577]
>> 18 WAServerAdaptor >> process: (envId 0) @2 line 5 [GsNMethod 8790258433]
>> 19 [] in WAFastCGIAdaptor >> process: (envId 0) @2 line 6 [GsNMethod
>> 8794996737]
>> 20 [] in GRGemStonePlatform >>
>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @2 line 13
>> [GsNMethod 10501067521]
>> 21 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 22 [] in GRGemStonePlatform >>
>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @14 line 14
>> [GsNMethod 10506395905]
>> 23 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>> 24 TransientRecursionLock >> critical: (envId 0) @11 line 12 [GsNMethod
>> 6527748609]
>> 25 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>> 26 [] in GRGemStonePlatform >>
>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>> 10506396417]
>> 27 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 28 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>> 29 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>> 9828418049]
>> 30 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>> [GsNMethod 8795113729]
>> 31 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 32 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>> [GsNMethod 9828248321]
>> 33 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
>> 34 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod 8854243329]
>> 35 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod 8853951745]
>> 36 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>> 9557561601]
>> 37 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>> 9322044673]
>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>> 9258731777]
>> 41 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>> 42 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod 8853958913]
>> 43 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>> 44 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>> 9261209601]
>> 45 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>> 46 <Reenter marker>
>> -----------
>> ----------- Internal FASTCGI LOG ENTRY: anArray-----------
>> ----------- Internal FASTCGI ERROR Encountered:
>> 2014-05-26T13:37:41.37823009490967+02:00
>> a TransactionError occurred (error 2249), Further commits have been
>> disabled for this session because: 'CorruptObj error'. This session must
>> logout.
>> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4
>> [GsNMethod 7469480705]
>> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod
>> 7469494785]
>> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14
>> [GsNMethod 9828254465]
>> 4 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>> (envId 0) @4 line 26 [GsNMethod 10506396161]
>> 5 AbstractException >> _executeOuterHandler: (envId 0) @3 line 7
>> [GsNMethod 1394117633]
>> 6 AbstractException >> _pass:with: (envId 0) @4 line 13 [GsNMethod
>> 1393936385]
>> 7 AbstractException >> pass (envId 0) @2 line 14 [GsNMethod 1393935361]
>> 8 [] in System class >> _localCommit: (envId 0) @2 line 34 [GsNMethod
>> 5944183041]
>> 9 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod
>> 1394121473]
>> 10 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1
>> [GsNMethod 1360964097]
>> 11 System class >> _primitiveCommit: (envId 0) @1 line 1 [GsNMethod
>> 1789434881]
>> 12 System class >> __commit: (envId 0) @2 line 8 [GsNMethod 1789345025]
>> 13 [] in System class >> _localCommit: (envId 0) @2 line 30 [GsNMethod
>> 5944183297]
>> 14 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod
>> 1064628225]
>> 15 System class >> _localCommit: (envId 0) @8 line 31 [GsNMethod
>> 1789345281]
>> 16 TransactionBoundaryDefaultPolicy >> commit: (envId 0) @2 line 3
>> [GsNMethod 5986577665]
>> 17 System class >> _commit: (envId 0) @7 line 16 [GsNMethod 1789345537]
>> 18 System class >> commitTransaction (envId 0) @5 line 7 [GsNMethod
>> 1789402113]
>> 19 System class >> _commitPrintingDiagnostics (envId 0) @2 line 8
>> [GsNMethod 1700522241]
>> 20 SystemCommitTransaction >> defaultAction (envId 0) @2 line 3
>> [GsNMethod 7468825857]
>> 21 AbstractException >> _signalWith: (envId 0) @5 line 25 [GsNMethod
>> 1394122241]
>> 22 AbstractException class >> signal (envId 0) @3 line 5 [GsNMethod
>> 1172775681]
>> 23 GRGemStonePlatform >> doCommitTransaction (envId 0) @4 line 3
>> [GsNMethod 7469481473]
>> 24 [] in GRGemStonePlatform >>
>> seasideProcessRequestWithRetry:resultBlock: (envId 0) @47 line 54
>> [GsNMethod 10506395905]
>> 25 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>> 26 TransientRecursionLock >> critical: (envId 0) @11 line 12 [GsNMethod
>> 6527748609]
>> 27 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
>> (envId 0) @3 line 5 [GsNMethod 10509761025]
>> 28 [] in GRGemStonePlatform >>
>> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod
>> 10506396417]
>> 29 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 30 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
>> (envId 0) @3 line 23 [GsNMethod 10509761281]
>> 31 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod
>> 9828418049]
>> 32 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4
>> [GsNMethod 8795113729]
>> 33 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 34 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5
>> [GsNMethod 9828248321]
>> 35 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
>> 36 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod 8854243329]
>> 37 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod 8853951745]
>> 38 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod
>> 9557561601]
>> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 40 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod
>> 9322044673]
>> 41 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
>> 42 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod
>> 9258731777]
>> 43 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
>> 44 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod 8853958913]
>> 45 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
>> 46 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod
>> 9261209601]
>> 47 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
>> 48 <Reenter marker>
>> -----------
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140527/98a40261/attachment-0001.html>
More information about the Glass
mailing list