[Glass] Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
Johan Brichau
johan at yesplan.be
Tue May 27 12:40:24 PDT 2014
Thanks Dale. This is really appreciated.
I'm a bit swamped until thursday but it's on my stack of things to test and I get back to that issue as well.
Johan
On 27 May 2014, at 21:38, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> ... and opened a GemStone Seaside3.1 issue[1]...
>
> Dale
>
> [1] https://github.com/glassdb/Seaside31/issues/20
>
>
> On Tue, May 27, 2014 at 12:18 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> I forgot to mention that internally we are tracking this as bug 44291...
>
> Dale
>
>
> On Tue, May 27, 2014 at 12:16 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> Okay, I've talked to engineering and they agree that the corruptObj error in this case is too extreme. We are in the final stages of a 3.2.1 release, so it may be too late to include a fix in 3.2.1, but it will most likely be included in a 3.2.2, but we don't have a schedule for that.
>
> Until then I think that the following patch to GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: is the best we can do:
>
>
> ----- Method: GRGemStonePlatform>>seasideProcessRequest:adaptor:resultBlock: (in category '*seaside-gemstone-core') -----seasideProcessRequest: aNativeRequest adaptor: adaptor resultBlock: resultBlock
> | result |
> [
> | retryCount result count |
> retryCount := self retryCount.
> count := 1.
> retryCount
> timesRepeat: [
> (result := self
> seasideProcessRequestWithRetry: aNativeRequest
> resultBlock: resultBlock) ~~ nil
> ifTrue: [ ^ result ].
> System _sessionCacheStatAt: 2 put: (System _sessionCacheStatAt: 2) + 1. "requests retried"
> (Delay forMilliseconds: self retryDelay * count) wait.
> count := count * 10 ]. "exceeded retry limit"
> ^ adaptor
> internalServerErrorMessage:
> 'Too many retries: ' , (retryCount + 1) printString ]
> on: Error
> do: [ :ex |
> self doAbortTransaction.
> + result := adaptor internalServerErrorMessage: ex description.
> + ex gsNumber == 2261
> + ifTrue:
> + [
> + "see discussion in: http://forum.world.st/Glass-Further-commits-have-been-disabled-for-this-session-because-CorruptObj-error-This-session-must-td4760409.html
> + "
> + Transcript
> + cr;
> + show:
> + 'Session terminating due to CorruptObj error ... session must logout'.
> + System logout ]
> + ex
> + return: nil "Do an explicit return. Because of the abort above, the default action for an exception (resume) is set. see bug39246." ].
> - result := adaptor internalServerErrorMessage: ex description. "Do an explicit return. Because of the abort above, the default action for an exception (resume) is set. see bug39246."
> - ex return: nil ].
> ^ result
>
> If you use this patch, make sure that you have a system in place for automatically restarting gems that get fatal errors. You should be using monit or daaemontools in production since gems may crash occasionally and they need to be restarted. Other than the extra time/cpu/disk for restarting a gem, stopping/restarting gems is not a fatal problem as no persistent data is lost or corrupted ... see my post on "GLASS 101: Disposable Gems, Durable Data"[1].
>
> I'm on the fence about including this patch in the standard release of Seaside for GemStone ... on the one hand if you get this particular error (or any other "commit prohibiting error" the gem is effectively out of service and needs to be restarted...
>
> An alternative to logout would be to update a field in SessionTemps that indicates the the gem needs to be restarted ... that session state can be queried by a monit http request (presumably) ... so at the end of the day we should probably invent some sort of gem termination policy that can be controlled at the application level...
>
> Dale
>
>
> [1] http://gemstonesoup.wordpress.com/2008/03/08/glass-101-disposable-gems-durable-data/
>
>
>
> On Mon, May 26, 2014 at 10:33 AM, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
> Johan,
>
> I think that a corruptObj error in this case may be a bit extreme. I have asked engineering if there is a "session safe" method for finding the obviously invalid utf8 ... Today is a holiday, so I may not get a response until tomorrow.
>
> I hope that crashed vms is an acceptable outcome of the security audit?
>
> The corruptObj error is our response to the error thrown by ICU ... the commit prohibiting error normally is thrown when we think that "memory stomping" has occurred and we want to avoid persisting potentially corrupt objects ... presumably the security audit folks knew that mishandling this particular ICU error condition could lead to a security breach:)
>
> Dale
>
>
> On Mon, May 26, 2014 at 9:44 AM, Johan Brichau <johan at yesplan.be> wrote:
> Hi,
>
> Today, we had a security audit on a Seaside 3.0.10 application running in a GS 3.1.0.5 stone with FastCGI behind nginx.
>
> I have no idea what exactly they did to obtain this, but the system went unresponsive after the following error until I restarted the gems.
> - a InternalError occurred (error 2261), The object with object ID 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
> - a TransactionError occurred (error 2249), Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
> - repeat last one
>
> The stacks from the gem log are attached.
>
> I am trying to trace what happened. Any clues?
>
> Johan
>
> ----------- Internal FASTCGI ERROR Encountered: 2014-05-26T13:37:41.31612992286682+02:00
> a InternalError occurred (error 2261), The object with object ID 'Hannes_Alfvén' is corrupt. Reason: 'carrysize > 0 at end of utf8 decode'
> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4 [GsNMethod 7469480705]
> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod 7469494785]
> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14 [GsNMethod 9828254465]
> 4 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod 10506396161]
> 5 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod 1394121473]
> 6 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1 [GsNMethod 1360964097]
> 7 String >> decodeFromUTF8 (envId 0) @1 line 1 [GsNMethod 1064104705]
> 8 UTF8PrimitiveEncoding >> decode: (envId 0) @2 line 4 [GsNMethod 7470277121]
> 9 GRUtf8GemStoneCodec >> decode: (envId 0) @3 line 3 [GsNMethod 7468617985]
> 10 [] in WAUrl >> decodedWith: (envId 0) @3 line 17 [GsNMethod 9230396417]
> 11 SequenceableCollection >> collect: (envId 0) @9 line 16 [GsNMethod 1064183041]
> 12 WAUrl >> decodedWith: (envId 0) @22 line 17 [GsNMethod 8789933313]
> 13 WAFastCGIRequestConverter >> requestUrlFor: (envId 0) @6 line 4 [GsNMethod 9828231425]
> 14 WAServerAdaptor >> requestFor: (envId 0) @3 line 6 [GsNMethod 8790261761]
> 15 WAFastCGIRequestConverter >> requestFor: (envId 0) @12 line 7 [GsNMethod 9828215809]
> 16 WAFastCGIAdaptor >> requestFor: (envId 0) @2 line 4 [GsNMethod 9828250881]
> 17 WAServerAdaptor >> contextFor: (envId 0) @2 line 5 [GsNMethod 8790264577]
> 18 WAServerAdaptor >> process: (envId 0) @2 line 5 [GsNMethod 8790258433]
> 19 [] in WAFastCGIAdaptor >> process: (envId 0) @2 line 6 [GsNMethod 8794996737]
> 20 [] in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: (envId 0) @2 line 13 [GsNMethod 10501067521]
> 21 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 22 [] in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: (envId 0) @14 line 14 [GsNMethod 10506395905]
> 23 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
> 24 TransientRecursionLock >> critical: (envId 0) @11 line 12 [GsNMethod 6527748609]
> 25 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: (envId 0) @3 line 5 [GsNMethod 10509761025]
> 26 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod 10506396417]
> 27 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 28 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @3 line 23 [GsNMethod 10509761281]
> 29 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod 9828418049]
> 30 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4 [GsNMethod 8795113729]
> 31 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 32 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5 [GsNMethod 9828248321]
> 33 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
> 34 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod 8854243329]
> 35 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod 8853951745]
> 36 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod 9557561601]
> 37 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 38 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod 9322044673]
> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 40 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod 9258731777]
> 41 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
> 42 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod 8853958913]
> 43 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
> 44 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod 9261209601]
> 45 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
> 46 <Reenter marker>
> -----------
> ----------- Internal FASTCGI LOG ENTRY: anArray-----------
> ----------- Internal FASTCGI ERROR Encountered: 2014-05-26T13:37:41.37823009490967+02:00
> a TransactionError occurred (error 2249), Further commits have been disabled for this session because: 'CorruptObj error'. This session must logout.
> 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4 [GsNMethod 7469480705]
> 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod 7469494785]
> 3 WAFastCGIAdaptor >> internalServerErrorMessage: (envId 0) @20 line 14 [GsNMethod 9828254465]
> 4 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @4 line 26 [GsNMethod 10506396161]
> 5 AbstractException >> _executeOuterHandler: (envId 0) @3 line 7 [GsNMethod 1394117633]
> 6 AbstractException >> _pass:with: (envId 0) @4 line 13 [GsNMethod 1393936385]
> 7 AbstractException >> pass (envId 0) @2 line 14 [GsNMethod 1393935361]
> 8 [] in System class >> _localCommit: (envId 0) @2 line 34 [GsNMethod 5944183041]
> 9 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod 1394121473]
> 10 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1 [GsNMethod 1360964097]
> 11 System class >> _primitiveCommit: (envId 0) @1 line 1 [GsNMethod 1789434881]
> 12 System class >> __commit: (envId 0) @2 line 8 [GsNMethod 1789345025]
> 13 [] in System class >> _localCommit: (envId 0) @2 line 30 [GsNMethod 5944183297]
> 14 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod 1064628225]
> 15 System class >> _localCommit: (envId 0) @8 line 31 [GsNMethod 1789345281]
> 16 TransactionBoundaryDefaultPolicy >> commit: (envId 0) @2 line 3 [GsNMethod 5986577665]
> 17 System class >> _commit: (envId 0) @7 line 16 [GsNMethod 1789345537]
> 18 System class >> commitTransaction (envId 0) @5 line 7 [GsNMethod 1789402113]
> 19 System class >> _commitPrintingDiagnostics (envId 0) @2 line 8 [GsNMethod 1700522241]
> 20 SystemCommitTransaction >> defaultAction (envId 0) @2 line 3 [GsNMethod 7468825857]
> 21 AbstractException >> _signalWith: (envId 0) @5 line 25 [GsNMethod 1394122241]
> 22 AbstractException class >> signal (envId 0) @3 line 5 [GsNMethod 1172775681]
> 23 GRGemStonePlatform >> doCommitTransaction (envId 0) @4 line 3 [GsNMethod 7469481473]
> 24 [] in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: (envId 0) @47 line 54 [GsNMethod 10506395905]
> 25 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
> 26 TransientRecursionLock >> critical: (envId 0) @11 line 12 [GsNMethod 6527748609]
> 27 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: (envId 0) @3 line 5 [GsNMethod 10509761025]
> 28 [] in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @6 line 15 [GsNMethod 10506396417]
> 29 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 30 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: (envId 0) @3 line 23 [GsNMethod 10509761281]
> 31 WAFastCGIAdaptor >> process: (envId 0) @3 line 4 [GsNMethod 9828418049]
> 32 [] in WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 4 [GsNMethod 8795113729]
> 33 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 34 WAFastCGIAdaptor >> answerResponderRole: (envId 0) @2 line 5 [GsNMethod 9828248321]
> 35 FSResponderRole >> answer (envId 0) @3 line 4 [GsNMethod 8854227457]
> 36 FSRole >> handleConnection (envId 0) @3 line 5 [GsNMethod 8854243329]
> 37 FSConnection >> unsafeServe (envId 0) @5 line 8 [GsNMethod 8853951745]
> 38 [] in FSConnection >> safeServe (envId 0) @2 line 8 [GsNMethod 9557561601]
> 39 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 40 [] in FSConnection >> safeServe (envId 0) @2 line 9 [GsNMethod 9322044673]
> 41 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 1064627457]
> 42 [] in FSConnection >> safeServe (envId 0) @2 line 12 [GsNMethod 9258731777]
> 43 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 1064640769]
> 44 FSConnection >> safeServe (envId 0) @2 line 15 [GsNMethod 8853958913]
> 45 FSConnection >> serve (envId 0) @2 line 4 [GsNMethod 8853957889]
> 46 [] in FSSocketServer >> listen: (envId 0) @3 line 15 [GsNMethod 9261209601]
> 47 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 1403422977]
> 48 <Reenter marker>
> -----------
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
>
>
More information about the Glass
mailing list