[Glass] fastcgi process unresponsive

[Dale K. Henrichs]

Otto,

That looks similar to a problem that showed up in 3.x ... the corruption was happening in tempobj space, so no persistent corruption was occuring ... probably should do an object audit to confirm that the corruption is localized to the vm ...

I'll check with the vm guys on monday and see what they say ...

The fact that the vm remains running should probably be tracked down ...

Dale

----- Original Message -----
| From: "Otto Behrens" <otto at finworks.biz>
| To: glass at lists.gemtalksystems.com
| Cc: support at finworks.biz
| Sent: Monday, December 30, 2013 2:41:56 AM
| Subject: [Glass] fastcgi process unresponsive
| 
| Hi,
| 
| We've been running FastCGI on our production sites for about 3 weeks
| now (GS 2.4.4.4). Periodically, one of the FCGI servers becomes
| unhappy, but remains running.
| 
| The symptoms are that the topaz process is running, but the listener
| on the socket is dead.
| 
| More details below, if you're interested. Any ideas will be
| appreciated.
| 
| Thanks
| Otto
| 
| This is what I see in the logs:
| 
| -----------
| failure
| Read-Write Conflicts...
| Write-Write Conflicts...
|     20782838529
|     .. bla, bla, list of about 30 oops ...
| Write-Dependency Conflicts...
| Write-ReadLock Conflicts...
| Write-WriteLock Conflicts...
| Rc-Write-Write Conflicts...
| Synchronized-Commit Conflicts...
| ----------- Commit failure - retrying LOG ENTRY:
| aSymbolDictionary-----------
| 
| The oops are objects in our domain. So, I'm assuming that it may be 2
| sessions trying to update the same objects.
| 
| Then we get
| 
| -----------------------------------------------------
| GemStone: Error         Nonfatal
| The object with object ID 81065894012259328 does not exist.
| Error Category: 231169 [GemStone] Number: 2101 Arg Count: 1 Context :
| 19918721793
| Arg 1: [648527152098074626 sz:0 cls: 74241 SmallInteger]
| 81065894012259328
| 
| Now executing the following command saved from "iferr 1":
|    where
| ==> 1 WAComponent >> lightbox: @9 line 4   [GsMethod 3438252033]
| 2 PortfolioInformationContractableSectionRenderer >> overview @5 line
| 3   [GsMethod 350239489]
| 
| ... and a long stack
| 
| 149 GsProcess >> _start @1 line 9   [GsMethod 260584961]
|   [GsProcess 19918721793]
| topaz 1> [268 sz:0 cls: 68097 Boolean] true
| topaz 1>
| HostCallDebugger: invoked at: Mon Dec 30 11:02:55 SAST 2013
| 
| UTL_GUARANTEE failed, File
| /export/toronto3/users/buildgss/244x-1/src/omverify.c line 28
| 
| Begin attempt to print C-level stack at: Mon Dec 30 11:02:55 SAST
| 2013
| 
| 
| End of C-level stack:
| 
| 
| hostcalldebugger invoked in process 90503, at 12/30/2013 11:02:55
| AM.065 SAST
|  notifying stone of fatal error
| 
| The stone log simply says:
| 
| --- 30/12/2013 11:02:55 SAST ---
|     Fatal Internal Error Notification from Gem, user = DataCurator
|      Session = 9
|       Gem hostName = localhost , Gem processId = 90503,
|       current  CommitRecord = 74986
| _______________________________________________
| Glass mailing list
| Glass at lists.gemtalksystems.com
| http://lists.gemtalksystems.com/mailman/listinfo/glass
|