[Glass] fastcgi process unresponsive

[Otto Behrens]

Hi,

We've been running FastCGI on our production sites for about 3 weeks
now (GS 2.4.4.4). Periodically, one of the FCGI servers becomes
unhappy, but remains running.

The symptoms are that the topaz process is running, but the listener
on the socket is dead.

More details below, if you're interested. Any ideas will be appreciated.

Thanks
Otto

This is what I see in the logs:

-----------
failure
Read-Write Conflicts...
Write-Write Conflicts...
    20782838529
    .. bla, bla, list of about 30 oops ...
Write-Dependency Conflicts...
Write-ReadLock Conflicts...
Write-WriteLock Conflicts...
Rc-Write-Write Conflicts...
Synchronized-Commit Conflicts...
----------- Commit failure - retrying LOG ENTRY: aSymbolDictionary-----------

The oops are objects in our domain. So, I'm assuming that it may be 2
sessions trying to update the same objects.

Then we get

-----------------------------------------------------
GemStone: Error         Nonfatal
The object with object ID 81065894012259328 does not exist.
Error Category: 231169 [GemStone] Number: 2101 Arg Count: 1 Context :
19918721793
Arg 1: [648527152098074626 sz:0 cls: 74241 SmallInteger] 81065894012259328

Now executing the following command saved from "iferr 1":
   where
==> 1 WAComponent >> lightbox: @9 line 4   [GsMethod 3438252033]
2 PortfolioInformationContractableSectionRenderer >> overview @5 line
3   [GsMethod 350239489]

... and a long stack

149 GsProcess >> _start @1 line 9   [GsMethod 260584961]
  [GsProcess 19918721793]
topaz 1> [268 sz:0 cls: 68097 Boolean] true
topaz 1>
HostCallDebugger: invoked at: Mon Dec 30 11:02:55 SAST 2013

UTL_GUARANTEE failed, File
/export/toronto3/users/buildgss/244x-1/src/omverify.c line 28

Begin attempt to print C-level stack at: Mon Dec 30 11:02:55 SAST 2013


End of C-level stack:


hostcalldebugger invoked in process 90503, at 12/30/2013 11:02:55 AM.065 SAST
 notifying stone of fatal error

The stone log simply says:

--- 30/12/2013 11:02:55 SAST ---
    Fatal Internal Error Notification from Gem, user = DataCurator  Session = 9
      Gem hostName = localhost , Gem processId = 90503,
      current  CommitRecord = 74986