[Glass] [3.1.0.4] Tranlog Restore Seems to Hang

[Ken Treis]

We had a cloud server crash (problem on the underlying hardware), and after things were fixed I find myself unable to restore certain transaction logs. It goes something like this:

1. startstone -N -R seaside.
2. topaz -l 

> successful login
> topaz 1> run
> SystemRepository restoreStatus
> %
> Restoring from transaction log files, restored to 20/08/13 18:26:21 UTC, file 926 record 1596051, nextFileId 926, oldest fileId 926
> topaz 1> run
> SystemRepository restoreFromCurrentLogs
> %

3. In the stone log, I see:

--- 08/22/13 23:56:27 UTC ---
    Opened a transaction log file for read_nolocks. 
       filename = /opt/gemstone/GemStone64Bit3.1.0.4-x86_64.Linux//seaside/data/tranlog926.dbf
    Restoring from current log directory to end of logs

and then nothing else happens. After an hour, I get impatient. Load average is low, and the only process using any appreciable CPU is pgsvrmain. So I kill pgsvrmain and let the stone crash.

4. I repeat 1 and 2, and restoreStatus shows me that no progress has been made. I'm still at the same record (1596051).

I've seen this happen in 3 different scenarios:

(A) After restoring a Smalltalk full backup.
(B) After restoring a filesystem backup.
(C) During normal startup after the crash, when there are just a couple of records to replay since the pre-crash checkpoint. I didn't confirm that the record ID wasn't changing at this point, but the behavior was identical to the other two).

All three hang at different record IDs, so I can't blame a single bad record. Also, in (A) both page and object audits came back OK. I haven't done audits on the other scenario.

Is this a known issue in 3.1.0.4? Anything else I should be checking/trying?

--
Ken Treis
Miriam Technologies, Inc.
(866) 652-2040 x221

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20130822/07f581a2/attachment.html>