[Glass] Problems with database, object corruption
Bill Erickson
bill.erickson at gemtalksystems.com
Mon Jan 27 11:46:33 PST 2014
John,
Generally, when you see a page audit failure your DB is pretty hosed, and
as you ultimately discovered it usually not worth the effort of trying to
fix -- you'll need to restore from the closest backup and restore from
tranlogs.
In addition, remember that "repair" only replaces references to
non-existent oops with nil -- which "fixes" the DB and makes things
consistent, but depending on where the "repair" is done, will not be
logically- or application-correct. This would be especially true for
complex objects like RcQueues. You will usually need to analyze the
objects "repaired" in this way and determine what *really* should be there
(if possible). For application data that might be possible, but for
complex GS objects, you're not likely to know how to do it. Again, you are
better off just going back to your last backup and restoring / replaying
tranlogs.
I noticed you didn't attempt to replay tranlogs after the restore -- why
not? You might not have lost *any* data by replaying tranlogs. Most types
of corruption of this form are not part of the tranlog sequence, so it
should be safe to replay tranlogs.
While it might not be possible to determine the root cause of the
corruption, if you're up to the task we could do a little looking around..
First off, determine what tranlog was in effect at the time you first
noticed the problem. Call it tranlogZ.dbf. Next determine the tranlog in
effect at your last clean object audit (you do do these regularly, right?
-- if not, we'll just pick a date to begin the analysis from -- say 1 week
ago). Refer to your stone logs to determine the tranlog -- call it
tranlogX.dbf.
Now, place copies of tranlogX.dbf through tranlogZ.dbf into a single
directory. CD to that directory.
I'd like you to then search these logs for a few representative oops:
70878209 -- oop of one of the bad page audit guys that's in the free oop
list and dead list.
34149121 -- one of the non-existent objects
635987969 -- object that refers to the non-existent object
Execute the following from unix on each one of these: (replace "oop" with
the oop number above)
unix> $GEMSTONE/bin/searchlogs.sh oop > oop.log
Post your resulting logs and let me take a look....
------------------------------------------------------------------------
Bill Erickson
GemTalk Systems Engineering
15220 NW Greenbrier Parkway #240, Beaverton OR 97006
------------------------------------------------------------------------
On Mon, Jan 27, 2014 at 11:19 AM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:
> John,
>
> This is interesting...to say the least, we're going to look into this a
> bit more internally and get back to you.
>
> Thanks for the detailed report,
>
> Dale
>
>
> On Fri, Jan 24, 2014 at 9:16 PM, John McIntosh <
> johnmci at smalltalkconsulting.com> wrote:
>
>> Earlier this evening I wanted to check my Object log for errors, and I
>> got a walk back saying there was references in a RcQueue to non existent
>> objects...
>>
>>
>> End result was taking the database down, removing the extent and tran log
>> files then restart with the base seaside extent and doing a restore to a
>> weekend backup. Fortunately my data model tolerates loss of data so going
>> back to last weekends backup wasn't a concern, and it seemed clean after
>> the restore.
>>
>>
>> topaz 1> printit
>>
>> SystemRepository objectAudit
>>
>> %
>>
>>
>> Object Audit: Audit successfully completed; no errors were detected.
>>
>> true
>>
>> topaz 1> exit
>>
>> Logging out session 1.
>>
>>
>>
>> However how I got there was...
>>
>>
>> Earlier
>>
>> Take production database down.
>>
>> run a page audit
>>
>>
>> pageaudit seaside
>>
>> pageaudit[Info]: GemStone version '3.1.0.5'
>>
>>
>> _____________________________________________________________________________
>>
>> | GemStone/S64 Object-Oriented Data Management System
>> |
>>
>> | Copyright (C) GemTalk Systems 1986-2013
>> |
>>
>> | All rights reserved.
>> |
>>
>>
>> +-----------------------------------------------------------------------------+
>>
>> | PROGRAM: STONE, Stone Repository Monitor
>> |
>>
>> | VERSION: 3.1.0.5, Thu Nov 14 13:28:38 2013
>> |
>>
>> | BUILD: gss64_3_1_0_x_branch-31878
>> |
>>
>> | BUILT FOR: x86-64 (Linux)
>> |
>>
>> | MODE: 64 bit
>> |
>>
>> | RUNNING ON: 1-CPU domU-12-31-39-0A-60-D9 x86_64 (Linux 3.2.0-58-virtual
>> #88-Ubuntu
>>
>> | SMP Tue Dec 3 17:58:13 UTC 2013) 1652MB
>> |
>>
>> | PROCESS ID: 16715 DATE: 01/25/14 03:43:14 UTC
>> |
>>
>> | USER IDS: REAL=ubuntu (1000) EFFECTIVE=ubuntu (1000)
>> |
>>
>> | COMMAND: /opt/gemstone/product/sys/stoned -A seaside
>> |
>>
>>
>> +-----------------------------------------------------------------------------+
>>
>>
>> _____________________________________________________________________________
>>
>> | Configuration Files
>> |
>>
>> |
>> |
>>
>> | System File:
>> /opt/gemstone/GemStone64Bit3.1.0.5-x86_64.Linux/seaside/data/system.conf
>>
>> |
>> |
>>
>> | Executable File:
>> /opt/gemstone/GemStone64Bit3.1.0.5-x86_64.Linux/seaside/data/seaside-audit.conf
>>
>> | Warning: File not found (errno=2,ENOENT, The file or directory
>> specified cannot
>>
>> | be found)
>> |
>>
>> | using defaults.
>> |
>>
>>
>> |_____________________________________________________________________________|
>>
>>
>> _____________________________________________________________________________
>>
>> | Non-Default Configuration Options for Stone
>> |
>>
>>
>> |_____________________________________________________________________________|
>>
>>
>> DBF_EXTENT_NAMES = "$GEMSTONE_DATADIR/extent0.dbf";
>>
>> DBF_SCRATCH_DIR = "$GEMSTONE_DATADIR/";
>>
>> KEYFILE = "$GEMSTONE_KEYFILE";
>>
>> SHR_PAGE_CACHE_SIZE_KB = 800000;
>>
>> STN_EPOCH_GC_ENABLED = TRUE;
>>
>> STN_HALT_ON_FATAL_ERR = FALSE;
>>
>> STN_TRAN_FULL_LOGGING = TRUE;
>>
>> STN_TRAN_LOG_DIRECTORIES = "$GEMSTONE_DATADIR/",
>>
>> "$GEMSTONE_DATADIR/";
>>
>> STN_TRAN_LOG_SIZES = 1000,
>>
>> 1000;
>>
>>
>> _____________________________________________________________________________
>>
>> | All Configuration Options for Stone
>> |
>>
>>
>> |_____________________________________________________________________________|
>>
>>
>> DBF_ALLOCATION_MODE = "SEQUENTIAL";
>>
>> DBF_EXTENT_NAMES = "$GEMSTONE_DATADIR/extent0.dbf";
>>
>> DBF_EXTENT_SIZES = "";
>>
>> DBF_PRE_GROW = FALSE;
>>
>> DBF_SCRATCH_DIR = "$GEMSTONE_DATADIR/";
>>
>> DUMP_OPTIONS = TRUE;
>>
>> KEYFILE = "$GEMSTONE_KEYFILE";
>>
>> LOG_WARNINGS = TRUE;
>>
>> SHR_NUM_FREE_FRAME_SERVERS = 1;
>>
>> SHR_PAGE_CACHE_LOCKED = FALSE;
>>
>> SHR_PAGE_CACHE_NUM_PROCS = 49;
>>
>> SHR_PAGE_CACHE_NUM_SHARED_COUNTERS = 1900;
>>
>> SHR_PAGE_CACHE_PERMISSIONS = 660;
>>
>> SHR_PAGE_CACHE_SIZE_KB = 800000;
>>
>> SHR_SPIN_LOCK_COUNT = 1;
>>
>> SHR_TARGET_FREE_FRAME_COUNT = -1;
>>
>> SHR_WELL_KNOWN_PORT_NUMBER = 0;
>>
>> STN_ADMIN_GC_SESSION_ENABLED = TRUE;
>>
>> STN_AIO_WAIT_TIME = 25;
>>
>> STN_ALLOCATE_HIGH_OOPS = 0;
>>
>> STN_CACHE_WARMER = 0;
>>
>> STN_CACHE_WARMER_SESSIONS = 0;
>>
>> STN_CHECKPOINT_INTERVAL = 300;
>>
>> STN_COMMITS_ASYNC = FALSE;
>>
>> STN_COMMIT_QUEUE_THRESHOLD = -1;
>>
>> STN_COMMIT_RECORD_QUEUE_SIZE = 40;
>>
>> STN_COMMIT_TOKEN_TIMEOUT = 0;
>>
>> STN_CR_BACKLOG_THRESHOLD = 80;
>>
>> STN_DISABLE_LOGIN_FAILURE_LIMIT = 15;
>>
>> STN_DISABLE_LOGIN_FAILURE_TIME_LIMIT = 15;
>>
>> STN_DISKFULL_TERMINATION_INTERVAL = 3;
>>
>> STN_EPOCH_GC_ENABLED = TRUE;
>>
>> STN_EXTENT_IO_FLAGS = 0;
>>
>> STN_FREE_FRAME_CACHE_SIZE = -1;
>>
>> STN_FREE_SPACE_THRESHOLD = 0;
>>
>> STN_GEM_ABORT_TIMEOUT = 1;
>>
>> STN_GEM_LOSTOT_TIMEOUT = 60;
>>
>> STN_GEM_TIMEOUT = 0;
>>
>> STN_HALT_ON_FATAL_ERR = FALSE;
>>
>> STN_LISTENING_ADDRESSES = "::";
>>
>> STN_LOG_IO_FLAGS = 0;
>>
>> STN_LOG_LOGIN_FAILURE_LIMIT = 10;
>>
>> STN_LOG_LOGIN_FAILURE_TIME_LIMIT = 10;
>>
>> STN_LOOP_NO_WORK_THRESHOLD = 0;
>>
>> STN_MAX_AIO_RATE = 3000;
>>
>> STN_MAX_AIO_REQUESTS = 128;
>>
>> STN_MAX_REMOTE_CACHES = 255;
>>
>> STN_MAX_SESSIONS = 40;
>>
>> STN_MAX_SLEEP_TIME = 1000;
>>
>> STN_MAX_VOTING_SESSIONS = 100;
>>
>> STN_NUM_AIO_WRITE_THREADS = 4;
>>
>> STN_NUM_GC_RECLAIM_SESSIONS = 1;
>>
>> STN_NUM_LOCAL_AIO_SERVERS = 1;
>>
>> STN_OBJ_LOCK_TIMEOUT = 0;
>>
>> STN_PRIVATE_PAGE_CACHE_KB = 40000;
>>
>> STN_PAGE_MGR_COMPRESSION_ENABLED = FALSE;
>>
>> STN_PAGE_MGR_MAX_WAIT_TIME = 200;
>>
>> STN_PAGE_MGR_PRINT_TIMEOUT_THRESHOLD = 5;
>>
>> STN_PAGE_MGR_REMOVE_MIN_PAGES = 40;
>>
>> STN_PAGE_MGR_REMOVE_MAX_PAGES = 16384;
>>
>> STN_REMOTE_CACHE_PGSVR_TIMEOUT = 15;
>>
>> STN_REMOTE_CACHE_TIMEOUT = 5;
>>
>> STN_SHR_TARGET_PERCENT_DIRTY = 20;
>>
>> STN_SIGNAL_ABORT_CR_BACKLOG = 20;
>>
>> STN_SYMBOL_GC_ENABLED = FALSE;
>>
>> STN_TRAN_FULL_LOGGING = TRUE;
>>
>> STN_TRAN_LOG_DEBUG_LEVEL = 0;
>>
>> STN_TRAN_LOG_DIRECTORIES = "$GEMSTONE_DATADIR/",
>>
>> "$GEMSTONE_DATADIR/";
>>
>> STN_TRAN_LOG_LIMIT = 1000;
>>
>> STN_TRAN_LOG_PREFIX = "tranlog";
>>
>> STN_TRAN_LOG_SIZES = 1000,
>>
>> 1000;
>>
>> STN_TRAN_Q_TO_RUN_Q_THRESHOLD = 6;
>>
>> STN_WELL_KNOWN_PORT_NUMBER = 0;
>>
>>
>> --- 01/25/14 03:43:14 UTC ---
>>
>>
>> _____________________________________________________________________________
>>
>> | Key file:
>> /opt/gemstone/GemStone64Bit3.1.0.5-x86_64.Linux/seaside/etc/gemstone.key
>>
>> | GemStone ID: Free GS/S Web Edition
>> |
>>
>> | LICENSED TO: GemStone Seaside Community
>> |
>>
>> | KEY ORIGIN: 0x5e0a1e8d (a Linux x86)
>> |
>>
>>
>> |_____________________________________________________________________________|
>>
>>
>> Attaching the Shared Cache using Stone name: seaside-audit
>>
>> Successfully started 1 free frame page servers.
>>
>>
>> Extent #0
>>
>> -----------
>>
>> Filename =
>> !#dbf!/opt/gemstone/GemStone64Bit3.1.0.5-x86_64.Linux/seaside/data/extent0.dbf
>>
>> Maximum size = NONE
>>
>> File size = 3156 Mbytes = 201984 pages
>>
>> Space available = 2340 Mbytes = 149797 pages
>>
>>
>> Totals
>>
>> ------
>>
>> Repository Size = 3156 Mbytes = 201984 pages
>>
>> Free Space = 2340 Mbytes = 149797 pages
>>
>> ---------------------------------------------------
>>
>>
>> GemStone is starting a page audit of the Repository.
>>
>>
>> Finished auditing reserved pages in extent 0.
>>
>>
>> Finished pages past end
>>
>>
>> Begin auditing current checkpoint.
>>
>>
>> Finished auditing checkpoint bitmaps.
>>
>>
>> Finished auditing scavengable pages.
>>
>>
>> Start auditing object table pages.
>>
>>
>> --- 01/25/14 03:43:30 UTC ---
>>
>> Finished auditing object table pages.
>>
>> Found 441 oops in OT free oops and also in rootPage.deadObjs
>>
>> START root 17288672 level 0 kind 0 preGrown 0, printing oops
>>
>> BitList pageId 17288672 beginId:(0 0) count 441 firstOfs 0
>>
>> 70878209 70878465 70878721 70878977 70887937 70888193 70888449
>>
>> 70888961 70889217 70889473 70889729 70889985 70890753 70891009
>>
>> 70891777 70968577 70969601 70969857 70970113 70970625 70974977
>>
>> 70986241 71021313 71032577 71042817 71047425 71060737 71076609
>>
>> 71084033 71088897 71101441 71174657 71174913 71175937 71402241
>>
>> 71452929 71467009 71473921 71474945 71481857 71510017 71528449
>>
>> 71714817 71774721 71775233 71807233 71833345 71907073 71907329
>>
>> 71907585 71909377 71981825 72098049 72126977 72132353 72132609
>>
>> 72189697 72190209 72202497 72202753 72203265 72214017 72216321
>>
>> 72229633 72231169 72231425 72242177 72292609 72425217 72425473
>>
>> 72499713 72536577 72536833 72537089 72538881 72545025 72545537
>>
>> 72548609 72558081 72569089 72569601 72572417 72573185 72574465
>>
>> 72574977 72577025 72577537 72580865 72581633 72581889 72582401
>>
>> 72582657 72582913 72583169 72583425 72583681 72583937 72584193
>>
>> 72584449 72585217 72585473 72585729 72585985 72586241 72586497
>>
>> 72586753 72587009 72587265 72587521 72587777 72588033 72588289
>>
>> 72588545 72589313 72589569 72591361 72591617 72591873 72592129
>>
>> 72592385 72592641 72592897 72593153 72593409 72593665 72593921
>>
>> 72594177 72594433 72599297 72599553 72599809 72600065 72600321
>>
>> 72600577 72600833 72601089 72602881 72610049 72612353 72612609
>>
>> 72612865 72613121 72613377 72613633 72613889 72614145 72614401
>>
>> 72614657 72614913 72615425 72615681 72615937 72616193 72616449
>>
>> 72616705 72616961 72624897 72626177 72626433 72626945 72627201
>>
>> 72627457 72631809 72632065 72632321 72632577 72632833 72633089
>>
>> 72633345 72633601 72634881 72642305 72643585 72644353 72644609
>>
>> 72644865 72645121 72645377 72645633 72645889 72646145 72646913
>>
>> 72648961 72649217 72649473 72649729 72649985 72650241 72657409
>>
>> 72657665 72657921 72658177 72658433 72658689 72658945 72659201
>>
>> 72659457 72659713 72659969 72660225 72660481 72660737 72663041
>>
>> 72663297 72663553 72663809 72664065 72664321 72664577 72664833
>>
>> 72665089 72665345 72665601 72665857 72666113 72666369 72666625
>>
>> 72666881 72672001 72672257 72672513 72672769 72673025 72673281
>>
>> 72673537 72673793 72674561 72674817 72675329 72675585 72675841
>>
>> 72676097 72676353 72689409 72689665 72689921 72690177 72690433
>>
>> 72690689 72691201 72691457 72691969 72692225 72692481 72692737
>>
>> 72692993 72693249 72693505 72693761 103643649 103644161 103644673
>>
>> 103645185 103645697 103646209 103646721 103647233 103647745
>> 103648257
>>
>> 103648769 103649281 103650305 103653889 103681537 103682049
>> 103682561
>>
>> 103683073 103683585 103684097 103684609 103685121 103685633
>> 103686145
>>
>> 103686657 103687169 103687681 103688193 103688705 103689217
>> 103689729
>>
>> 103690241 103691777 103692289 103695873 103696385 103696897
>> 103697409
>>
>> 103697921 103698433 103698945 103699457 103699969 103700481
>> 103700993
>>
>> 103701505 103702017 103702529 103703041 103703553 103704065
>> 103704577
>>
>> 103705089 103705601 103706113 103706625 103707137 103707649
>> 103708161
>>
>> 103708673 103709185 103709697 103710209 103710721 103711233
>> 103729665
>>
>> 103730177 103731201 103731713 103732225 103732737 103733249
>> 103733761
>>
>> 103734273 103734785 103735297 103735809 103736321 103736833
>> 103737857
>>
>> 103738369 103738881 103739393 103739905 103740417 103740929
>> 103741441
>>
>> 103741953 103742465 103742977 103743489 103744513 103745025
>> 103745537
>>
>> 103746049 103746561 103747073 126834945 126835201 126835457
>> 126835713
>>
>> 126835969 126836225 126836481 126836737 126836993 126837249
>> 126837505
>>
>> 126837761 126838017 126838273 126838529 126838785 126839041
>> 126839297
>>
>> 126839553 126839809 126840065 126840321 126974209 152136449
>> 152136705
>>
>> 152136961 152142593 161810689 161810945 161817857 161819649
>> 161841153
>>
>> 161841921 161850113 161951489 161951745 162097921 162275073
>> 162786305
>>
>> 163019265 163019521 163270145 163632641 164219649 164219905
>> 164220161
>>
>> 164220417 164221185 164221441 164221953 164222209 164222465
>> 164223745
>>
>> 164224257 164224769 164225025 164225281 164225537 164225793
>> 164226049
>>
>> 164226305 164226561 164226817 164245761 164742913 164791297
>> 164791553
>>
>> 164791809 164792065 164792321 164792577 164793345 165060865
>> 165061121
>>
>> 165069057 165093633 165214465 165221889 165222145 165222401
>> 165222657
>>
>> 165222913 165223169 165223425 165224449 165224705 165224961
>> 165225217
>>
>> 165225729 165225985 165226241 165226497 165226753 165227009
>> 165227265
>>
>> END root 17288672 level 0 numBitsSet = 441
>>
>>
>> Finished auditing commit records.
>>
>>
>> Finished auditing alloc pages shadowed.
>>
>>
>>
>> Begin auditing data pages. Count=45232
>>
>>
>> --- 01/25/14 03:43:54 UTC ---
>>
>> Finished auditing data pages
>>
>>
>> PAGE AUDIT STATISTICS domU-12-31-39-0A-60-D9 x86_64 (Linux
>> 3.2.0-58-virtual #88-Ubuntu SMP Tue Dec 3 17:58:13 UTC 2013) - 01/25/14
>> 03:43:54 UTC
>>
>>
>> 16384 bytes = 1 GemStone Page
>>
>> 1048576 bytes = 1 Mbytes
>>
>> Repository Size 3156 Mbytes
>>
>> Data Pages 706 Mbytes
>>
>> Meta Information Pages 91 Mbytes
>>
>> Shadow Pages 17 Mbytes
>>
>> Free Space in Repository 2340 Mbytes
>>
>> **** Number of differences found in page allocation = 0
>>
>> Page Audit of Repository completed successfully.
>>
>>
>>
>>
>> -----------------------------------------------------------------------------
>>
>>
>> Run a Object Audit
>>
>>
>>
>> topaz 1> printit
>>
>> SystemRepository objectAudit
>>
>> %
>>
>> In object [635987969] of class WARcLastAccessEntry [27309569], at 1 based
>> offset 2, references non existent object [34149121]
>>
>> In object [1631238657] of class RcCollisionBucket [114689], at 1 based
>> offset 3, references non existent object [34149633]
>>
>> In object [1631246337] of class RcCollisionBucket [114689], at 1 based
>> offset 6, references non existent object [34149633]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [149999361]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 2,
>> references non existent object [149999105]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 3,
>> references non existent object [149997313]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 4,
>> references non existent object [149997057]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 5,
>> references non existent object [149996801]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [150003969]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 2,
>> references non existent object [150003457]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 3,
>> references non existent object [150003201]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 4,
>> references non existent object [150002945]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 5,
>> references non existent object [149999617]
>>
>> In object [1546009345] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [150012929]
>>
>> In object [1546009345] of class RcCounter [109313], at 1 based offset 2,
>> references non existent object [150012673]
>>
>> In object [1546009345] of class RcCounter [109313], at 1 based offset 3,
>> references non existent object [150011393]
>>
>> In object [1546009345] of class RcCounter [109313], at 1 based offset 4,
>> references non existent object [150010881]
>>
>> In object [1546009345] of class RcCounter [109313], at 1 based offset 5,
>> references non existent object [150008577]
>>
>> In object [1546009601] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [150007041]
>>
>> In object [1546009601] of class RcCounter [109313], at 1 based offset 2,
>> references non existent object [150006017]
>>
>> In object [1546009601] of class RcCounter [109313], at 1 based offset 3,
>> references non existent object [150005761]
>>
>> In object [1546009601] of class RcCounter [109313], at 1 based offset 4,
>> references non existent object [150005505]
>>
>> In object [1546009601] of class RcCounter [109313], at 1 based offset 5,
>> references non existent object [150004481]
>>
>> In object [1567151105] of class WARequestFields [27325441], at 1 based
>> offset 3, references non existent object [153057793]
>>
>> In object [1567151617] of class WARequestFields [27325441], at 1 based
>> offset 2, references non existent object [153059073]
>>
>> In object [1567151617] of class WARequestFields [27325441], at 1 based
>> offset 3, references non existent object [153058561]
>>
>> In object [1567152385] of class Association [67073], at 1 based offset 2,
>> references non existent object [153062657]
>>
>> In object [1567153409] of class KeyValueDictionary [79361], at 1 based
>> offset 6, references non existent object [153088513]
>>
>> In object [1567153409] of class KeyValueDictionary [79361], at 1 based
>> offset 8, references non existent object [153085185]
>>
>> In object [1448877313] of class RcKeyValueDictionary [87297], at 1 based
>> offset 9, references non existent object [66163457]
>>
>> In object [1448877313] of class RcKeyValueDictionary [87297], at 1 based
>> offset 10, references non existent object [66162945]
>>
>> In o
>>
>>
>>
>> 00000000000000000
>>
>>
>> let's not put ALL the log.... But I have the data
>>
>>
>> 00000000000000000
>>
>>
>>
>> Object Audit: 1518 errors were found
>>
>> ERROR 3022 , a InternalError occurred (error 3022),
>> reason:abortErrObjAuditFail (InternalError)
>>
>> topaz 1> SystemRepository markForCollection
>>
>> unknown command: SystemRepository
>>
>> topaz 1> printit
>>
>> SystemRepository markForCollection
>>
>> %
>>
>> a Warning
>>
>> gsResumable true
>>
>> gsTrappable true
>>
>> gsNumber 2515
>>
>> currGsHandler nil
>>
>> gsStack nil
>>
>> gsReason nil
>>
>> gsDetails markForCollection found 11062019 live objects,
>> 26243 dead objects(occupying approx 23...
>>
>> tag nil
>>
>> messageText nil
>>
>> gsArgs a Array
>>
>> topaz 1> printit
>>
>> SystemRepository reclaimAll
>>
>> %
>>
>> --reclaimAll: changed StnSignalAbortCrBacklog from 20 to 3
>>
>> --setGcConfig: set reclaimMinPages to 1
>>
>> --reclaimAll: changed reclaimMinPages from 300 to 1
>>
>> --setGcConfig: set reclaimSleepTime to 1
>>
>> --setGcConfig: set sleepTimeBetweenReclaimMs to 0
>>
>> --reclaimAll: using reclaimDeadEnabled true
>>
>> --reclaimAll: simpleCommit loopCount 10
>>
>> --reclaimAll: simpleCommit loopCount 20
>>
>> --reclaimAll: simpleCommit loopCount 30
>>
>> --reclaimAll: simpleCommit loopCount 40
>>
>> --reclaimAll: simpleCommit loopCount 50
>>
>> --reclaimAll: simpleCommit loopCount 60
>>
>> --reclaimAll: simpleCommit loopCount 70
>>
>> --reclaimAll: simpleCommit loopCount 80
>>
>> --reclaimAll: simpleCommit loopCount 90
>>
>> --reclaimAll: simpleCommit loopCount 100
>>
>> --reclaimAll: simpleCommit loopCount 110
>>
>> --reclaimAll: simpleCommit loopCount 120
>>
>> --reclaimAll: simpleCommit loopCount 130
>>
>> --reclaimAll: simpleCommit loopCount 140
>>
>> --reclaimAll: simpleCommit loopCount 150
>>
>> --reclaimAll: simpleCommit loopCount 160
>>
>> --reclaimAll: simpleCommit loopCount 170
>>
>> --reclaimAll: simpleCommit loopCount 180
>>
>> --reclaimAll: simpleCommit loopCount 190
>>
>> --reclaimAll: simpleCommit loopCount 200
>>
>> --reclaimAll: simpleCommit loopCount 210
>>
>> --reclaimAll: simpleCommit loopCount 220
>>
>> --reclaimAll: simpleCommit loopCount 230
>>
>> --reclaimAll: simpleCommit loopCount 240
>>
>>
>>
>> ***** Irritated, thinking of a database restore in about 5 minutes....
>>
>> ctl-C
>>
>>
>> [Sending soft break]
>>
>> ERROR 6003 , (Break)
>>
>> ------------------------------------------------------------------
>>
>> topaz 1> run
>>
>> SystemRepository repair
>>
>> %
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [149999361]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 2,
>> references non existent object [149999105]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 3,
>> references non existent object [149997313]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 4,
>> references non existent object [149997057]
>>
>> In object [1546008321] of class RcCounter [109313], at 1 based offset 5,
>> references non existent object [149996801]
>>
>> In object [1546009089] of class RcCounter [109313], at 1 based offset 1,
>> references non existent object [150003969]
>> ,,,,,,
>>
>>
>> ----------------------------------------------------
>>
>> Object Audit: 1518 errors were found
>>
>> Repairing error: BadReference - In object [1546008321] of class RcCounter
>> [109313], at 1 based offset 1, references non existent object [149999361]
>>
>> resetting offset 1 to OOP_NIL
>>
>> Repairing error: BadReference - In object [1546008321] of class RcCounter
>> [109313], at 1 based offset 2, references non existent object [149999105]
>>
>> resetting offset 2 to OOP_NIL
>>
>> Repairing error: BadReference - In object [1546008321] of class RcCounter
>> [109313], at 1 based offset 3, references non existent object [149997313]
>>
>> resetting offset 3 to OOP_NIL
>>
>> Repairing error: BadReference - In object [1546008321] of class RcCounter
>> [109313], at 1 based offset 4, references non existent object [149997057]
>>
>> resetting offset 4 to OOP_NIL
>>
>> Re
>>
>>
>> --------------------------------------------
>>
>>
>> Object Audit: successfully repaired 1518 errors
>>
>> 1518
>>
>> topaz 1> SystemRepository objectAudit
>>
>> unknown command: SystemRepository
>>
>> topaz 1> printit
>>
>> SystemRepository objectAudit
>>
>> %
>>
>>
>> Object Audit: Audit successfully completed; no errors were detected.
>>
>> true
>>
>> topaz 1> printit
>>
>> SystemRepository markForCollection
>>
>> %
>>
>> a Warning
>>
>> gsResumable true
>>
>> gsTrappable true
>>
>> gsNumber 2515
>>
>> currGsHandler nil
>>
>> gsStack nil
>>
>> gsReason nil
>>
>> gsDetails markForCollection found 11060528 live objects, 0
>> dead objects(occupying approx 0 byte...
>>
>> tag nil
>>
>> messageText nil
>>
>> gsArgs a Array
>>
>> topaz 1> exit
>>
>>
>>
>> 00000000000000000000
>>
>> Ok then trying to show the Object Log gives...
>>
>>
>>
>>
>> [1] MessageNotUnderstood >> defaultAction (envId 0)
>>
>> result: nil
>>
>> .t1: a MessageNotUnderstood occurred (error 2010), a UndefinedObject does
>> not understand #'isOlderThan:'
>>
>> receiver: a MessageNotUnderstood occurred (error 2010), a UndefinedObject
>> does not understand #'isOlderThan:'
>>
>> [2] AbstractException >> _signalWith: (envId 0)
>>
>> handleInCextensionBool: nil
>>
>> res: nil
>>
>> .t1: a MessageNotUnderstood occurred (error 2010), a UndefinedObject does
>> not understand #'isOlderThan:'
>>
>> receiver: a MessageNotUnderstood occurred (error 2010), a UndefinedObject
>> does not understand #'isOlderThan:'
>>
>> [3] AbstractException >> signal (envId 0)
>>
>> receiver: a MessageNotUnderstood occurred (error 2010), a UndefinedObject
>> does not understand #'isOlderThan:'
>>
>> [4] Object >> doesNotUnderstand: (envId 0)
>>
>> aMessageDescriptor: anArray( #'isOlderThan:', anArray( nil))
>>
>> args: anArray( nil)
>>
>> sel: #'isOlderThan:'
>>
>> ex: a MessageNotUnderstood occurred (error 2010), a UndefinedObject does
>> not understand #'isOlderThan:'
>>
>> .t1: #'isOlderThan:'
>>
>> .t2: anArray( nil)
>>
>> .t3: a MessageNotUnderstood occurred (error 2010), a UndefinedObject does
>> not understand #'isOlderThan:'
>>
>> receiver: nil
>>
>> [5] Object >> _doesNotUnderstand:args:envId:reason: (envId 0)
>>
>> aSymbol: #'isOlderThan:'
>>
>> anArray: anArray( nil)
>>
>> aSmallInt: 0
>>
>> dnuKind: 0
>>
>> receiver: nil
>>
>> [6] [] in RcQueue >> _timeSortedComponents (envId 0)
>>
>> a: nil
>>
>> b: nil
>>
>> receiver: nil
>>
>> [7] SortedCollection >> _findIndex: (envId 0)
>>
>> anObject: nil
>>
>> obj: nil
>>
>> probe: 45
>>
>> half: 0
>>
>> upper: 45
>>
>> lower: 45
>>
>> .t1: 45
>>
>> .t2: 0
>>
>> .t3: 45
>>
>> .t4: anExecBlock2
>>
>> .t5: nil
>>
>> .t6: nil
>>
>> receiver: aSortedCollection( aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, nil)
>>
>> [8] SortedCollection >> add: (envId 0)
>>
>> anObject: nil
>>
>> idx: nil
>>
>> .t1: aSortedCollection( aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, nil)
>>
>> .t2: nil
>>
>> receiver: aSortedCollection( aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, nil)
>>
>> [9] [] in RcQueue >> _timeSortedComponents (envId 0)
>>
>> each: nil
>>
>> collection: aSortedCollection( aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, nil)
>>
>> receiver: nil
>>
>> [10] RcQueue >> _doComponents: (envId 0)
>>
>> aBlock: anExecBlock1
>>
>> j: 6
>>
>> firstValid: 1
>>
>> sessionComponent: aRcQueueSessionComponent( aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, nil, nil)
>>
>> i: 12
>>
>> .t1: 6
>>
>> .t2: 100
>>
>> .t3: 12
>>
>> .t4: 6
>>
>> .t5: 6
>>
>> .t6: anExecBlock1
>>
>> .t7: nil
>>
>> receiver: aRcQueue( 6 CLIENT: Repository Size(1869)->Repository:
>> 3104.00M, Free: 2365.70M, Used: 738.30M, 6 CLIENT: MFC(1869)->a Warning
>> occurred (notification 2515), markForCollection found 7982074 live objects,
>> 11751 dead objects(occupying approx 1057590 bytes), 4 FastCGI(1915)->pid:
>> 1915 port: 9001, 4 FastCGI(1406)->pid: 1406 port: 9003, 7 'Error creating
>> WAWalkback: WAGsInvalidCallbackContext: You can only #call: and #answer:
>> from within a callback or a Task.'(1406)->Error creating WAWalkback:
>> WAGsInvalidCallbackContext: You can only #call: and #answer: from within a
>> callback or a Task., 2 -- resumable continuation --(1406)->a InternalError
>> occurred (error 2101), The object with object ID 153048833 does not exist.,
>> ...)
>>
>> [11] RcQueue >> _timeSortedComponents (envId 0)
>>
>> collection: aSortedCollection( aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry, aRcQueueEntry,
>> aRcQueueEntry, aRcQueueEntry, nil)
>>
>> receiver: aRcQueue( 6 CLIENT: Repository Size(1869)->Repository:
>> 3104.00M, Free: 2365.70M, Used: 738.30M, 6 CLIENT: MFC(1869)->a Warning
>> occurred (notification 2515), markForCollection found 7982074 live objects,
>> 11751 dead objects(occupying approx 1057590 bytes), 4 FastCGI(1915)->pid:
>> 1915 port: 9001, 4 FastCGI(1406)->pid: 1406 port: 9003, 7 'Error creating
>> WAWalkback: WAGsInvalidCallbackContext: You can only #call: and #answer:
>> from within a callback or a Task.'(1406)->Error creating WAWalkback:
>> WAGsInvalidCallbackContext: You can only #call: and #answer: from within a
>> callback or a Task., 2 -- resumable continuation --(1406)->a InternalError
>> occurred (error 2101), The object with object ID 153048833 does not exist.,
>> ...)
>>
>> [12] ObjectLogEntry class >> objectLog (envId 0)
>>
>> receiver: ObjectLogEntry
>>
>> [13] Executed Code
>>
>> count: 1
>>
>> result: ''
>>
>> receiver: nil
>>
>> [14] GsNMethod class >> _gsReturnToC (envId 0)
>>
>> receiver: nil
>>
>>
>> 0000000000000000000
>>
>> My Hack was to reinit the ObjectLog class var bits...
>>
>> -------------------------------------------------------
>> However then it turns out the maintenance gem won't start as it dies in
>> some seaside code so off we go to data recovery.
>>
>> --
>>
>> ===========================================================================
>> John M. McIntosh <johnmci at smalltalkconsulting.com>
>> Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
>>
>> ===========================================================================
>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140127/be5cbc26/attachment-0001.html>
More information about the Glass
mailing list