[Glass] gc settings

[Otto Behrens]

Thanks. This is seem to help. A few methods work together here. There
is some room for improvement, but perhaps this helps someone. When
writing scripts or migrating, we use this method:

SequenceableCollection | committingInBatchesOf: aNumber do: aBlock
  self
    doWithIndex: [ :instance :index |
      aBlock value: instance.
      index \\ aNumber = 0
        ifTrue: [ instance commitOrFail; waitForReclaim ] ];
    commitOrFail;
    waitForReclaim

commitOrFail is a method that we use to raise an exception if a commit failed.

waitForReclaim
  self pagesNeedReclaimSize > 2000
    ifTrue: [
      Transcript show: 'Waiting for reclaim of ' , self
pagesNeedReclaimSize printString , ' pages'.
      [ self pagesNeedReclaimSize > 250 ]
        whileTrue: [
          (Delay forSeconds: 1) wait.
          System abortTransaction ].
      Transcript show: 'Reclaimed' ]

pagesNeedReclaimSize
^ (System cacheStatistics: 1) at: (System cacheStatisticsDescription
indexOf: 'PagesNeedReclaimSize')
www.FinWorks.biz
+27 82 809 2375


On Sat, Feb 15, 2014 at 12:37 AM, Dale Henrichs
<dale.henrichs at gemtalksystems.com> wrote:
> Yeah a couple thousand was a very conservative figure :) ... depends upon
> how much extra space you can afford in your repo ... at 4M (16k) pages,
> that's 64G which might be bigger than a lot of GLASS data bases (not all,
> though:)
>
> Dale
>
>
> On Fri, Feb 14, 2014 at 2:32 PM, Jon Paynter <kittle31 at gmail.com> wrote:
>>
>> Another option is to allocate extra time for extra aggressive cleanup of
>> your shadow pages.  Then let the migration run all out for as long as it
>> needs.  When its done, kick everyone out of the stone and spin up a bunch of
>> reclaim gems.  Like Dale said, they quite fast.
>>
>> Im not sure where the "couple thousand" number came from -- but we will
>> only pause a migration for cleanup if the shadowpages get above 4million.
>>
>>
>> On Fri, Feb 14, 2014 at 1:35 PM, Otto Behrens <otto at finworks.biz> wrote:
>>>
>>> Thanks a lot. Trying it out.
>>>
>>> On Fri, Feb 14, 2014 at 9:44 PM, Dale Henrichs
>>> <dale.henrichs at gemtalksystems.com> wrote:
>>> > Otto,
>>> >
>>> > I talked this over with one of the guys here and he suggests that the
>>> > trick
>>> > is to not create shadow pages faster than the gc gem can handle them
>>> > ... the
>>> > gcgems are pretty aggreessive by default ...
>>> >
>>> > He suggests that you  commit frequently and the  monitor the
>>> > PagesNeedsReclaiming stat in your migration loop and when it gets above
>>> > a
>>> > certain threshold (a couple thousand pages?) stop migrating and go into
>>> > an
>>> > "idle loop" where you abort (to avoid CR backlog and allow pages to run
>>> > through their normal life cycle) and check for the stat to drop below
>>> > another threshold (250 pages? once it's below your threshold get going
>>> > again
>>> > ...
>>> >
>>> > regarding the epoch gc. the become operation does not itself generate a
>>> > lot
>>> > of dead if you dow a become between a persistent object and a temp
>>> > object
>>> > and commit, the temp object will be written as the persist object and
>>> > no
>>> > garbage will be produced ... you will however create shadowpages as the
>>> > new
>>> > versions of the objects are copied to new pages (page reclaim is why
>>> > the
>>> > gcgems get revved up) ... now if you are creating short-lived persist
>>> > objects as some other part of your  migration process then an epoch
>>> > will
>>> > help keep those dead boys in check ...
>>> >
>>> > Dale
>>> >
>>> >
>>> > On Fri, Feb 14, 2014 at 4:07 AM, Otto Behrens <otto at finworks.biz>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Do you have a set of GcUser settings that we can use for aggressive
>>> >> garbage collection during a migration process? We would like to do the
>>> >> garbage collection (epoch and reclaim) while the migration process is
>>> >> generating a lot of dead stuff. We don't mind if the migration takes a
>>> >> bit longer. We do mind if the migration generates a lot of free space
>>> >> in the extent.
>>> >>
>>> >> Thanks
>>> >> Otto
>>> >> _______________________________________________
>>> >> Glass mailing list
>>> >> Glass at lists.gemtalksystems.com
>>> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> >
>>> >
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>