[Glass] taming growth during migration

[Bill Erickson via Glass]

Hey Otto,

Depends on how deep you want to go.. ;-)

If I see a question on the GLASS list that I can throw out a quick response
for (like yours), I will.  But for anything requiring looking at logs or
detailed analysis, I have to switch over to my official "Tech Support" hat
and direct you there.

Regards,
Bill Erickson


------------------------------------------------------------------------
Bill Erickson
GemTalk Systems Engineering
15220 NW Greenbrier Parkway #240, Beaverton OR 97006
------------------------------------------------------------------------

On Thu, Feb 5, 2015 at 9:32 AM, Otto Behrens <otto at finworks.biz> wrote:

> Thanks.
>
> We have only the session doing the migrations logged in.
>
> What I also noticed was that the repository grows close to the maximum
> quite quickly and then leaves about 185M free on the disk where the
> extent is. The process continues for quite a while at that point,
> doing a number of migrations until it eventually goes over the free
> space threshold. The reclaim gem is quite busy also.
>
> Perhaps this is a support call?
>
> Cheers
> Otto
>
> On Thu, Feb 5, 2015 at 7:20 PM, Bill Erickson
> <bill.erickson at gemtalksystems.com> wrote:
> > Otto,
> >
> > I just dealt with this same problem from another customer in Tech
> Support.
> >
> > One thing that is essential is that you keep your commit record backlog
> down
> > as much as possible.  Make sure you don't have any sessions during this
> time
> > period that are in transaction and not periodically committing/aborting,
> as
> > this can cause serious DB growth using space to hang onto the commit
> > records.  These also fill up the cache, flushing out needed data pages
> and
> > slowing down other sessions as they will need to re-load needed pages
> back
> > into the cache, which tends to exacerbate the problem.
> >
> > Regards,
> > ------------------------------------------------------------------------
> > Bill Erickson
> > GemTalk Systems Engineering
> > 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
> > ------------------------------------------------------------------------
> >
> > On Thu, Feb 5, 2015 at 5:55 AM, Otto Behrens via Glass
> > <glass at lists.gemtalksystems.com> wrote:
> >>
> >> Hi,
> >>
> >> We're migrating a large number of objects in our systems in this
> release.
> >> Our problem is that while this migration is running, the repository
> grows
> >> too much and runs out of space.
> >>
> >> We can just give it more space, but I feel it is a better solution to
> get
> >> the garbage collector to clean up faster. So, if I'm not smoking my
> socks,
> >> this boils down to tuning parameters so that the Gc keeps up with the
> >> process that generates the garbage.
> >>
> >> We did the following which helped a lot: we commit in batches while
> >> looping over objects and after each commit, we monitor
> PagesNeedReclaimSize
> >> and wait for it to drop below 250 before we continue with the next
> batch.
> >>
> >> We played with the batch sizes. Too few objects in a batch increases the
> >> commit rate so high it runs out of space very quickly. Too many creates
> a
> >> larger number of objects to Gc in a epoch / reclaim. I found a batch
> size
> >> that sortof works.
> >>
> >> What I can figure out from the stats (using VSD) is that the
> >> PagesNeedReclaimSize stays low.
> >>
> >> After a restore, loaded new code, just before we start the migration
> >> process:
> >>
> >> SystemRepository fileSizeReport
> >>    File size =       10334.00 Megabytes
> >>    Space available = 55.67 Megabytes
> >>
> >> Just after the stone stops allowings commits because of the free space
> >> threshold (and the topaz session exits):
> >>
> >> SystemRepository fileSizeReport
> >>    Repository size = 14000.00 Megabytes
> >>    Free Space =      2732.08 Megabytes
> >>
> >> So, it appears as if there is enough free space to work with, but it
> does
> >> not become available.
> >>
> >> I see this in the reclaimgcgem log, and wondered if I should lower /
> >> increase this parameter:
> >> "Committing the transaction. Reason: #objsMovedPerCommitThreshold
> >> exceeded"
> >>
> >> If you have any ideas on this, or if you need logs / statmonitor files
> or
> >> something else, please let me know. Still investigating...
> >>
> >> We are running GS 3.1.0.5.
> >>
> >> Cheers
> >> Otto
> >>
> >> _______________________________________________
> >> Glass mailing list
> >> Glass at lists.gemtalksystems.com
> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150205/51d74797/attachment.html>