[Glass] taming growth during migration

[Bill Erickson via Glass]

Forgot to add -- taking this any further would require analysis of your
statmonitor data, so yes, I *would* have to wear my "Tech Support" hat.  ;-)

Bill.

------------------------------------------------------------------------
Bill Erickson
GemTalk Systems Engineering
15220 NW Greenbrier Parkway #240, Beaverton OR 97006
------------------------------------------------------------------------

On Thu, Feb 5, 2015 at 9:42 AM, Bill Erickson <
bill.erickson at gemtalksystems.com> wrote:

> Hey Otto,
>
> Depends on how deep you want to go.. ;-)
>
> If I see a question on the GLASS list that I can throw out a quick
> response for (like yours), I will.  But for anything requiring looking at
> logs or detailed analysis, I have to switch over to my official "Tech
> Support" hat and direct you there.
>
> Regards,
> Bill Erickson
>
>
> ------------------------------------------------------------------------
> Bill Erickson
> GemTalk Systems Engineering
> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
> ------------------------------------------------------------------------
>
> On Thu, Feb 5, 2015 at 9:32 AM, Otto Behrens <otto at finworks.biz> wrote:
>
>> Thanks.
>>
>> We have only the session doing the migrations logged in.
>>
>> What I also noticed was that the repository grows close to the maximum
>> quite quickly and then leaves about 185M free on the disk where the
>> extent is. The process continues for quite a while at that point,
>> doing a number of migrations until it eventually goes over the free
>> space threshold. The reclaim gem is quite busy also.
>>
>> Perhaps this is a support call?
>>
>> Cheers
>> Otto
>>
>> On Thu, Feb 5, 2015 at 7:20 PM, Bill Erickson
>> <bill.erickson at gemtalksystems.com> wrote:
>> > Otto,
>> >
>> > I just dealt with this same problem from another customer in Tech
>> Support.
>> >
>> > One thing that is essential is that you keep your commit record backlog
>> down
>> > as much as possible.  Make sure you don't have any sessions during this
>> time
>> > period that are in transaction and not periodically
>> committing/aborting, as
>> > this can cause serious DB growth using space to hang onto the commit
>> > records.  These also fill up the cache, flushing out needed data pages
>> and
>> > slowing down other sessions as they will need to re-load needed pages
>> back
>> > into the cache, which tends to exacerbate the problem.
>> >
>> > Regards,
>> > ------------------------------------------------------------------------
>> > Bill Erickson
>> > GemTalk Systems Engineering
>> > 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
>> > ------------------------------------------------------------------------
>> >
>> > On Thu, Feb 5, 2015 at 5:55 AM, Otto Behrens via Glass
>> > <glass at lists.gemtalksystems.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> We're migrating a large number of objects in our systems in this
>> release.
>> >> Our problem is that while this migration is running, the repository
>> grows
>> >> too much and runs out of space.
>> >>
>> >> We can just give it more space, but I feel it is a better solution to
>> get
>> >> the garbage collector to clean up faster. So, if I'm not smoking my
>> socks,
>> >> this boils down to tuning parameters so that the Gc keeps up with the
>> >> process that generates the garbage.
>> >>
>> >> We did the following which helped a lot: we commit in batches while
>> >> looping over objects and after each commit, we monitor
>> PagesNeedReclaimSize
>> >> and wait for it to drop below 250 before we continue with the next
>> batch.
>> >>
>> >> We played with the batch sizes. Too few objects in a batch increases
>> the
>> >> commit rate so high it runs out of space very quickly. Too many
>> creates a
>> >> larger number of objects to Gc in a epoch / reclaim. I found a batch
>> size
>> >> that sortof works.
>> >>
>> >> What I can figure out from the stats (using VSD) is that the
>> >> PagesNeedReclaimSize stays low.
>> >>
>> >> After a restore, loaded new code, just before we start the migration
>> >> process:
>> >>
>> >> SystemRepository fileSizeReport
>> >>    File size =       10334.00 Megabytes
>> >>    Space available = 55.67 Megabytes
>> >>
>> >> Just after the stone stops allowings commits because of the free space
>> >> threshold (and the topaz session exits):
>> >>
>> >> SystemRepository fileSizeReport
>> >>    Repository size = 14000.00 Megabytes
>> >>    Free Space =      2732.08 Megabytes
>> >>
>> >> So, it appears as if there is enough free space to work with, but it
>> does
>> >> not become available.
>> >>
>> >> I see this in the reclaimgcgem log, and wondered if I should lower /
>> >> increase this parameter:
>> >> "Committing the transaction. Reason: #objsMovedPerCommitThreshold
>> >> exceeded"
>> >>
>> >> If you have any ideas on this, or if you need logs / statmonitor files
>> or
>> >> something else, please let me know. Still investigating...
>> >>
>> >> We are running GS 3.1.0.5.
>> >>
>> >> Cheers
>> >> Otto
>> >>
>> >> _______________________________________________
>> >> Glass mailing list
>> >> Glass at lists.gemtalksystems.com
>> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> >>
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150205/e651a2ba/attachment-0001.html>