[Glass] Transaction Logs so large ....

[Paul Baumann via Glass]

Tranlog files at first have transaction sequence not yet represented in the
random-access database extent files. Sequential files are faster to update
and changes can be replayed to extents of different ages (like from restore
or cloning). Tranlog files that only contain transactions older than the
last Checkpoint Commit can be archived. A checkpoint commit is the point
where extents now contain the tranlog changes. You do not need to keep full
tranlog files older than the checkpoint commit record in the extents backup
that you may wish restore. GS DBAs write shell scripts to manage these
things, knowing what tranlog files are no longer useful to keep.

If you already know this stuff then your question would be related to
checkpoint commits not happening in a timely manner. How that can happen
and how to improve that is a much deeper discussion, but is usually
remedied by changing gems to not stay with old views of the database. That
can mean more frequent aborts for example, and starting a transaction a
short time before making changes to commit. Just one gem staying
in-transaction for a long time can hold up checkpoint commits. A developer
logged into (and in-transaction) a very active database can hold up
checkpoints causing a backlog of transactions not yet applied to extent
files.

Paul Baumann

On Oct 24, 2016 5:20 AM, "Otto Behrens via Glass" <
glass at lists.gemtalksystems.com> wrote:

> I suspect it is the persistence of the seaside continuations, with all
> the temps and things that go with it. We have the same problem; our
> number of users are picking up.
>
> I believe that there is a way to make the continuations transient,
> which we have not tried.
>
> We load balance across gemstone sessions, which may influence what you do.
>
> On Mon, Oct 24, 2016 at 10:48 AM, Marten Feldtmann via Glass
> <glass at lists.gemtalksystems.com> wrote:
> > Ok, I have to make this perhaps a little bit clearer:
> >
> > In my test cases I have 200 to 400 clients (written in Python) playing
> their
> > role as client simulators. They do their requests via http/rest against
> > Gemstone/s against 8 responding gem/zinc tasks (running with different
> > configurations). They produce in total (10-20) events per seconds against
> > the database. These events are changing or adding data - no deletion at
> this
> > point.
> >
> > In terms of speed Gemstone has no problem answering the requests (even
> with
> > only one answering tasks it would work).
> >
> > The current database file is around 64 GB of size.
> >
> > The size of the uncompressed backup file is around 4 GB.
> >
> > As I mentioned in my older postings: this system produces 1GB transaction
> > log files within 4-5 minutes.Therefore we have around 100GB transaction
> data
> > each day ... wow.
> >
> > This seems to indicate, that not the extents files will be a problem, but
> > the management of the transaction files is far more difficult to handle
> (and
> > I was not aware of this :-))).
> >
> > Question to the more experienced Gemstone/S developers: is this ok (a
> > general pattern) ?
> >
> >
> > Marten Feldtmann via Glass <glass at lists.gemtalksystems.com> hat am 22.
> > Oktober 2016 um 22:08 geschrieben:
> >
> >
> > One of the most surprising things I noticed with my application is the
> > tremendous storage need of my transaction logs.
> >
> > Whenever I test my application (with around 200 clients) I get one
> > transaction log file within 4 minutes - each with a size of 1 GB. That
> would
> > lead to 120GB transaction logs within one days with 8 hours working time.
> >
> >
> > And of course I do full transaction log .... the database size after
> doing a
> > backup is 3.4 GB. Normally the clients do not do so much work - but they
> > send lots of events ...
> >
> > I looked at the inventory and I did not find any strange things ... any
> > ideas how to get an idea why such an amount of space is needed for the
> > transaction logs ....
> >
> > How do classes like Set, Arrays and changes to them results in
> transaction
> > logs changes ??? I prefer to use Arrays at this moment ...
> >
> >
> >
> >
> > _______________________________________________ Glass mailing list
> > Glass at lists.gemtalksystems.com
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
> >
> > _______________________________________________
> > Glass mailing list
> > Glass at lists.gemtalksystems.com
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20161024/577a8bc1/attachment.html>