[Glass] Transaction Logs so large ....

[Dale Henrichs via Glass]

On 10/24/2016 02:20 AM, Otto Behrens via Glass wrote:
> I suspect it is the persistence of the seaside continuations, with all
> the temps and things that go with it. We have the same problem; our
> number of users are picking up.
>
> I believe that there is a way to make the continuations transient,
> which we have not tried.
I think that if you make continuations non-persistent, then you will 
need to make sure that you have session affinity[1] ... this could cost 
in needing to have enough gems to handle the number of concurrent "long 
term" sessions you have ... I did experiments many years ago and I think 
that if you have a known number of users/browser sessions, then this 
approach could work ... let me know if there is interest here and I can 
blow the dust off of that work and give you some pointers/help.

I've also done some work with notTranlogged session data ... back when I 
was doing scaling trying to get to 10,000 requests second (hit 7000) I 
noticed the growth in tranlogs and a notTranlogged feature was 
subsequently added to GemStone ... until now I haven't heard about users 
really needing this feature -- even now it is in a stress testing 
context -- but a year ago I took some time to some to do some 
experiments with notTranlogged session state for Seaside and the work is 
on gitub[2]. I was able to my NotTranlogged experiment to function, but 
I never did any comparison tests to find out how much would actually be 
saved ...

If there is interest in bringing this up-to-date perhaps we could work 
together as a group to start doing experiments in this area and see if 
we can make a dent on the size of the large tranlogs.

Dale

[1] 
https://gemstonesoup.wordpress.com/2008/08/19/1-session-per-vm-another-scaling-alternative/
[2] https://github.com/dalehenrich/Seaside/tree/notTranlogged
>
> We load balance across gemstone sessions, which may influence what you do.
>
> On Mon, Oct 24, 2016 at 10:48 AM, Marten Feldtmann via Glass
> <glass at lists.gemtalksystems.com> wrote:
>> Ok, I have to make this perhaps a little bit clearer:
>>
>> In my test cases I have 200 to 400 clients (written in Python) playing their
>> role as client simulators. They do their requests via http/rest against
>> Gemstone/s against 8 responding gem/zinc tasks (running with different
>> configurations). They produce in total (10-20) events per seconds against
>> the database. These events are changing or adding data - no deletion at this
>> point.
>>
>> In terms of speed Gemstone has no problem answering the requests (even with
>> only one answering tasks it would work).
>>
>> The current database file is around 64 GB of size.
>>
>> The size of the uncompressed backup file is around 4 GB.
>>
>> As I mentioned in my older postings: this system produces 1GB transaction
>> log files within 4-5 minutes.Therefore we have around 100GB transaction data
>> each day ... wow.
>>
>> This seems to indicate, that not the extents files will be a problem, but
>> the management of the transaction files is far more difficult to handle (and
>> I was not aware of this :-))).
>>
>> Question to the more experienced Gemstone/S developers: is this ok (a
>> general pattern) ?
>>
>>
>> Marten Feldtmann via Glass <glass at lists.gemtalksystems.com> hat am 22.
>> Oktober 2016 um 22:08 geschrieben:
>>
>>
>> One of the most surprising things I noticed with my application is the
>> tremendous storage need of my transaction logs.
>>
>> Whenever I test my application (with around 200 clients) I get one
>> transaction log file within 4 minutes - each with a size of 1 GB. That would
>> lead to 120GB transaction logs within one days with 8 hours working time.
>>
>>
>> And of course I do full transaction log .... the database size after doing a
>> backup is 3.4 GB. Normally the clients do not do so much work - but they
>> send lots of events ...
>>
>> I looked at the inventory and I did not find any strange things ... any
>> ideas how to get an idea why such an amount of space is needed for the
>> transaction logs ....
>>
>> How do classes like Set, Arrays and changes to them results in transaction
>> logs changes ??? I prefer to use Arrays at this moment ...
>>
>>
>>
>>
>> _______________________________________________ Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass