[Glass] Transaction Logs so large ....

[Otto Behrens via Glass]

Thanks, Dale

> I think that if you make continuations non-persistent, then you will need to
> make sure that you have session affinity[1] ... this could cost in needing
> to have enough gems to handle the number of concurrent "long term" sessions
> you have ... I did experiments many years ago and I think that if you have a
> known number of users/browser sessions, then this approach could work ...
> let me know if there is interest here and I can blow the dust off of that
> work and give you some pointers/help.

We prefer to load balance over sessions because we have some requests
that take a few seconds to complete. We do not want one session to
hold up another when there are gems idle.

OTOH, we could reconsider and perhaps start a new gem for each
session. Our gems eat a lot of RAM though, which we need to sort out.

Yes, we can run some of these requests in the background. But they
creep up on us and we don't realise there's a problem until users
complain that their requests time out.

> I've also done some work with notTranlogged session data ... back when I was
> doing scaling trying to get to 10,000 requests second (hit 7000) I noticed
> the growth in tranlogs and a notTranlogged feature was subsequently added to
> GemStone ... until now I haven't heard about users really needing this
> feature -- even now it is in a stress testing context -- but a year ago I
> took some time to some to do some experiments with notTranlogged session
> state for Seaside and the work is on gitub[2]. I was able to my
> NotTranlogged experiment to function, but I never did any comparison tests
> to find out how much would actually be saved ...

I'm not there yet. I think that it is manageable at this stage. 100GB
disk space is not overly expensive. Other things to do now.

Just a thought: I wonder if the not tranlogged will work with a hot
standby. Should be ok? Have to be sure that one can recover when you
lose the extent. That would be the major test.

> [2] https://github.com/dalehenrich/Seaside/tree/notTranlogged
>
>>
>> We load balance across gemstone sessions, which may influence what you do.
>>
>> On Mon, Oct 24, 2016 at 10:48 AM, Marten Feldtmann via Glass
>> <glass at lists.gemtalksystems.com> wrote:
>>>
>>> Ok, I have to make this perhaps a little bit clearer:
>>>
>>> In my test cases I have 200 to 400 clients (written in Python) playing
>>> their
>>> role as client simulators. They do their requests via http/rest against
>>> Gemstone/s against 8 responding gem/zinc tasks (running with different
>>> configurations). They produce in total (10-20) events per seconds against
>>> the database. These events are changing or adding data - no deletion at
>>> this
>>> point.
>>>
>>> In terms of speed Gemstone has no problem answering the requests (even
>>> with
>>> only one answering tasks it would work).
>>>
>>> The current database file is around 64 GB of size.
>>>
>>> The size of the uncompressed backup file is around 4 GB.
>>>
>>> As I mentioned in my older postings: this system produces 1GB transaction
>>> log files within 4-5 minutes.Therefore we have around 100GB transaction
>>> data
>>> each day ... wow.
>>>
>>> This seems to indicate, that not the extents files will be a problem, but
>>> the management of the transaction files is far more difficult to handle
>>> (and
>>> I was not aware of this :-))).
>>>
>>> Question to the more experienced Gemstone/S developers: is this ok (a
>>> general pattern) ?
>>>
>>>
>>> Marten Feldtmann via Glass <glass at lists.gemtalksystems.com> hat am 22.
>>> Oktober 2016 um 22:08 geschrieben:
>>>
>>>
>>> One of the most surprising things I noticed with my application is the
>>> tremendous storage need of my transaction logs.
>>>
>>> Whenever I test my application (with around 200 clients) I get one
>>> transaction log file within 4 minutes - each with a size of 1 GB. That
>>> would
>>> lead to 120GB transaction logs within one days with 8 hours working time.
>>>
>>>
>>> And of course I do full transaction log .... the database size after
>>> doing a
>>> backup is 3.4 GB. Normally the clients do not do so much work - but they
>>> send lots of events ...
>>>
>>> I looked at the inventory and I did not find any strange things ... any
>>> ideas how to get an idea why such an amount of space is needed for the
>>> transaction logs ....
>>>
>>> How do classes like Set, Arrays and changes to them results in
>>> transaction
>>> logs changes ??? I prefer to use Arrays at this moment ...
>>>
>>>
>>>
>>>
>>> _______________________________________________ Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass