[Glass] Transaction Logs so large ....

Tue Oct 25 05:59:38 PDT 2016


On 10/24/16 9:20 PM, Otto Behrens wrote:
> Thanks, Dale
>
>> I think that if you make continuations non-persistent, then you will need to
>> make sure that you have session affinity[1] ... this could cost in needing
>> to have enough gems to handle the number of concurrent "long term" sessions
>> you have ... I did experiments many years ago and I think that if you have a
>> known number of users/browser sessions, then this approach could work ...
>> let me know if there is interest here and I can blow the dust off of that
>> work and give you some pointers/help.
> We prefer to load balance over sessions because we have some requests
> that take a few seconds to complete. We do not want one session to
> hold up another when there are gems idle.
I had done some work on "self load balancing" where a gem would bounce 
the request if it was "running a long operation" ... things get 
interesting when all of the gems are running long transactions though:)

if you are happy with what you are doing there's no need to change ..
>
> OTOH, we could reconsider and perhaps start a new gem for each
> session. Our gems eat a lot of RAM though, which we need to sort out.
Yeah that's the big one ... there's a hard cap on number of users if you 
go this route ...
>
> Yes, we can run some of these requests in the background. But they
> creep up on us and we don't realise there's a problem until users
> complain that their requests time out.
>
>> I've also done some work with notTranlogged session data ... back when I was
>> doing scaling trying to get to 10,000 requests second (hit 7000) I noticed
>> the growth in tranlogs and a notTranlogged feature was subsequently added to
>> GemStone ... until now I haven't heard about users really needing this
>> feature -- even now it is in a stress testing context -- but a year ago I
>> took some time to some to do some experiments with notTranlogged session
>> state for Seaside and the work is on gitub[2]. I was able to my
>> NotTranlogged experiment to function, but I never did any comparison tests
>> to find out how much would actually be saved ...
> I'm not there yet. I think that it is manageable at this stage. 100GB
> disk space is not overly expensive. Other things to do now.
Yeah, it didn't seem to be a big problem yet ... but it might be worth 
starting to look into notTranlogged so that Seaside is ready when you 
need it .. Like I said I didn't check in to the tranlog yield in my 
experiments, but I had gone as far as I could without starting to making 
structural changes to Seaside ... which might turn out to be necessary ...
>
> Just a thought: I wonder if the not tranlogged will work with a hot
> standby. Should be ok? Have to be sure that one can recover when you
> lose the extent. That would be the major test.
Nottranlogged objects are persisted in the main disk bug not written to 
tranlogs .. so the implication is that notTranlogged objects are not 
present upon recovery --- but then Seaside session state is not needed 
if the stone crashes ... so if it's not written to the tranlog it will 
not be sent to the hot standby which should be fine ...

Between disk and hot standby, the notTranlogged might be worth having in 
our pockets .
>
>> [2] https://github.com/dalehenrich/Seaside/tree/notTranlogged
>>
>>> We load balance across gemstone sessions, which may influence what you do.
>>>
>>> On Mon, Oct 24, 2016 at 10:48 AM, Marten Feldtmann via Glass
>>> <glass at lists.gemtalksystems.com> wrote:
>>>> Ok, I have to make this perhaps a little bit clearer:
>>>>
>>>> In my test cases I have 200 to 400 clients (written in Python) playing
>>>> their
>>>> role as client simulators. They do their requests via http/rest against
>>>> Gemstone/s against 8 responding gem/zinc tasks (running with different
>>>> configurations). They produce in total (10-20) events per seconds against
>>>> the database. These events are changing or adding data - no deletion at
>>>> this
>>>> point.
>>>>
>>>> In terms of speed Gemstone has no problem answering the requests (even
>>>> with
>>>> only one answering tasks it would work).
>>>>
>>>> The current database file is around 64 GB of size.
>>>>
>>>> The size of the uncompressed backup file is around 4 GB.
>>>>
>>>> As I mentioned in my older postings: this system produces 1GB transaction
>>>> log files within 4-5 minutes.Therefore we have around 100GB transaction
>>>> data
>>>> each day ... wow.
>>>>
>>>> This seems to indicate, that not the extents files will be a problem, but
>>>> the management of the transaction files is far more difficult to handle
>>>> (and
>>>> I was not aware of this :-))).
>>>>
>>>> Question to the more experienced Gemstone/S developers: is this ok (a
>>>> general pattern) ?
>>>>
>>>>
>>>> Marten Feldtmann via Glass <glass at lists.gemtalksystems.com> hat am 22.
>>>> Oktober 2016 um 22:08 geschrieben:
>>>>
>>>>
>>>> One of the most surprising things I noticed with my application is the
>>>> tremendous storage need of my transaction logs.
>>>>
>>>> Whenever I test my application (with around 200 clients) I get one
>>>> transaction log file within 4 minutes - each with a size of 1 GB. That
>>>> would
>>>> lead to 120GB transaction logs within one days with 8 hours working time.
>>>>
>>>>
>>>> And of course I do full transaction log .... the database size after
>>>> doing a
>>>> backup is 3.4 GB. Normally the clients do not do so much work - but they
>>>> send lots of events ...
>>>>
>>>> I looked at the inventory and I did not find any strange things ... any
>>>> ideas how to get an idea why such an amount of space is needed for the
>>>> transaction logs ....
>>>>
>>>> How do classes like Set, Arrays and changes to them results in
>>>> transaction
>>>> logs changes ??? I prefer to use Arrays at this moment ...
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________ Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass