[Glass] Transaction Logs so large ....

[Dale Henrichs via Glass]

Marten,

To be clear.

1GB in 5 minutes is 60Mb of tranlog data per second. At 20 events/second 
you are producing 3Mb per event ... is that reasonable given the 
application data you are saving per event from an application perspective?

If you still think that that is excessive, then I think submitting a HR 
is called for. Tech support can walk you through the tranlog analysis 
steps and help determine if what you are experiencing a problem that has 
already been fixed (the other customer with tranlog growth problems) or 
if you are experiencing "normal growth" --- we are aware of additional 
optimizations that can be made for tranlogs, but those optimizations are 
not yet available and there also the possibility of switching to another 
data structure, or you may even be persisting more data than you expect 
-- without analysis we can only guess.

I'm not sure that you need to change the SigAbort handler that we talked 
about at ESUG. I am under the impression that you are not concerned 
about extent growth and if so, then  that SigAbort handler is doing it's 
job.

Dale


On 10/24/2016 11:12 AM, Dale Henrichs wrote:
>
>
> On 10/24/2016 06:12 AM, Marten Feldtmann via Glass wrote:
>>
>> Concerning transactions: Well, all Topaz scripts (starting the Zinc 
>> server) are started in #manualBegin transactionMode. If not in that 
>> state the handling of SigAbort handlers is more difficult.
>>
> SigAbort handling has nothing to do with tranlog growth, so don't mess 
> with this ...
>>
>> Other than that: after an incoming request a BeginTransaction is 
>> done, the work is done and a commit or abort is then done ... then 
>> the gem is again out of an transaction - and then it should be no 
>> candidate for SigAbort handler (I'm not sure of that last statement).
>>
> Again ... Sigabort handling does not affect the amount of data that is 
> dumped to a tranlog
>>
>> Conflicts - yes initially lots of. After that we introduce retries on 
>> the server side and now the conflicts are not very often - though the 
>> external programmers should always consider this ( ... and its not 
>> done).
>>
>> Other conflicts can not be solved in this way and need a different 
>> API behaviour (normally implemented with RcQueue and worker topaz 
>> processes on the server side). The offered API then returns a TASK id 
>> and the programmer has to watch for its task ... long-term logic 
>> tasks can be programmed in a conflict-free way then on the server side.
>>
>> For logic conflicts we have a revision counter in the objects.
>>
>> (We use an API-HTTP-REST-RPC oriented way of programming the 
>> Gemstone/S database when interacting via C#, Python or Javascript 
>> with the database).
>>
>
> Since you are not using Seaside we cannot blame continuations, so we 
> have to look for other sources of tranlog growth ...
>
> Do you have large sorted collections or other SequenceableCollection 
> collections where you insert data in the middle of the collection? If 
> you do we write tranlog entries for the shift of the data in these 
> types of collections ...
>
> If you use indexes it is possible for the  RcIndexDictionary collision 
> buckets to get large and they will behave like a 
> SequenceableCollection collection when it comes to inserts ...
>
> I think the best bet here is to first identify what's being dumped 
> into your tranlogs and then go from there ... we've recently had 
> another customer with tranlog growth issues and the engineer that 
> handled that is out of the office ... This would actually be a good 
> case for submitting an HR and then work directly on this issue with 
> tech support to help you identify the source of the tranlog growth and 
> work out strategies for reducing or otherwise addressing the problem.
>
> Dale