[Glass] ObjectLog cleanup

Thu Aug 6 07:41:48 PDT 2015

On Wed, Aug 5, 2015 at 6:48 PM, James Foster <
james.foster at gemtalksystems.com> wrote:

> Here is my script...
>
>
Thanks for sharing your script James. I answer inline below some questions.

>
> cleanupObjectLog
>
>   | log date1 date2 sortedCopy count |
>   log := ObjectLogEntry objectLog.
>
> “keep only one per day of various maintenance tasks (useful to go back
> years on size!)"
>   #('MTCE: Repository Size' 'MTCE: MFC' 'MTCE: expired sessions') do:
> [:eachLabel |
>   log copy do: [:each |
>   each label = eachLabel ifTrue: [
>   each stamp date = date1 ifTrue: [
>   log remove: each.
>   ] ifFalse: [
>   date1 := each stamp date.
>   ].
>   ].
>   ].
>   ].
>

Good!

>
> “Keep only one per week of anything more than a month old"
>   date1 := Date today subtractDays: 30.
>   log copy do: [:each |
>   (each stamp date < date1 and: [each stamp date day \\ 7 ~= 0]) ifTrue: [
>   log remove: each.
>   ].
>   ].
>
>

This is the part I understand the less. Why would make sense to keep one
random entry once per week of things older than a month?
I mean...a random entry will probably tell me anything. In which case do
you think this could be useful?

> “keep only last three days of session expiration"
>   date1 := Date today subtractDays: 3.
>   log copy do: [:each |
>   (each stamp date < date1 and: [each label = 'MTCE: expired sessions'])
> ifTrue: [
>   log remove: each.
>   ].
>   ].
>
>

Good!

> “keep only five per day of lock failures"
> date1 := nil.
> log copy do: [:each |
> each stamp date = date1 ifTrue: [
> each label = 'Lock not acquired - retrying' ifTrue: [
> count < 5 ifTrue: [
> count := count + 1.
> ] ifFalse: [
> log remove: each.
> ].
> ].
> ] ifFalse: [
> date1 := each stamp date.
> count := 0.
> ].
> ].
>

Good!

>
> “beyond ten days keep only twenty errors per day"
> date1 := nil.
>   date2 := Date today subtractDays: 10.
> log copy do: [:each |
> each stamp date = date1 ifTrue: [
> each priority == 2 ifTrue: [
> count < 20 ifTrue: [
> count := count + 1.
> ] ifFalse: [
> date1 < date2 ifTrue: [
> log remove: each.
> ].
> ].
> ].
> ] ifFalse: [
> date1 := each stamp date.
> count := 0.
> ].
> ].
>
>
Good.

>   sortedCopy := log asSortedCollection: [:a :b |
>   a priority > b priority or: [
>   a priority = b priority and: [a stamp < b stamp].
>   ].
>   ].
>   log
>   size: 0;
>   addAll: sortedCopy asOrderedCollection;
> yourself.
>

A couple of questions:

1) Don't you need to get the lock (as I did in my script) before trying to
write the object log?

2) I also have some Transcript show (priority 7), some inspect/explore
(priority 8) and some trace:  (priority 6).  I guess I should add a similar
part for this...not sure how many days though...

Thanks in advance,

>
%
>
> On Aug 5, 2015, at 2:33 PM, Mariano Martinez Peck via Glass <
> glass at lists.gemtalksystems.com> wrote:
>
>
>
> On Wed, Aug 5, 2015 at 5:41 PM, Dale Henrichs via Glass <
> glass at lists.gemtalksystems.com> wrote:
>
>>
>>
>> On 07/31/2015 08:58 AM, Trussardi Dario Romano via Glass wrote:
>>
>>
>> James,
>>
>>
>> On Jul 31, 2015, at 3:37 AM, Trussardi Dario Romano via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>> Ciao,
>>
>> i have a deployment system based on GemStone version '3.1.0.6'
>>
>>
>> Now i use Gemtools and todeClient to do some works on the repository:
>>  update code,  save repository .....
>>
>>
>> When i login with the tools  i note the repository grows significantly,  and
>> also, the relative  full backups grows significantly.
>>
>> To reset the system i need to clear the Object Log with the relative
>> clear command  ( ol clear in tode )
>>
>>
>> Do you think that the Object Log grows significantly during your time
>> logged in with the tools? What is its size at the beginning and end of your
>> session?
>>
>>
>> I don't have real data about the extent0.dbf change size from beginning
>> and end of a gemtools session.
>>
>> But the full backup size  before Object Log  clear is :  4 258 529280
>>
>> after Object Log clear is: 1 376 124 928
>>
>> The SystemRepository freeSpace size before Object Log clear is :  1 092
>> 419 584
>>
>> after Object Log clear ( and one hour  of system background works )  is:
>>  5 267 701 760
>>
>>
>> Okay as James has mentioned, it definitely looks like the Object Log is
>> the source of the "excess data" ... it would be interesting to understand
>> what is in the object log ... My guess would be that you are having
>> recurring errors and the continuations created to record those errors can
>> easily cause a bunch of data to be stored in your extent ... if the errors
>> are providing useful data, than this can be viewed as a "cost of doing
>> business" and you will just need to have a regularly scheduled task for
>> clearing the object log ... you could arrange to regularly clear
>> continuation entries from the object log that are more than a day or week
>> or ? old then at least you would expect to reach a steady state for extent
>> size ... then you would only need to consider shrinking the size of the
>> extent when you had an anomaly where a series of unexpected errors cropped
>> up ....
>>
>>
> I do exactly that as part of my daily cleaning:
>
>
> cleanObjectLogEntriesDaysAgo: aNumberOfDays
> | log |
> log := self objectLogEntries: true.
> (log select: [:ea | (Date today - ea stamp asDate) asDays >= aNumberOfDays
> ]) do: [:ea | log remove: ea ifAbsent: []].
> System commitTransaction.
>
>
> objectLogEntries: shouldLock
>
> shouldLock
> ifTrue: [
> System writeLock: ObjectLogEntry objectQueue
> ifDenied: [
> Transcript show: 'ObjectLogEntry objectQueue lock denied'.
> ^ nil
> ]
> ifChanged: [
> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue.
> Transcript show: 'ObjectLogEntry objectQueue lock dirty'.
> ^ nil
> ].
> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue].
> ^ObjectLogEntry objectLog
>
>
> Maybe we could add this directly to class side of ObjectLogEntry so that
> others can benefit from it?
>
>
>
>
>> As Mariano and James point out, there are other possible sources of
>> extent growth that are related to a commit record backlog because you have
>> a GemTools/tODE/topaz session open (and idle) on a production system that
>> is "busy committing" ... commit record backlog data is transient data that
>> is stored in the extent and will cause the extent to grow, but once the
>> session or sessions causing the commit record backlog log out commit or
>> abort, the transient data is no longer needed by the system and will
>> "magically turn into free space" at checkpoint boundaries ....
>>
>> The key point here is that object log data is persistent "user data" and
>> will not turn into free space until 1) you break the link to the persistent
>> root (object log clear) and 2) you run an MFC. Commit record backlog data
>> "system data" and will turn into free space as soon as the sessions
>> abort/commit/logout and a checkpoint is hit, i.e., no MFC is needed ... The
>> default checkpoint interval is 5 minutes I think ...
>>
>>
> Dale, it is not clear to me what the checkpoint interval is. I understood
> it was at abort/commit/logout..so how is this internal related? Is there a
> way I can check the stone configuration parameter of this (the 5 minutes)?
>
> So for the "system data" to turn into free space, the other gems need to
> abort/commit/logout AND only after 5 minutes that turning into free space
> will happen?   I ask because I am scheduling some batch jobs running at
> night, and as soon as they all finish, I run GC...and sometimes it seems I
> do not really GC what I should have...
>
> Thanks!
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
>

-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150806/89bdf457/attachment-0001.html>