[Glass] ObjectLog cleanup
James Foster via Glass
glass at lists.gemtalksystems.com
Thu Aug 6 08:53:22 PDT 2015
Good comments and questions (with further inline responses)…
> On Aug 6, 2015, at 7:41 AM, Mariano Martinez Peck <marianopeck at gmail.com> wrote:
>
>
> On Wed, Aug 5, 2015 at 6:48 PM, James Foster <james.foster at gemtalksystems.com <mailto:james.foster at gemtalksystems.com>> wrote:
> Here is my script...
>
>
> Thanks for sharing your script James. I answer inline below some questions.
>
>
> cleanupObjectLog
>
> | log date1 date2 sortedCopy count |
> log := ObjectLogEntry objectLog.
>
> “keep only one per day of various maintenance tasks (useful to go back years on size!)"
> #('MTCE: Repository Size' 'MTCE: MFC' 'MTCE: expired sessions') do: [:eachLabel |
> log copy do: [:each |
> each label = eachLabel ifTrue: [
> each stamp date = date1 ifTrue: [
> log remove: each.
> ] ifFalse: [
> date1 := each stamp date.
> ].
> ].
> ].
> ].
>
>
> Good!
>
>
>
> “Keep only one per week of anything more than a month old"
> date1 := Date today subtractDays: 30.
> log copy do: [:each |
> (each stamp date < date1 and: [each stamp date day \\ 7 ~= 0]) ifTrue: [
> log remove: each.
> ].
> ].
>
>
>
> This is the part I understand the less. Why would make sense to keep one random entry once per week of things older than a month?
> I mean...a random entry will probably tell me anything. In which case do you think this could be useful?
The comment should say “Keep only one DAY per week of anything more than month old”.
This is an attempt to balance keeping everything and keeping nothing (and I don’t claim to have the perfect balance!). Beyond 30 days, for things like MFC reports, four per month seems quite adequate to give an idea of the trend. For things like errors, if you haven’t looked at it in 30 days then some pruning seems appropriate. I figured that picking one random day per week gives an idea if an error is common or rare.
> “keep only last three days of session expiration"
> date1 := Date today subtractDays: 3.
> log copy do: [:each |
> (each stamp date < date1 and: [each label = 'MTCE: expired sessions']) ifTrue: [
> log remove: each.
> ].
> ].
>
>
>
> Good!
>
>
> “keep only five per day of lock failures"
> date1 := nil.
> log copy do: [:each |
> each stamp date = date1 ifTrue: [
> each label = 'Lock not acquired - retrying' ifTrue: [
> count < 5 ifTrue: [
> count := count + 1.
> ] ifFalse: [
> log remove: each.
> ].
> ].
> ] ifFalse: [
> date1 := each stamp date.
> count := 0.
> ].
> ].
>
> Good!
>
>
>
> “beyond ten days keep only twenty errors per day"
> date1 := nil.
> date2 := Date today subtractDays: 10.
> log copy do: [:each |
> each stamp date = date1 ifTrue: [
> each priority == 2 ifTrue: [
> count < 20 ifTrue: [
> count := count + 1.
> ] ifFalse: [
> date1 < date2 ifTrue: [
> log remove: each.
> ].
> ].
> ].
> ] ifFalse: [
> date1 := each stamp date.
> count := 0.
> ].
> ].
>
>
> Good.
>
>
> sortedCopy := log asSortedCollection: [:a :b |
> a priority > b priority or: [
> a priority = b priority and: [a stamp < b stamp].
> ].
> ].
> log
> size: 0;
> addAll: sortedCopy asOrderedCollection;
> yourself.
>
> A couple of questions:
>
> 1) Don't you need to get the lock (as I did in my script) before trying to write the object log?
Good catch. In the few applications where I’ve used this script the users are active during the day and the maintenance (including this script) run during the night. Thus, I haven’t had a locking problem (that I know of!). But, yes, I should follow your example of getting the lock.
> 2) I also have some Transcript show (priority 7), some inspect/explore (priority 8) and some trace: (priority 6). I guess I should add a similar part for this...not sure how many days though…
I think that the above will trim your other things to one day per week. From there you could reduce the count as I’ve done with errors.
>
> Thanks in advance,
I appreciate Dale’s suggestion of combining the best of the scripts and adding it to the framework.
James
>
>
> %
>
>> On Aug 5, 2015, at 2:33 PM, Mariano Martinez Peck via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>
>>
>>
>> On Wed, Aug 5, 2015 at 5:41 PM, Dale Henrichs via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>
>>
>> On 07/31/2015 08:58 AM, Trussardi Dario Romano via Glass wrote:
>>>
>>> James,
>>>
>>>>
>>>>> On Jul 31, 2015, at 3:37 AM, Trussardi Dario Romano via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>>>
>>>>> Ciao,
>>>>>
>>>>> i have a deployment system based on GemStone version '3.1.0.6'
>>>>>
>>>>>
>>>>> Now i use Gemtools and todeClient to do some works on the repository: update code, save repository .....
>>>>>
>>>>>
>>>>> When i login with the tools i note the repository grows significantly, and also,
>>>>> the relative full backups grows significantly.
>>>>>
>>>>> To reset the system i need to clear the Object Log with the relative clear command ( ol clear
>>>>> in tode )
>>>>
>>>> Do you think that the Object Log grows significantly during your time logged in with the tools? What is its size at the beginning and end of your session?
>>>
>>> I don't have real data about the extent0.dbf change size from beginning and end of a gemtools session.
>>>
>>> But the full backup size before Object Log clear is : 4 258 529280
>>>
>>> after Object Log clear is: 1 376 124 928
>>>
>>>
>>> The SystemRepository freeSpace size before Object Log clear is : 1 092 419 584
>>>
>>> after Object Log clear ( and one hour of system background works ) is: 5 267 701 760
>>>
>>
>> Okay as James has mentioned, it definitely looks like the Object Log is the source of the "excess data" ... it would be interesting to understand what is in the object log ... My guess would be that you are having recurring errors and the continuations created to record those errors can easily cause a bunch of data to be stored in your extent ... if the errors are providing useful data, than this can be viewed as a "cost of doing business" and you will just need to have a regularly scheduled task for clearing the object log ... you could arrange to regularly clear continuation entries from the object log that are more than a day or week or ? old then at least you would expect to reach a steady state for extent size ... then you would only need to consider shrinking the size of the extent when you had an anomaly where a series of unexpected errors cropped up ....
>>
>>
>> I do exactly that as part of my daily cleaning:
>>
>>
>> cleanObjectLogEntriesDaysAgo: aNumberOfDays
>> | log |
>> log := self objectLogEntries: true.
>> (log select: [:ea | (Date today - ea stamp asDate) asDays >= aNumberOfDays ]) do: [:ea | log remove: ea ifAbsent: []].
>> System commitTransaction.
>>
>>
>> objectLogEntries: shouldLock
>>
>> shouldLock
>> ifTrue: [
>> System writeLock: ObjectLogEntry objectQueue
>> ifDenied: [
>> Transcript show: 'ObjectLogEntry objectQueue lock denied'.
>> ^ nil
>> ]
>> ifChanged: [
>> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue.
>> Transcript show: 'ObjectLogEntry objectQueue lock dirty'.
>> ^ nil
>> ].
>> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue].
>> ^ObjectLogEntry objectLog
>>
>>
>> Maybe we could add this directly to class side of ObjectLogEntry so that others can benefit from it?
>>
>>
>>
>> As Mariano and James point out, there are other possible sources of extent growth that are related to a commit record backlog because you have a GemTools/tODE/topaz session open (and idle) on a production system that is "busy committing" ... commit record backlog data is transient data that is stored in the extent and will cause the extent to grow, but once the session or sessions causing the commit record backlog log out commit or abort, the transient data is no longer needed by the system and will "magically turn into free space" at checkpoint boundaries ....
>>
>> The key point here is that object log data is persistent "user data" and will not turn into free space until 1) you break the link to the persistent root (object log clear) and 2) you run an MFC. Commit record backlog data "system data" and will turn into free space as soon as the sessions abort/commit/logout and a checkpoint is hit, i.e., no MFC is needed ... The default checkpoint interval is 5 minutes I think ...
>>
>>
>> Dale, it is not clear to me what the checkpoint interval is. I understood it was at abort/commit/logout..so how is this internal related? Is there a way I can check the stone configuration parameter of this (the 5 minutes)?
>>
>> So for the "system data" to turn into free space, the other gems need to abort/commit/logout AND only after 5 minutes that turning into free space will happen? I ask because I am scheduling some batch jobs running at night, and as soon as they all finish, I run GC...and sometimes it seems I do not really GC what I should have...
>>
>> Thanks!
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>> http://lists.gemtalksystems.com/mailman/listinfo/glass <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150806/6be318a6/attachment-0001.html>
More information about the Glass
mailing list