[Glass] ObjectLog cleanup
James Foster via Glass
glass at lists.gemtalksystems.com
Wed Aug 5 14:48:40 PDT 2015
Here is my script...
cleanupObjectLog
| log date1 date2 sortedCopy count |
log := ObjectLogEntry objectLog.
“keep only one per day of various maintenance tasks (useful to go back years on size!)"
#('MTCE: Repository Size' 'MTCE: MFC' 'MTCE: expired sessions') do: [:eachLabel |
log copy do: [:each |
each label = eachLabel ifTrue: [
each stamp date = date1 ifTrue: [
log remove: each.
] ifFalse: [
date1 := each stamp date.
].
].
].
].
“Keep only one per week of anything more than a month old"
date1 := Date today subtractDays: 30.
log copy do: [:each |
(each stamp date < date1 and: [each stamp date day \\ 7 ~= 0]) ifTrue: [
log remove: each.
].
].
“keep only last three days of session expiration"
date1 := Date today subtractDays: 3.
log copy do: [:each |
(each stamp date < date1 and: [each label = 'MTCE: expired sessions']) ifTrue: [
log remove: each.
].
].
“keep only five per day of lock failures"
date1 := nil.
log copy do: [:each |
each stamp date = date1 ifTrue: [
each label = 'Lock not acquired - retrying' ifTrue: [
count < 5 ifTrue: [
count := count + 1.
] ifFalse: [
log remove: each.
].
].
] ifFalse: [
date1 := each stamp date.
count := 0.
].
].
“beyond ten days keep only twenty errors per day"
date1 := nil.
date2 := Date today subtractDays: 10.
log copy do: [:each |
each stamp date = date1 ifTrue: [
each priority == 2 ifTrue: [
count < 20 ifTrue: [
count := count + 1.
] ifFalse: [
date1 < date2 ifTrue: [
log remove: each.
].
].
].
] ifFalse: [
date1 := each stamp date.
count := 0.
].
].
sortedCopy := log asSortedCollection: [:a :b |
a priority > b priority or: [
a priority = b priority and: [a stamp < b stamp].
].
].
log
size: 0;
addAll: sortedCopy asOrderedCollection;
yourself.
%
> On Aug 5, 2015, at 2:33 PM, Mariano Martinez Peck via Glass <glass at lists.gemtalksystems.com> wrote:
>
>
>
> On Wed, Aug 5, 2015 at 5:41 PM, Dale Henrichs via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>
>
> On 07/31/2015 08:58 AM, Trussardi Dario Romano via Glass wrote:
>>
>> James,
>>
>>>
>>>> On Jul 31, 2015, at 3:37 AM, Trussardi Dario Romano via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>>
>>>> Ciao,
>>>>
>>>> i have a deployment system based on GemStone version '3.1.0.6'
>>>>
>>>>
>>>> Now i use Gemtools and todeClient to do some works on the repository: update code, save repository .....
>>>>
>>>>
>>>> When i login with the tools i note the repository grows significantly, and also,
>>>> the relative full backups grows significantly.
>>>>
>>>> To reset the system i need to clear the Object Log with the relative clear command ( ol clear
>>>> in tode )
>>>
>>> Do you think that the Object Log grows significantly during your time logged in with the tools? What is its size at the beginning and end of your session?
>>
>> I don't have real data about the extent0.dbf change size from beginning and end of a gemtools session.
>>
>> But the full backup size before Object Log clear is : 4 258 529280
>>
>> after Object Log clear is: 1 376 124 928
>>
>>
>> The SystemRepository freeSpace size before Object Log clear is : 1 092 419 584
>>
>> after Object Log clear ( and one hour of system background works ) is: 5 267 701 760
>>
>
> Okay as James has mentioned, it definitely looks like the Object Log is the source of the "excess data" ... it would be interesting to understand what is in the object log ... My guess would be that you are having recurring errors and the continuations created to record those errors can easily cause a bunch of data to be stored in your extent ... if the errors are providing useful data, than this can be viewed as a "cost of doing business" and you will just need to have a regularly scheduled task for clearing the object log ... you could arrange to regularly clear continuation entries from the object log that are more than a day or week or ? old then at least you would expect to reach a steady state for extent size ... then you would only need to consider shrinking the size of the extent when you had an anomaly where a series of unexpected errors cropped up ....
>
>
> I do exactly that as part of my daily cleaning:
>
>
> cleanObjectLogEntriesDaysAgo: aNumberOfDays
> | log |
> log := self objectLogEntries: true.
> (log select: [:ea | (Date today - ea stamp asDate) asDays >= aNumberOfDays ]) do: [:ea | log remove: ea ifAbsent: []].
> System commitTransaction.
>
>
> objectLogEntries: shouldLock
>
> shouldLock
> ifTrue: [
> System writeLock: ObjectLogEntry objectQueue
> ifDenied: [
> Transcript show: 'ObjectLogEntry objectQueue lock denied'.
> ^ nil
> ]
> ifChanged: [
> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue.
> Transcript show: 'ObjectLogEntry objectQueue lock dirty'.
> ^ nil
> ].
> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue].
> ^ObjectLogEntry objectLog
>
>
> Maybe we could add this directly to class side of ObjectLogEntry so that others can benefit from it?
>
>
>
> As Mariano and James point out, there are other possible sources of extent growth that are related to a commit record backlog because you have a GemTools/tODE/topaz session open (and idle) on a production system that is "busy committing" ... commit record backlog data is transient data that is stored in the extent and will cause the extent to grow, but once the session or sessions causing the commit record backlog log out commit or abort, the transient data is no longer needed by the system and will "magically turn into free space" at checkpoint boundaries ....
>
> The key point here is that object log data is persistent "user data" and will not turn into free space until 1) you break the link to the persistent root (object log clear) and 2) you run an MFC. Commit record backlog data "system data" and will turn into free space as soon as the sessions abort/commit/logout and a checkpoint is hit, i.e., no MFC is needed ... The default checkpoint interval is 5 minutes I think ...
>
>
> Dale, it is not clear to me what the checkpoint interval is. I understood it was at abort/commit/logout..so how is this internal related? Is there a way I can check the stone configuration parameter of this (the 5 minutes)?
>
> So for the "system data" to turn into free space, the other gems need to abort/commit/logout AND only after 5 minutes that turning into free space will happen? I ask because I am scheduling some batch jobs running at night, and as soon as they all finish, I run GC...and sometimes it seems I do not really GC what I should have...
>
> Thanks!
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150805/9eaf8c71/attachment-0001.html>
More information about the Glass
mailing list