[Glass] ObjectLog cleanup

Thu Aug 6 09:12:44 PDT 2015

>
>
>
>> “Keep only one per week of anything more than a month old"
>>   date1 := Date today subtractDays: 30.
>>   log copy do: [:each |
>>   (each stamp date < date1 and: [each stamp date day \\ 7 ~= 0]) ifTrue:
>> [
>>   log remove: each.
>>   ].
>>   ].
>>
>>
>
> This is the part I understand the less. Why would make sense to keep one
> random entry once per week of things older than a month?
> I mean...a random entry will probably tell me anything. In which case do
> you think this could be useful?
>
>
> The comment should say “Keep only one DAY per week of anything more than
> month old”.
>
> This is an attempt to balance keeping everything and keeping nothing (and
> I don’t claim to have the perfect balance!). Beyond 30 days, for things
> like MFC reports, four per month seems quite adequate to give an idea of
> the trend. For things like errors, if you haven’t looked at it in 30 days
> then some pruning seems appropriate. I figured that picking one random day
> per week gives an idea if an error is common or rare.
>
>
Ufff now it totally makes sense :)  I also misread the code hahahahhaa.

  sortedCopy := log asSortedCollection: [:a :b |
>>   a priority > b priority or: [
>>   a priority = b priority and: [a stamp < b stamp].
>>   ].
>>   ].
>>   log
>>   size: 0;
>>   addAll: sortedCopy asOrderedCollection;
>> yourself.
>>
>
> A couple of questions:
>
> 1) Don't you need to get the lock (as I did in my script) before trying to
> write the object log?
>
>
> Good catch. In the few applications where I’ve used this script the users
> are active during the day and the maintenance (including this script) run
> during the night. Thus, I haven’t had a locking problem (that I know of!).
> But, yes, I should follow your example of getting the lock.
>

Great. I also run it at night...but I also may have some background
processes running which do Transcript show: and hence end up writing the
object log. So...just to minimize chances of a commit conflict that would
prevent the cleaning.

>
> 2) I also have some Transcript show (priority 7), some inspect/explore
> (priority 8) and some trace:  (priority 6).  I guess I should add a similar
> part for this...not sure how many days though…
>
>
> I think that the above will trim your other things to one day per week.
> From there you could reduce the count as I’ve done with errors.
>
>
Indeed. I would let that for priority 6 and 7. But...for inspect/explore as
this is mostly a developer-oriented thing, I would expect they do not live
long as the developer should have probably finished working with that just
a couple of hours later. And the inspected object may hold quite a lot of
stuff (it's not a string as the case of transcript and trace). So... I
would keep these guys only for....say... 2 days :)

Thoughts?

>
> Thanks in advance,
>
>
> I appreciate Dale’s suggestion of combining the best of the scripts and
> adding it to the framework.
>
>
Yes, I would do that. However...I do not expect this code to be THE used
one for everybody. But if at least this method is stored in class side of
ObjectLogEntry with a selector like #objectLogCleanupExampleScript or
something, that would be nice. Some may use it directly, and others would
write their own script probably based on the idea of this one.

What do you think?

> James
>
>
>
>>
> %
>>
>> On Aug 5, 2015, at 2:33 PM, Mariano Martinez Peck via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>>
>>
>> On Wed, Aug 5, 2015 at 5:41 PM, Dale Henrichs via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>>>
>>>
>>> On 07/31/2015 08:58 AM, Trussardi Dario Romano via Glass wrote:
>>>
>>>
>>> James,
>>>
>>>
>>> On Jul 31, 2015, at 3:37 AM, Trussardi Dario Romano via Glass <
>>> glass at lists.gemtalksystems.com> wrote:
>>>
>>> Ciao,
>>>
>>> i have a deployment system based on GemStone version '3.1.0.6'
>>>
>>>
>>> Now i use Gemtools and todeClient to do some works on the repository:
>>>  update code,  save repository .....
>>>
>>>
>>> When i login with the tools  i note the repository grows significantly,  and
>>> also, the relative  full backups grows significantly.
>>>
>>> To reset the system i need to clear the Object Log with the relative
>>> clear command  ( ol clear in tode )
>>>
>>>
>>> Do you think that the Object Log grows significantly during your time
>>> logged in with the tools? What is its size at the beginning and end of your
>>> session?
>>>
>>>
>>> I don't have real data about the extent0.dbf change size from beginning
>>> and end of a gemtools session.
>>>
>>> But the full backup size  before Object Log  clear is :  4 258 529280
>>>
>>> after Object Log clear is: 1 376 124 928
>>>
>>> The SystemRepository freeSpace size before Object Log clear is :  1 092
>>> 419 584
>>>
>>> after Object Log clear ( and one hour  of system background works )  is:
>>>  5 267 701 760
>>>
>>>
>>> Okay as James has mentioned, it definitely looks like the Object Log is
>>> the source of the "excess data" ... it would be interesting to understand
>>> what is in the object log ... My guess would be that you are having
>>> recurring errors and the continuations created to record those errors can
>>> easily cause a bunch of data to be stored in your extent ... if the errors
>>> are providing useful data, than this can be viewed as a "cost of doing
>>> business" and you will just need to have a regularly scheduled task for
>>> clearing the object log ... you could arrange to regularly clear
>>> continuation entries from the object log that are more than a day or week
>>> or ? old then at least you would expect to reach a steady state for extent
>>> size ... then you would only need to consider shrinking the size of the
>>> extent when you had an anomaly where a series of unexpected errors cropped
>>> up ....
>>>
>>>
>> I do exactly that as part of my daily cleaning:
>>
>>
>> cleanObjectLogEntriesDaysAgo: aNumberOfDays
>> | log |
>> log := self objectLogEntries: true.
>> (log select: [:ea | (Date today - ea stamp asDate) asDays >=
>> aNumberOfDays ]) do: [:ea | log remove: ea ifAbsent: []].
>> System commitTransaction.
>>
>>
>> objectLogEntries: shouldLock
>>
>> shouldLock
>> ifTrue: [
>> System writeLock: ObjectLogEntry objectQueue
>> ifDenied: [
>> Transcript show: 'ObjectLogEntry objectQueue lock denied'.
>> ^ nil
>> ]
>> ifChanged: [
>> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue.
>> Transcript show: 'ObjectLogEntry objectQueue lock dirty'.
>> ^ nil
>> ].
>> System addToCommitOrAbortReleaseLocksSet: ObjectLogEntry objectQueue].
>> ^ObjectLogEntry objectLog
>>
>>
>> Maybe we could add this directly to class side of ObjectLogEntry so that
>> others can benefit from it?
>>
>>
>>
>>
>>> As Mariano and James point out, there are other possible sources of
>>> extent growth that are related to a commit record backlog because you have
>>> a GemTools/tODE/topaz session open (and idle) on a production system that
>>> is "busy committing" ... commit record backlog data is transient data that
>>> is stored in the extent and will cause the extent to grow, but once the
>>> session or sessions causing the commit record backlog log out commit or
>>> abort, the transient data is no longer needed by the system and will
>>> "magically turn into free space" at checkpoint boundaries ....
>>>
>>> The key point here is that object log data is persistent "user data" and
>>> will not turn into free space until 1) you break the link to the persistent
>>> root (object log clear) and 2) you run an MFC. Commit record backlog data
>>> "system data" and will turn into free space as soon as the sessions
>>> abort/commit/logout and a checkpoint is hit, i.e., no MFC is needed ... The
>>> default checkpoint interval is 5 minutes I think ...
>>>
>>>
>> Dale, it is not clear to me what the checkpoint interval is. I understood
>> it was at abort/commit/logout..so how is this internal related? Is there a
>> way I can check the stone configuration parameter of this (the 5 minutes)?
>>
>> So for the "system data" to turn into free space, the other gems need to
>> abort/commit/logout AND only after 5 minutes that turning into free space
>> will happen?   I ask because I am scheduling some batch jobs running at
>> night, and as soon as they all finish, I run GC...and sometimes it seems I
>> do not really GC what I should have...
>>
>> Thanks!
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
>
>

-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150806/1308a949/attachment-0001.html>