[Glass] instances not being garbage collected?

Dale Henrichs dale.henrichs at gemtalksystems.com
Thu Jan 23 12:08:46 PST 2014 

Johan,

After reading the comment in #cleanupBag it seems to me that the cleanupBag
is something that should be done on a regular basis as part of regular db
maintenance ... the sticky wicket is that the cleanup activity could result
in a commit conflict if the bag is operated on by another session ...so
technically it isn't necessary to shut down the active sessions, but you do
want to ensure that the other sessions remain idle while doing the cleanup
(at least not accessing the bag(s) in question so it isn't a simple matter
of scheduling regular clean up...

In your specific case if you have some off hours production time where at a
minimum you've got the web server inactive, you can schedule the cleanup
during that period ..

Otherwise we need to dig deeper for a solution. I've create a bug[1] for
tracking this issue ... I hate to resort to a lock for this purpose, but a
generic bag maintenance lock may be the ticket ...

Dale

[1] https://github.com/glassdb/glass/issues/17


On Thu, Jan 23, 2014 at 7:11 AM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:

> Johan,
>
> I'll check around today and see what I can find out ...
>
> Dale
>
>
> On Wed, Jan 22, 2014 at 11:19 AM, James Foster <
> james.foster at gemtalksystems.com> wrote:
>
>> Johan,
>>
>> The conditions for RC cleanup are a bit vague and not well documented. It
>> looks to me like prior to an MFC you should stop all user sessions and run
>> the cleanup but that doesn’t seem completely reasonable. In any case, it
>> explains the earlier mystery!
>>
>> James
>>
>> On Jan 22, 2014, at 10:11 AM, Johan Brichau <johan at yesplan.be> wrote:
>>
>> > Hi James,
>> >
>> > Are there particular ways to detect when I would need to send
>> #cleanupBag?
>> > In this case, I noticed the problem because the 'stuck object' was
>> keeping a lot of other objects from being garbage collected. But if there
>> are just objects that are stuck in these collections, they might suddenly
>> pop up again?
>> >
>> > Johan
>> >
>> > On 22 Jan 2014, at 06:29, James Foster <james.foster at gemtalksystems.com>
>> wrote:
>> >
>> >> Johan,
>> >>
>> >> While this seems weird, it can actually make sense. If you watch
>> http://www.youtube.com/watch?v=1rb_A8hwKmc you will see part of a
>> lecture I gave about RC classes. Consider the following code (run in 3.2 to
>> take advantage of the new GsExternalSession). We create an RcIdentityBag,
>> look at its contents and the internal components (all empty), add something
>> to it in one session, look at the contents and internal components (it
>> appears in the collection and in one of the components), remove the
>> contents in another session, look at the contents and internal components
>> (it is not in the collection but is in two of the components), cleanup, and
>> finally look at the contents and components (all empty). The take-away is
>> that the components are used to avoid conflict and the object can be "not
>> in the collection" but still referenced (at least) twice from the
>> components. This is because it is contained in the “additions" for one
>> session and the “removals" for another session.
>> RcIdentityBag>>#’cleanupBag’ is the way to clean up the components. If you
>> remove the object from the removals then it magically reappears in the main
>> collection (since it is still in the additions).
>> >>
>> >>
>> >> | rcBag session1 session2 stream block |
>> >> rcBag := RcIdentityBag new: 10.
>> >> UserGlobals at: #'James' put: rcBag.
>> >> stream := WriteStream on: String new.
>> >> block := [
>> >>      | components |
>> >>      components := (rcBag instVarAt: 5) reject: [:each | each isEmpty].
>> >>      rcBag asArray printOn: stream.
>> >>      stream nextPutAll: ' - '.
>> >>      components printOn: stream.
>> >>      stream lf.
>> >> ].
>> >> System commit.
>> >> stream lf.
>> >> block value.
>> >> session1 := GsExternalSession
>> >>      gemNRS: '!tcp at vienna#netldi:netldi3#task!gemnetobject'
>> >>      stoneNRS: '!tcp at localhost#server!gs64stone3'
>> >>      username: 'DataCurator'
>> >>      password: 'swordfish'.
>> >> session1 login; executeString: 'James add: 42. System commit.'.
>> >> System commit.
>> >> block value.
>> >> session2 := GsExternalSession
>> >>      gemNRS: '!tcp at vienna#netldi:netldi3#task!gemnetobject'
>> >>      stoneNRS: '!tcp at localhost#server!gs64stone3'
>> >>      username: 'DataCurator'
>> >>      password: 'swordfish'.
>> >> session2 login; executeString: 'James remove: 42. System commit.'.
>> >> System commit.
>> >> block value.
>> >> session1 logout.
>> >> session2 logout.
>> >> rcBag cleanupBag.
>> >> block value.
>> >> stream contents.
>> >> "
>> >> anArray( ) - anArray( )
>> >> anArray( 42) - anArray( anIdentityBag( 42))
>> >> anArray( ) - anArray( anIdentityBag( 42), anIdentityBag( 42))
>> >> anArray( ) - anArray( )
>> >> "
>> >>
>> >> James
>> >>
>> >> On Jan 21, 2014, at 8:29 AM, Johan Brichau <johan at yesplan.be> wrote:
>> >>
>> >>> Dale,
>> >>>
>> >>> Things get even weirder...
>> >>>
>> >>> First off, the good news is that this instance of
>> WAGemStoneServiceTask is the culprit. After nilling the variable holding on
>> to our DALi object (which is the start of a CR backlog), all those objects
>> got removed from the database in an MFC.
>> >>>
>> >>> However, the things I saw happening with the RcIdentityBag are very
>> strange:
>> >>>
>> >>> Remember that the instance of WAGemStoneServiceTask was not found in
>> the RcIdentityBag but in an IdentityBag that is referenced from this
>> RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the
>> IdentityBag and ran an mfc. After that, the objects were still there and
>> something very weird happened: when I asked for the reference path to the
>> instance of WAGemStoneServiceTask, the RcIdentityBag now contained the
>> object that I removed. So, now, the reference path looked ok. However,
>> after removing it again and running an MFC again, I again get a similar
>> situation as it was before: the instance of WAGemStoneServiceTask could be
>> found in two instances of the IdentityBag but not in the RcIdentityBag.
>> >>>
>> >>> I took a backup at each step. For now I will just nil the same
>> variable in production so I get rid of the rapid object build up. There is
>> a chain of 3 million objects (and their various contents) that gets garbage
>> collected at once which fixes my rapid repository growth since Dec 30th
>> (which is when the cuplrit object got created).
>> >>>
>> >>> Johan
>> >>>
>> >>> On 21 Jan 2014, at 07:55, Johan Brichau <johan at yesplan.be> wrote:
>> >>>
>> >>>> Dale,
>> >>>>
>> >>>> I think I found the blocking reference but it does not make sense to
>> me.
>> >>>>
>> >>>> In the path that is returned for this object (let's call it A), I
>> notice an RcIdentityBag that is empty. The next object on the reference
>> path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and
>> that one holds on to a WAGemStoneServiceTask, where a VariableContext is
>> holding on to the object A. The object A is part of our DALi framework and
>> because it is not garbage collected, it is building up something very
>> similar to a CR backlog (hence a lot of objects).
>> >>>>
>> >>>> The step of the one RcIdentityBag to the Array of IdentityBags is
>> the one I do not understand. The RcIdentityBag is the #inProcess bag of
>> WAGemStoneServiceTask but this collection is empty. Nevertheless, one of
>> the IdentityBags (referenced by the RcIdentityBag) is holding on to the
>> WAGemStoneServiceTask instance.
>> >>>>
>> >>>> I'm now checking to see if this is the main culprit but it
>> definitely seems like it.
>> >>>>
>> >>>> Johan
>> >>>>
>> >>>> On 20 Jan 2014, at 23:01, Johan Brichau <johan at yesplan.be> wrote:
>> >>>>
>> >>>>> Hi Dale,
>> >>>>>
>> >>>>> The objects that stick around are not part of any indexes. But I
>> would interested to know what to do with such references too at some point.
>> >>>>> However, right now, I have the issue with a set of objects of
>> different classes, which are referencing each other (which is logical) but
>> their transitive closure is large.
>> >>>>>
>> >>>>> I am playing around in a copy of the extent, so it went through a
>> mfc/backup/restore and several mfc already afterwards with no seaside gems
>> running.
>> >>>>>
>> >>>>> I fiddled around some more, nilling some instance variables part of
>> those objects to try to 'cut some references' to prune them.
>> >>>>> Some of them are now gone but I still have a long way to go to get
>> the real stuck object(s).
>> >>>>>
>> >>>>> If you think of any places I should search, let me know, in the
>> meantime I hope that I will be able to prune the objects enough to get a
>> better idea.
>> >>>>>
>> >>>>> Johan
>> >>>>>
>> >>>>> On 20 Jan 2014, at 22:47, Dale Henrichs <
>> dale.henrichs at gemtalksystems.com> wrote:
>> >>>>>
>> >>>>>> Johan,
>> >>>>>>
>> >>>>>> Two thing come to mind right off the bat:
>> >>>>>> 1. did these objects participate in an index?
>> >>>>>> 2. have you cycled all of your gems recently
>> >>>>>>
>> >>>>>> If you are using indexes, strong references to objects will be
>> made from the DepMap (an ObjectTable-like entity that maps objects to their
>> dependents for modification tracking). Objects are managed correctly with
>> respect to the DepMap, however, if you drop a reference to a collection
>> that has an index without first removing the index, the DepMap will keep
>> objects alive. If you are not using indexes the DepMap should be empty.
>> >>>>>>
>> >>>>>> Long running gems may end up keeping references to otherwise dead
>> objects [1], especially if you haven't haven't set
>>  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90:
>> >>>>>>
>> >>>>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90;
>> >>>>>>
>> >>>>>> in your system.conf. Even with the above, you are not guaranteed
>> to prune all of the persistent references on the vote. You should be
>> suspicious that you are seeing this particular problem if you consistently
>> see a high number of possible dead in your mfc report. Also look at vsd and
>> after an mfc you should see the stone stat PossibleDeadObjs jump up to the
>> value that was reported when your mfc ran ... Then a short while later
>> (after voting is complete) you should see the stone stat
>> DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less
>> than PossibleDeadObjs, then those objects were "voted down" because one or
>> more live gems had a reference to the object in its head. If you've already
>> got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of
>> your gems and run another mfc ....
>> >>>>>>
>> >>>>>> If your PossibleDeadObjs from an mfc is not as high as you think
>> it should be, then I will have to do some more research on finding out if
>> there are any other "hidden" references to objects ...
>> >>>>>>
>> >>>>>> Dale
>> >>>>>>
>> >>>>>>
>> >>>>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <johan at yesplan.be>
>> wrote:
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> My repository has a (large) number of objects that are no longer
>> referenced from the persistent root but that are never garbage collected.
>> >>>>>>
>> >>>>>> When I do:
>> >>>>>>
>> >>>>>>    ClassXX allInstances
>> >>>>>>
>> >>>>>> I get a number of instances but when I then do for an instance i:
>> >>>>>>
>> >>>>>>    SystemRepository findReferencePathForObject: i
>> >>>>>>
>> >>>>>> The answer is that there is no path (aka 'false' as the second
>> element in the array that is returned).
>> >>>>>>
>> >>>>>> Any ideas how I can investigate this?
>> >>>>>>
>> >>>>>> thx
>> >>>>>> Johan
>> >>>>>> _______________________________________________
>> >>>>>> Glass mailing list
>> >>>>>> Glass at lists.gemtalksystems.com
>> >>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>> _______________________________________________
>> >>> Glass mailing list
>> >>> Glass at lists.gemtalksystems.com
>> >>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> >>
>> >
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140123/297c667a/attachment-0001.html>

More information about the Glass mailing list