[Glass] instances not being garbage collected?

[Johan Brichau]

Dale,

Things get even weirder...

First off, the good news is that this instance of WAGemStoneServiceTask is the culprit. After nilling the variable holding on to our DALi object (which is the start of a CR backlog), all those objects got removed from the database in an MFC.

However, the things I saw happening with the RcIdentityBag are very strange: 

Remember that the instance of WAGemStoneServiceTask was not found in the RcIdentityBag but in an IdentityBag that is referenced from this RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the IdentityBag and ran an mfc. After that, the objects were still there and something very weird happened: when I asked for the reference path to the instance of WAGemStoneServiceTask, the RcIdentityBag now contained the object that I removed. So, now, the reference path looked ok. However, after removing it again and running an MFC again, I again get a similar situation as it was before: the instance of WAGemStoneServiceTask could be found in two instances of the IdentityBag but not in the RcIdentityBag. 

I took a backup at each step. For now I will just nil the same variable in production so I get rid of the rapid object build up. There is a chain of 3 million objects (and their various contents) that gets garbage collected at once which fixes my rapid repository growth since Dec 30th (which is when the cuplrit object got created).

Johan

On 21 Jan 2014, at 07:55, Johan Brichau <johan at yesplan.be> wrote:

> Dale,
> 
> I think I found the blocking reference but it does not make sense to me.
> 
> In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects).
> 
> The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance.
> 
> I'm now checking to see if this is the main culprit but it definitely seems like it.
> 
> Johan
> 
> On 20 Jan 2014, at 23:01, Johan Brichau <johan at yesplan.be> wrote:
> 
>> Hi Dale,
>> 
>> The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point.
>> However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large.
>> 
>> I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running.
>> 
>> I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them.
>> Some of them are now gone but I still have a long way to go to get the real stuck object(s).
>> 
>> If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea.
>> 
>> Johan
>> 
>> On 20 Jan 2014, at 22:47, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
>> 
>>> Johan,
>>> 
>>> Two thing come to mind right off the bat:
>>> 1. did these objects participate in an index?
>>> 2. have you cycled all of your gems recently
>>> 
>>> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty.
>>> 
>>> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90:
>>> 
>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90;
>>> 
>>> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc ....
>>> 
>>> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ...
>>> 
>>> Dale
>>> 
>>> 
>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136
>>> 
>>> 
>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <johan at yesplan.be> wrote:
>>> Hi,
>>> 
>>> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected.
>>> 
>>> When I do:
>>> 
>>>       ClassXX allInstances
>>> 
>>> I get a number of instances but when I then do for an instance i:
>>> 
>>>       SystemRepository findReferencePathForObject: i
>>> 
>>> The answer is that there is no path (aka 'false' as the second element in the array that is returned).
>>> 
>>> Any ideas how I can investigate this?
>>> 
>>> thx
>>> Johan
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> 
>> 
>