[Glass] instances not being garbage collected?

Dale Henrichs dale.henrichs at gemtalksystems.com
Thu Jan 23 07:11:22 PST 2014 

Johan,

I'll check around today and see what I can find out ...

Dale


On Wed, Jan 22, 2014 at 11:19 AM, James Foster <
james.foster at gemtalksystems.com> wrote:

> Johan,
>
> The conditions for RC cleanup are a bit vague and not well documented. It
> looks to me like prior to an MFC you should stop all user sessions and run
> the cleanup but that doesn’t seem completely reasonable. In any case, it
> explains the earlier mystery!
>
> James
>
> On Jan 22, 2014, at 10:11 AM, Johan Brichau <johan at yesplan.be> wrote:
>
> > Hi James,
> >
> > Are there particular ways to detect when I would need to send
> #cleanupBag?
> > In this case, I noticed the problem because the 'stuck object' was
> keeping a lot of other objects from being garbage collected. But if there
> are just objects that are stuck in these collections, they might suddenly
> pop up again?
> >
> > Johan
> >
> > On 22 Jan 2014, at 06:29, James Foster <james.foster at gemtalksystems.com>
> wrote:
> >
> >> Johan,
> >>
> >> While this seems weird, it can actually make sense. If you watch
> http://www.youtube.com/watch?v=1rb_A8hwKmc you will see part of a lecture
> I gave about RC classes. Consider the following code (run in 3.2 to take
> advantage of the new GsExternalSession). We create an RcIdentityBag, look
> at its contents and the internal components (all empty), add something to
> it in one session, look at the contents and internal components (it appears
> in the collection and in one of the components), remove the contents in
> another session, look at the contents and internal components (it is not in
> the collection but is in two of the components), cleanup, and finally look
> at the contents and components (all empty). The take-away is that the
> components are used to avoid conflict and the object can be "not in the
> collection" but still referenced (at least) twice from the components. This
> is because it is contained in the “additions" for one session and the
> “removals" for another session. RcIdentityBag>>#’cleanupBag’ is the way to
> clean up the components. If you remove the object from the removals then it
> magically reappears in the main collection (since it is still in the
> additions).
> >>
> >>
> >> | rcBag session1 session2 stream block |
> >> rcBag := RcIdentityBag new: 10.
> >> UserGlobals at: #'James' put: rcBag.
> >> stream := WriteStream on: String new.
> >> block := [
> >>      | components |
> >>      components := (rcBag instVarAt: 5) reject: [:each | each isEmpty].
> >>      rcBag asArray printOn: stream.
> >>      stream nextPutAll: ' - '.
> >>      components printOn: stream.
> >>      stream lf.
> >> ].
> >> System commit.
> >> stream lf.
> >> block value.
> >> session1 := GsExternalSession
> >>      gemNRS: '!tcp at vienna#netldi:netldi3#task!gemnetobject'
> >>      stoneNRS: '!tcp at localhost#server!gs64stone3'
> >>      username: 'DataCurator'
> >>      password: 'swordfish'.
> >> session1 login; executeString: 'James add: 42. System commit.'.
> >> System commit.
> >> block value.
> >> session2 := GsExternalSession
> >>      gemNRS: '!tcp at vienna#netldi:netldi3#task!gemnetobject'
> >>      stoneNRS: '!tcp at localhost#server!gs64stone3'
> >>      username: 'DataCurator'
> >>      password: 'swordfish'.
> >> session2 login; executeString: 'James remove: 42. System commit.'.
> >> System commit.
> >> block value.
> >> session1 logout.
> >> session2 logout.
> >> rcBag cleanupBag.
> >> block value.
> >> stream contents.
> >> "
> >> anArray( ) - anArray( )
> >> anArray( 42) - anArray( anIdentityBag( 42))
> >> anArray( ) - anArray( anIdentityBag( 42), anIdentityBag( 42))
> >> anArray( ) - anArray( )
> >> "
> >>
> >> James
> >>
> >> On Jan 21, 2014, at 8:29 AM, Johan Brichau <johan at yesplan.be> wrote:
> >>
> >>> Dale,
> >>>
> >>> Things get even weirder...
> >>>
> >>> First off, the good news is that this instance of
> WAGemStoneServiceTask is the culprit. After nilling the variable holding on
> to our DALi object (which is the start of a CR backlog), all those objects
> got removed from the database in an MFC.
> >>>
> >>> However, the things I saw happening with the RcIdentityBag are very
> strange:
> >>>
> >>> Remember that the instance of WAGemStoneServiceTask was not found in
> the RcIdentityBag but in an IdentityBag that is referenced from this
> RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the
> IdentityBag and ran an mfc. After that, the objects were still there and
> something very weird happened: when I asked for the reference path to the
> instance of WAGemStoneServiceTask, the RcIdentityBag now contained the
> object that I removed. So, now, the reference path looked ok. However,
> after removing it again and running an MFC again, I again get a similar
> situation as it was before: the instance of WAGemStoneServiceTask could be
> found in two instances of the IdentityBag but not in the RcIdentityBag.
> >>>
> >>> I took a backup at each step. For now I will just nil the same
> variable in production so I get rid of the rapid object build up. There is
> a chain of 3 million objects (and their various contents) that gets garbage
> collected at once which fixes my rapid repository growth since Dec 30th
> (which is when the cuplrit object got created).
> >>>
> >>> Johan
> >>>
> >>> On 21 Jan 2014, at 07:55, Johan Brichau <johan at yesplan.be> wrote:
> >>>
> >>>> Dale,
> >>>>
> >>>> I think I found the blocking reference but it does not make sense to
> me.
> >>>>
> >>>> In the path that is returned for this object (let's call it A), I
> notice an RcIdentityBag that is empty. The next object on the reference
> path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and
> that one holds on to a WAGemStoneServiceTask, where a VariableContext is
> holding on to the object A. The object A is part of our DALi framework and
> because it is not garbage collected, it is building up something very
> similar to a CR backlog (hence a lot of objects).
> >>>>
> >>>> The step of the one RcIdentityBag to the Array of IdentityBags is the
> one I do not understand. The RcIdentityBag is the #inProcess bag of
> WAGemStoneServiceTask but this collection is empty. Nevertheless, one of
> the IdentityBags (referenced by the RcIdentityBag) is holding on to the
> WAGemStoneServiceTask instance.
> >>>>
> >>>> I'm now checking to see if this is the main culprit but it definitely
> seems like it.
> >>>>
> >>>> Johan
> >>>>
> >>>> On 20 Jan 2014, at 23:01, Johan Brichau <johan at yesplan.be> wrote:
> >>>>
> >>>>> Hi Dale,
> >>>>>
> >>>>> The objects that stick around are not part of any indexes. But I
> would interested to know what to do with such references too at some point.
> >>>>> However, right now, I have the issue with a set of objects of
> different classes, which are referencing each other (which is logical) but
> their transitive closure is large.
> >>>>>
> >>>>> I am playing around in a copy of the extent, so it went through a
> mfc/backup/restore and several mfc already afterwards with no seaside gems
> running.
> >>>>>
> >>>>> I fiddled around some more, nilling some instance variables part of
> those objects to try to 'cut some references' to prune them.
> >>>>> Some of them are now gone but I still have a long way to go to get
> the real stuck object(s).
> >>>>>
> >>>>> If you think of any places I should search, let me know, in the
> meantime I hope that I will be able to prune the objects enough to get a
> better idea.
> >>>>>
> >>>>> Johan
> >>>>>
> >>>>> On 20 Jan 2014, at 22:47, Dale Henrichs <
> dale.henrichs at gemtalksystems.com> wrote:
> >>>>>
> >>>>>> Johan,
> >>>>>>
> >>>>>> Two thing come to mind right off the bat:
> >>>>>> 1. did these objects participate in an index?
> >>>>>> 2. have you cycled all of your gems recently
> >>>>>>
> >>>>>> If you are using indexes, strong references to objects will be made
> from the DepMap (an ObjectTable-like entity that maps objects to their
> dependents for modification tracking). Objects are managed correctly with
> respect to the DepMap, however, if you drop a reference to a collection
> that has an index without first removing the index, the DepMap will keep
> objects alive. If you are not using indexes the DepMap should be empty.
> >>>>>>
> >>>>>> Long running gems may end up keeping references to otherwise dead
> objects [1], especially if you haven't haven't set
>  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90:
> >>>>>>
> >>>>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90;
> >>>>>>
> >>>>>> in your system.conf. Even with the above, you are not guaranteed to
> prune all of the persistent references on the vote. You should be
> suspicious that you are seeing this particular problem if you consistently
> see a high number of possible dead in your mfc report. Also look at vsd and
> after an mfc you should see the stone stat PossibleDeadObjs jump up to the
> value that was reported when your mfc ran ... Then a short while later
> (after voting is complete) you should see the stone stat
> DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less
> than PossibleDeadObjs, then those objects were "voted down" because one or
> more live gems had a reference to the object in its head. If you've already
> got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of
> your gems and run another mfc ....
> >>>>>>
> >>>>>> If your PossibleDeadObjs from an mfc is not as high as you think it
> should be, then I will have to do some more research on finding out if
> there are any other "hidden" references to objects ...
> >>>>>>
> >>>>>> Dale
> >>>>>>
> >>>>>>
> >>>>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <johan at yesplan.be>
> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> My repository has a (large) number of objects that are no longer
> referenced from the persistent root but that are never garbage collected.
> >>>>>>
> >>>>>> When I do:
> >>>>>>
> >>>>>>    ClassXX allInstances
> >>>>>>
> >>>>>> I get a number of instances but when I then do for an instance i:
> >>>>>>
> >>>>>>    SystemRepository findReferencePathForObject: i
> >>>>>>
> >>>>>> The answer is that there is no path (aka 'false' as the second
> element in the array that is returned).
> >>>>>>
> >>>>>> Any ideas how I can investigate this?
> >>>>>>
> >>>>>> thx
> >>>>>> Johan
> >>>>>> _______________________________________________
> >>>>>> Glass mailing list
> >>>>>> Glass at lists.gemtalksystems.com
> >>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> Glass mailing list
> >>> Glass at lists.gemtalksystems.com
> >>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>
> >
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140123/be26263a/attachment-0001.html>

More information about the Glass mailing list