[Glass] [GLASS] Seaside - growing extent - normal?
Lawrence Kellogg via Glass
glass at lists.gemtalksystems.com
Fri Aug 7 13:46:28 PDT 2015
Hello,
So, perhaps I will be forced to finally get a handle on my problem with a growing extent.
Something funny happened a couple of weeks ago that I thought I would share.
I needed a little backend service to do some testing with an iPhone app so I wrote it in Gemstone.
I wrote a simple queue mechanism that uses a few RcQueues that are accessible from a call to a WAComponent. That’s it.
Well, my iPhone app polls the back-end very fifteen seconds or so, calling this WAComponent.
I found that my extent ballooned from 36 gig to over 60 gig and filled up the whole file system!
Can we deduce anything as a result of my experience? Is the Seaside session stuff taking up all the space in the extent?
I’ll dig up those steps to try to shrink the extent but I don’t remember having much luck.
Best,
Larry
> On Mar 30, 2015, at 3:23 PM, Lawrence Kellogg <mac.hive at me.com> wrote:
>
>
>> On Mar 30, 2015, at 2:06 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
>>
>> A couple of things:
>>
>> 1. You appear to be doing rpc logins when using topaz? You
>> should be able to login in using a linking login by using the -l
>> option for topaz ... you will need to do this on the machine that
>> is running .. linked logins bypass the netldi
>>
>
>
>
>
>> 2. I suspect that you have started the netldi, without using the -g
>> option ... so double check how you started the netldi ...
>>
>
> Yesh, I don’t know what the -g option does so I
>
>
>> Dale
>> On 03/30/2015 10:06 AM, Lawrence Kellogg wrote:
>>> Thanks Dale, I’ll try again, but, unfortunately, I can no longer log into GemTools. I get this error:
>>>
>>> 'Unable to create a GemStone session.
>>> Netldi ''gs64ldi'' on host ''ip-10-32-103-83'' reports the request ''gemnetobject'' failed:
>>> Password validation failed for user seasideuser because getspnam() returned an error: errno=13,EACCES, Authorization failure (permission denied)’
>>>
>>> Did the restore somehow change the password on the glass login in the launcher?
>>>
>>> I can’t log into Topaz either and I did in order to do the restore:
>>>
>>> |_____________________________________________________________________________|
>>> topaz> set username DataCurator
>>> topaz> set gemstone seaside
>>> topaz> set password swordfish
>>> topaz> login
>>> -----------------------------------------------------
>>> GemStone: Error Fatal
>>> Unable to create a GemStone session.
>>> Netldi 'gs64ldi' on host 'ip-10-32-103-83' reports the request 'gemnetobject'
>>> failed:
>>> Password validation failed for user seasideuser because getspnam()
>>> returned an error: errno=13,EACCES, Authorization failure (permission
>>> denied)
>>> Error Category: [GemStone] Number: 4042 Arg Count: 0
>>>
>>>
>>> Larry
>>>
>>>> On Mar 30, 2015, at 12:28 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>>
>>>> Okay,
>>>>
>>>> I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...
>>>>
>>>> So we need to find out what's going on in the new extent ... so do another mfc and send me the results
>>>>
>>>> In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...
>>>>
>>>> By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...
>>>>
>>>> Dale
>>>>
>>>> On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
>>>>> Hello Dale,
>>>>>
>>>>> Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled.
>>>>> Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?
>>>>>
>>>>>
>>>>> <Mail Attachment.png>
>>>>>
>>>>> After the ObjectLog init, and mark, I did a restore into a fresh extent.
>>>>>
>>>>> Here is the size of the new extent vs the old, saved extent:
>>>>>
>>>>> <Mail Attachment.png>
>>>>>
>>>>>
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Larry
>>>>>
>>>>>
>>>>>
>>>>>> On Mar 25, 2015, at 2:15 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>>>>
>>>>>> Okay here's the sequence of steps that I think you should take:
>>>>>>
>>>>>> 1. expire all of your sessions:
>>>>>>
>>>>>> | expired |
>>>>>> Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
>>>>>> expired := WABasicDevelopment reapSeasideCache.
>>>>>> expired > 0
>>>>>> ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
>>>>>> Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
>>>>>> System commitTransactions
>>>>>>
>>>>>> 2. initalize your object log
>>>>>>
>>>>>> 3. run MFC
>>>>>>
>>>>>> [
>>>>>> System abortTransaction.
>>>>>> SystemRepository markForCollection ]
>>>>>> on: Warning
>>>>>> do: [ :ex |
>>>>>> Transcript
>>>>>> cr;
>>>>>> show: ex description.
>>>>>> ex resume ]
>>>>>>
>>>>>> 4. Then do a backup and restore ... you can use GemTools to do the restore,
>>>>>> but then you should read the SysAdmin docs[1] for instructions to do the restore
>>>>>> (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
>>>>>> much be the same, but it's best to look up the docs for your GemStone version[2]
>>>>>> and follow those instructions)
>>>>>>
>>>>>> As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).
>>>>>>
>>>>>> Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...
>>>>>>
>>>>>> Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...
>>>>>>
>>>>>> Dale
>>>>>>
>>>>>> [1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm <http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm>
>>>>>> [2] http://gemtalksystems.com/techsupport/resources/ <http://gemtalksystems.com/techsupport/resources/>
>>>>>>
>>>>>> On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
>>>>>>> Hello Dale,
>>>>>>>
>>>>>>> Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL.
>>>>>>>
>>>>>>> I’m not running the maintenance VM and I haven’t been doing regular mark for collects.
>>>>>>>
>>>>>>> I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup?
>>>>>>>
>>>>>>> I’ll also try the ObjectLog trick.
>>>>>>>
>>>>>>> I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Larry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>>>>>>
>>>>>>>> Lawrence,
>>>>>>>>
>>>>>>>> Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?
>>>>>>>>
>>>>>>>> Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...
>>>>>>>>
>>>>>>>> Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...
>>>>>>>>
>>>>>>>> The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...
>>>>>>>>
>>>>>>>> Dale
>>>>>>>>
>>>>>>>> On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
>>>>>>>>> Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.
>>>>>>>>>
>>>>>>>>> I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal?
>>>>>>>>>
>>>>>>>>> --- 03/07/2015 02:44:14 PM UTC ---
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22528 megabytes.
>>>>>>>>> Repository has grown to 22528 megabytes.
>>>>>>>>>
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22544 megabytes.
>>>>>>>>> Repository has grown to 22544 megabytes.
>>>>>>>>>
>>>>>>>>> --- 03/08/2015 03:31:45 PM UTC ---
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22560 megabytes.
>>>>>>>>> Repository has grown to 22560 megabytes.
>>>>>>>>>
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22576 megabytes.
>>>>>>>>> Repository has grown to 22576 megabytes.
>>>>>>>>>
>>>>>>>>> --- 03/10/2015 03:19:34 AM UTC ---
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22592 megabytes.
>>>>>>>>> Repository has grown to 22592 megabytes.
>>>>>>>>>
>>>>>>>>> --- 03/10/2015 03:46:39 PM UTC ---
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22608 megabytes.
>>>>>>>>> Repository has grown to 22608 megabytes.
>>>>>>>>>
>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>> has grown to 22624 megabytes.
>>>>>>>>> Repository has grown to 22624 megabytes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My extent has now grown to
>>>>>>>>>
>>>>>>>>> -rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?
>>>>>>>>>
>>>>>>>>> I suppose I should also do a SystemRepository backup, if I can remember the commands.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Larry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Glass mailing list
>>>>>>>>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>>>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Glass mailing list
>>>>>>>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150807/e62463d3/attachment-0001.html>
More information about the Glass
mailing list