[Glass] GLASS performance & cleanup scripts

Wed Dec 4 09:15:06 PST 2013

On Dec 4, 2013, at 5:37 PM, leodm <leo at smalltalking.net> wrote:

> James, 
> 
> I'm back :)

Good!

> I have made some test based on your post...
> 
>> Welcome to GemStone. 
> Thnks! I'm happy I can finally try it.
> 
>> First, as to configuring the Shared Page Cache (SCP), the rule is simple:
>> get the most that you can afford (up to slightly more than the repository
>> size). With the free license the maximum size is 2 GB, but that should be
>> pretty good for your database.
> OK, so I change (in testing) the /etc/sysctl.conf file like this according
> to 2GB free license limit
> kernel.shmmax = 1610612736
> kernel.shmall = 393216

These appear to total ~1.5GB, so you could push it a bit more if you wanted.

> I assume I must not change SHR_PAGE_CACHE_SIZE_KB parameter because is the
> extent page chache size, actually is 128000

That is 128MB, far short of the maximum. You could easily add a zero to the end and have 1.28 GB. (The SPC needs to be slightly smaller than the allowed shared memory since the system allocates other shared space besides the page cache.)

>> The free license does not have a limit on the database size, so you can
>> keep growing.
> Perfect.
> 
>> The #'fullBackupCompressedTo' method does an object-by-object backup that
>> is not identical to the extent0.dbf file in layout (it has all the objects,
>> but is typically more compact since the extents can have free space). The
>> alternative is to suspend checkpoints, copy the extent(s), and resume
>> checkpoints. That will give you an extent backup that can be used directly
>> as a repository. Unless you have reason to prefer the extent-file-copy
>> approach, I suggest you stick with the existing backup (and practice a
>> restore occasionally).
> Ok, I prefer the object-by-object backup approach. 
> One question: When I make a #fullBackupCompressedTo, I can restore with the
> Gemstools "restore w/o tranlog" option?

I think so, but I don’t use GemTools much. Are you using Linux exclusively, or Linux in a VM on Windows, or Mac, or …?

> Or how can I make a restore from a
> compress database (I prefer the command line, but any help is ok)

To use the command line you would use Topaz. There is an entire chapter in the System Administration Guide on backup/restore. I haven’t tested the following code, but from memory it would be something like the following (in Topaz comments begin with the ‘!’ character):

$ topaz -l << EOF
set user DataCurator pass swordfish gemstone gs64stone
login
run
SystemRepository restoreFromBackup: ‘/path/to/my/backup.bak.gz’
%
! this should give an “error” (logging you out) reporting the successful restore 
login
send SystemRepository restoreStatus
! this should report that it is waiting for transaction logs
! if you don’t want to restore any logs, then commits as follows
send SystemRepository commitRestore
! this should give an “error” (logging you out) and report success
logout
exit

>> Often one will choose to start a new transaction log just before taking a
>> backup. Then you don't have to wonder which transaction logs are needed to
>> go with a particular backup. I have seen scripts that delete transaction
>> logs that are older than the oldest backup that you would use for disaster
>> recovery (and you typically keep at least two backups).
> Ok, thats great. If I make a good backup I have no worry about tranlogs.
> 
>> Strategies for making reports faster typically involve indexes and/or
>> clustering. You can read about them in the programming guide.
> Ok, I began reading about it. So next I'm going to check is this.
> 
>> Keep asking questions and let us help you get the system running smoothly.
>> Of course, at some point I might suggest you pay for consulting and/or a
>> license to have a larger Shared Page Cache. ;-)
> Yes of course! I assume that's the reason I'm not gain more performance with
> more RAM (maybe the only thing is with a SSD disk). 
> I think is more convenient to pay for consulting here, because is more
> bourocratic to get money for licenses... I'm going to send you a private
> email for this.
> 
> Thnks!
> Leo
> 
> 
> 
> 
> On Fri, Nov 29, 2013 at 4:12 PM, <leo@> wrote:
> 
>> Hi all!
>> 
>> Recently I have began to work with GLASS, I have experience working with
>> smalltalk but no experience working with Gemstone & Linux, so be patient
>> with me :)
>> 
>> My client already have a GLASS system working on production enviroment,
>> but with some performance problems when they perform large reports, so I
>> plan a strategy of 3 stages:
>> 1) Review the hardware
>> 2) Review the Server & Gemstone configuration
>> 3) Review the code
>> 
>> The first issue, the hardware, already check and I think its ok for a 15GB
>> repository and only 10 concurrent users. Is composed of a
>> -#HP ProLiant ML350 G5
>> -#8 CPU Cores (Intel Xeon E5405 @ 2Ghz)
>> -#8 GB RAM
>> -#3 146GB disks in RAID 5 (for storage) (bay 1-2-3)
>> 
>> The only thing here that I read in some post is changing the disk drives
>> to SSD but is not a quick decision here where I am :)
>> 
>> The second one, is where actually I am. I change the host SO, I mounted a
>> VMWare ESXi and then mounted a GemStone/S 64 Bit in a VMware Appliance to
>> it. Then I added a 2 more spindles for the repository and the tranlogs and
>> also increase the swap partition (see image attached).
>> 
>> Questions:
>> -My repository size & concurrent users dont fit to pre-Selected
>> configuration (repository size is for a Medium conf and users fit with
>> Small one), so what is a recommended main variables for the config file
>> according to my repository values?
>> 
>> -*Repository size*: My repository is 15 GB, I read somewhere that 16 GB
>> is a limit. Why is that? Is recommended in these case to split in 2
>> repositories?
>> 
>> -*Estimating Shared Page Cache issue*: I read here:
>> http://programminggems.wordpress.com/2012/04/06/configuring-shared-memory/
>> and
>> then read also the Gemstone installation guide documentation (
>> http://community.gemstone.com/download/attachments/6816862/GS64-InstallGuide-Linux-3.1.pdf),
>> but the calculation are different, so Im confused about it. What are the
>> things I have to consider to properly configure shred parge cache?
>> 
>> -*Cleanup scripts*: I read here to automate a cleanup scrip, but also I
>> find differents approach with others scripts:
>> a) The backup make with #fullBackupCompressedTo: make a gzip file that
>> when decompress have no extension (is a extent.dbf copy no?). The other
>> approach tell my to make a direct copy of the extent (cp command).
>> b) The tranlog clean up script is cheking the files datetime but in other
>> post (http://forum.world.st/Translog-space-problem-td2403379.html) I read
>> about the #oldestLogFileIdForRecovery that tell me the oldest tranlog
>> needed for a recovery, is more convenient to make a script with that info?
>> 
>> Thnks in advance!
>> Leo
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Glass mailing list
>> Glass at .gemtalksystems
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> 
>> 
> 
> _______________________________________________
> Glass mailing list
> Glass at .gemtalksystems
> http://lists.gemtalksystems.com/mailman/listinfo/glass
> 
> 
> 
> 
> --
> View this message in context: http://forum.world.st/Glass-GLASS-performance-cleanup-scripts-tp4726189p4727407.html
> Sent from the GLASS mailing list archive at Nabble.com.
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass