[Glass] Backup procedure
Mariano Martinez Peck via Glass
glass at lists.gemtalksystems.com
Mon Jun 8 11:11:27 PDT 2015
On Mon, Jun 8, 2015 at 2:27 PM, Johan Brichau via Glass <
glass at lists.gemtalksystems.com> wrote:
> Mariano,
>
> Yes, the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 90 setting is related and it
> cleans the temp obj memory, which is holding onto the objects.
>
> In fact, I never recycle the seaside fastcgi gems but I sometimes have to
> recycle the servicevm gem once I notice that several Gb are not being
> reclaimed after an MFC. It always helps.
>
You mean you do not even ever stop/start seaside gems??? mmmmmm
> I have not figured out why the servicevm is behaving different in our case
> because the flag is set for all gems.
>
>
Did you check if one of the gems is fired linked while the others don't? if
so, then you may be getting the flags of the gem.exe rather than the one of
the seaside gems (I had this issue in the past)
> I never experienced a voting issue though, except if I accidentally
> launched another MFC concurrently. I don’t know but maybe a backup also
> blocks the voting process for an MFC? I think the manual mentions the
> conditions of the voting though.
>
>
mmmm in my case I THINK that the one gem that was opened and rejecting the
MFC because of voting was actually gemtools...not seaside gems. So...I
guess I am stopping the gems while MFC just to be super sure it will be
fine and the MFC takes little time so far. ...
> cheers
> Johan
>
> On 08 Jun 2015, at 19:12, Mariano Martinez Peck <marianopeck at gmail.com>
> wrote:
>
>
>
> On Mon, Jun 8, 2015 at 2:04 PM, Johan Brichau <johan at yesplan.be> wrote:
>
>> Hi Mariano,
>>
>> Why are you stopping the Seaside gems during an MFC?
>> If this is because you’ve noticed that some ‘garbage’ is retained by the
>> gems: a simple recycle (stop/start) does the job as well.
>>
>>
> Hi johan,
>
> Thanks for asking. Yes, that was one of my concerns. Good to know a
> stop/start would work too.
>
>
>> Once your repository grows larger, it will take a considerable amount of
>> time to complete a full backup. You don’t want your site to be offline that
>> long.
>>
>>
> Indeed, and I was already thinking about that. At some point, I remembered
> having a problem in which the GC would not run because they were waiting
> for the "vote" of some seaside gems (at least that's what I thought). I
> THINK this is related to GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 90; thingy.
> Right?
> So..if I make sure my seaside gems do have such flag, I am safe now? I am
> using 3.1.0.6 and latest GLASS.
>
> Thanks Johan for asking!
>
>
>
>
>
>> Johan
>>
>> On 08 Jun 2015, at 17:00, Mariano Martinez Peck via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>> Hi Dario,
>>
>> Yes, I have to do that and another couple of things (like disabling monit
>> so that it won't start my gems while they are down for GC)
>>
>> Here is the full script. While it won't work out of the box for you, I
>> think you can really get an idea.
>>
>> Dale, you may want to take something from it too (in fact, this script
>> was a fork of yours, but I added quite a few things).
>>
>> I am not a bash expert, so it's far from perfect.
>>
>> Cheers,
>>
>>
>>
>> #!/bin/bash
>>
>> # Actually I only need this for the gemstone.secret check..because the
>> rest gets from the site source
>> source /opt/gemstone/product/seaside/defSeaside
>>
>> if [ -s $GEMSTONE/seaside/etc/gemstone.secret ]; then
>> . $GEMSTONE/seaside/etc/gemstone.secret
>> else
>> echo 'Missing password file $GEMSTONE/seaside/etc/gemstone.secret'
>> exit 1
>> fi
>>
>> SiteSubfolder="$1"
>>
>> # Requires a Sites subfolder password as a parameter
>> if [ "a$1" = "a" ]; then
>> echo 'Missing argument <Sites subfolder>'
>> exit 1
>> fi
>>
>> source $MYAPP_SITES_DIRECTORY/$SiteSubfolder/gemstone/env
>>
>>
>> date=`date +%Y%m%d_%H%M%S`
>>
>> backupfile=${GEMSTONE_DATADIR}/backups/${GEMSTONE_NAME}_backup_${date}.dbf.gz
>>
>> su -m $GEMSTONE_USER -c "touch
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log"
>> su -m $GEMSTONE_USER -c "touch
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log"
>>
>>
>> echo "`date` ------------------------ Starting backup of ${GEMSTONE_NAME}
>> ------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>>
>>
>> if [ ! -d "${GEMSTONE_DATADIR}/backups" ]; then
>> su -m $GEMSTONE_USER -c "mkdir -p '${GEMSTONE_DATADIR}/backups'"
>> fi
>>
>>
>>
>> echo "`date` Adding backup to object log" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>> begin
>> run
>> (ObjectLogEntry trace: 'BACKUP: begin ' object: '${backupfile}') addToLog.
>> %
>> commit
>> logout
>> quit
>> "
>>
>>
>>
>> echo "`date` Stopping monit" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> monit unmonitor -g ${GEMSTONE_NAME} >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
>>
>> echo "`date` Stopping seaside gems" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> sh $APPLICATION_DIR/scripts/stopSeasideGems.sh
>>
>> echo "`date` Performing #markForCollection and #reclaimAll" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>> begin
>> run
>> [
>> [
>> MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].
>> MCDefinition clearInstances.
>> MCMethodDefinition cachedDefinitions removeKeys:
>> (MCMethodDefinition cachedDefinitions keys).
>> MCMethodDefinition shutDown.
>> MethodVersionHistory uniqueInstance cleanUp.
>> SystemRepository markForCollection.
>> SystemRepository reclaimAll.
>>
>> ] on: Halt, Warning do: [:ex | ex resume]
>> ] on: Error do: [:ex | ].
>> %
>> commit
>> logout
>> quit
>> "
>>
>>
>>
>> echo "`date` Start new tranlog and perform backup" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>> run
>> | id |
>> id := SystemRepository startNewLog.
>> [ id < 0 ] whileTrue: [
>> System sleep: 1.
>> id := SystemRepository startNewLog ].
>> SystemRepository fullBackupCompressedTo: '${backupfile}'
>> %
>> logout
>> quit
>> "
>>
>> echo "`date` Start again Seaside Gems" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> sh $APPLICATION_DIR/scripts/startSeasideGems.sh
>> sleep 10
>>
>> echo "`date` Starting monit" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> monit monitor -g ${GEMSTONE_NAME} >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
>>
>>
>>
>>
>> echo "`date` Writing object log" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>> begin
>> run
>> (ObjectLogEntry trace: 'BACKUP: completed ' object: '${backupfile}')
>> addToLog.
>> %
>> commit
>>
>> logout
>> quit
>> "
>>
>>
>> if [ $? -eq 0 ]
>> then
>> echo "`date` Successful backup ... starting validation of backup" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> echo "------------------------------------------------------------" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $GEMSTONE/bin/copydbf ${backupfile} /dev/null >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
>> if [ $? -eq 0 ]
>> then
>> echo "`date` Successful validation " >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> echo
>> "------------------------------------------------------------" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> echo
>> "------------------------------------------------------------" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>>
>>
>> echo "`date` Delete old backups" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>>
>> run
>> | autoGeneratedBackups backupsToKeep |
>> autoGeneratedBackups := (FACompatibilityUtils current
>> directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries
>> select: [ :anEntry | ((anEntry faBasename beginsWith:
>> '${GEMSTONE_NAME}_backup_')
>> and: [ anEntry faBasename endsWith: '.dbf.gz' ])
>> and: [anEntry faBasename size =
>> '${GEMSTONE_NAME}_backup_${date}.dbf.gz' size ]
>> ].
>> (FileStream oldFileNamed:
>> '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log')
>> setToEnd;
>> nextPutAll: ('There are ', ((FACompatibilityUtils current
>> directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries) size
>> asString, ' entries in backup folder'); crlf;
>> nextPutAll: ('There are ', autoGeneratedBackups size asString, '
>> autogenerated backups'); crlf;
>> flush.
>> autoGeneratedBackups := autoGeneratedBackups asSortedCollection:
>> [:entryA :entryB | entryA faBasename < entryB faBasename].
>> backupsToKeep := autoGeneratedBackups last: (7 min:
>> autoGeneratedBackups size).
>> (autoGeneratedBackups reject: [ :each | backupsToKeep includes: each
>> ]) do: [:anEntry |
>> (FileStream oldFileNamed:
>> '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log')
>> setToEnd; nextPutAll: ('File to be deleted: ', anEntry faBasename);
>> crlf; flush.
>> anEntry faParentDirectory deleteFileNamed: anEntry faBasename.
>> ].
>> System performOnServer: 'ln -sf ''', backupsToKeep first fullName ,
>> ''' ''', backupsToKeep last faParentDirectory pathName, '/OldestBackup'''.
>> %
>> commit
>> logout
>> quit
>> "
>>
>>
>> echo "`date` Delete old tranlogs file and only keep the needed ones
>> for the oldest backup `readlink ${GEMSTONE_DATADIR}/backups/OldestBackup` "
>> >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> sh $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/delete-old-tranlogs.sh -d
>> ${GEMSTONE_DATADIR} -f ${GEMSTONE_DATADIR}/backups/OldestBackup -g
>> /opt/gemstone/product -r >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> 2>&1
>>
>> echo "`date` ---------------------- Backup finihed
>> -----------------------"
>>
>>
>> exit 0
>> else
>> reason="Failed validatino"
>> echo "`date` Failed validation" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> fi
>> else
>> reason="Failed backup"
>> echo "`date` Failed backup" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> fi
>>
>>
>>
>> $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
>>
>> run
>> 'For details about the failure, scan backward in log file to previous
>> topaz session or copydbf sessions'
>> %
>> logout
>> quit
>>
>> run
>> (ObjectLogEntry fatal: 'BACKUP: failed ' object: '${reason}. See
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log for details') addToLog.
>> %
>> commit
>> logout
>> quit
>>
>> "
>>
>>
>>
>> echo "------------------------------------------------------------" >>
>> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
>> exit 101 #failure
>>
>>
>>
>>
>> On Mon, Jun 8, 2015 at 11:46 AM, Dario Trussardi via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>>> Ciao,
>>>
>>> i need to setup procedure for backup my devKit glass environment.
>>>
>>> For now i use:
>>>
>>> https://github.com/glassdb/webEditionHome/blob/master/bin/startGemStoneBackup.sh
>>>
>>>
>>>
>>> This procedure do:
>>> runs a GemStone full backup (compressed) and validates the resulting
>>> backup file
>>> It work fine and startNewLog before do the backup.
>>>
>>> Now i need to complete the backup.
>>>
>>> I think:
>>>
>>> 1) save the new backup into another system into local network ( named
>>> backupSystem )
>>>
>>> 2) remove the old backup from the server and from the backupSystem
>>>
>>> 3) remove the old tranlog from the server
>>> 4) remove other .... logs from the server
>>>
>>> Someone has already addressed this issue?
>>>
>>> The goal is to have a system which automatically manages all the step
>>> and is ready for restoring in the break case.
>>>
>>> Thanks for considerations..
>>>
>>>
>>> Dario
>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
--
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150608/a1992df5/attachment-0001.html>
More information about the Glass
mailing list