[Glass] Backup procedure
Mariano Martinez Peck via Glass
glass at lists.gemtalksystems.com
Tue Jun 9 08:32:25 PDT 2015
On Tue, Jun 9, 2015 at 12:10 PM, Dario Trussardi via Glass <
glass at lists.gemtalksystems.com> wrote:
>
> Ciao Mariano,
>
> thanks for sharing your work.
>
> I have some questions: see below
>
Hi Dario,
My answers below.
>
> > Hi Dario,
> >
> > Yes, I have to do that and another couple of things (like disabling
> monit so that it won't start my gems while they are down for GC)
> >
> > Here is the full script. While it won't work out of the box for you, I
> think you can really get an idea.
> >
> > Dale, you may want to take something from it too (in fact, this script
> was a fork of yours, but I added quite a few things).
> >
> > I am not a bash expert, so it's far from perfect.
> >
> > Cheers,
> >
> >
> >
> > #!/bin/bash
> >
> > # Actually I only need this for the gemstone.secret check..because the
> rest gets from the site source
> > source /opt/gemstone/product/seaside/defSeaside
> >
> > if [ -s $GEMSTONE/seaside/etc/gemstone.secret ]; then
> > . $GEMSTONE/seaside/etc/gemstone.secret
> > else
> > echo 'Missing password file $GEMSTONE/seaside/etc/gemstone.secret'
> > exit 1
> > fi
> >
> > SiteSubfolder="$1"
> >
> > # Requires a Sites subfolder password as a parameter
> > if [ "a$1" = "a" ]; then
> > echo 'Missing argument <Sites subfolder>'
> > exit 1
> > fi
> >
> > source $MYAPP_SITES_DIRECTORY/$SiteSubfolder/gemstone/env
> >
> >
> > date=`date +%Y%m%d_%H%M%S`
> >
> backupfile=${GEMSTONE_DATADIR}/backups/${GEMSTONE_NAME}_backup_${date}.dbf.gz
> >
> > su -m $GEMSTONE_USER -c "touch
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log"
> > su -m $GEMSTONE_USER -c "touch
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log"
> >
> >
> > echo "`date` ------------------------ Starting backup of
> ${GEMSTONE_NAME} ------------------------" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> >
> >
> > if [ ! -d "${GEMSTONE_DATADIR}/backups" ]; then
> > su -m $GEMSTONE_USER -c "mkdir -p '${GEMSTONE_DATADIR}/backups'"
> > fi
> >
> >
> >
> > echo "`date` Adding backup to object log" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
> > begin
> > run
> > (ObjectLogEntry trace: 'BACKUP: begin ' object: '${backupfile}')
> addToLog.
> > %
> > commit
> > logout
> > quit
> > "
> >
> >
> >
> > echo "`date` Stopping monit" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > monit unmonitor -g ${GEMSTONE_NAME} >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
> >
> > echo "`date` Stopping seaside gems" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > sh $APPLICATION_DIR/scripts/stopSeasideGems.sh
> >
> > echo "`date` Performing #markForCollection and #reclaimAll" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
> > begin
> > run
> > [
> > [
>
> I don't know the next commands :
>
> > MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].
> > MCDefinition clearInstances.
> > MCMethodDefinition cachedDefinitions removeKeys:
> > (MCMethodDefinition cachedDefinitions keys).
> > MCMethodDefinition shutDown.
> > MethodVersionHistory uniqueInstance cleanUp.
>
> But you run markForCollection every time ?
>
> I think to run markForCollection only at the end of a cycle ( week
> - month ) when i remove link of hold data
>
>
Yes, this is up to you. YOU decide when to run it. In my case, I run it
every night as part of the cleaning and backup script. Currently, my
repositories are not that big so MFC does not take very long and I do not
have that many users at the time the cron runs (3am or so). So I prefer to
run it everyday and keep my system clean and small.
If you do not want to run MFC before the backup, then do not do it. You may
want to backup everyday yet run MFC once a week or month.
BTW...be careful with " I think to run markForCollection ". If you are
using the glass scripts "runSeasideGems30" that will be running a
maintenance vm which will be running MFC every hour. To disable that, what
I am doing is as part of my load code I do:
(Smalltalk at: #WAGemStoneMaintenanceTask) removeTaskNamed: 'Mark For
Collect'.
Note that you must do that as part of your post-load code or similar
because if you happen to load a new version of seaside packages at some
point, it might fire again the class side #initialize and hence rebuild the
tasks.
> > SystemRepository markForCollection.
> > SystemRepository reclaimAll.
> >
> > ] on: Halt, Warning do: [:ex | ex resume]
> > ] on: Error do: [:ex | ].
> > %
> > commit
> > logout
> > quit
> > "
> >
> >
> >
> > echo "`date` Start new tranlog and perform backup" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
> > run
> > | id |
> > id := SystemRepository startNewLog.
> > [ id < 0 ] whileTrue: [
> > System sleep: 1.
> > id := SystemRepository startNewLog ].
> > SystemRepository fullBackupCompressedTo: '${backupfile}'
> > %
> > logout
> > quit
> > "
> >
> > echo "`date` Start again Seaside Gems" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > sh $APPLICATION_DIR/scripts/startSeasideGems.sh
> > sleep 10
> >
> > echo "`date` Starting monit" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > monit monitor -g ${GEMSTONE_NAME} >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
> >
> >
> >
> >
> > echo "`date` Writing object log" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
> > begin
> > run
> > (ObjectLogEntry trace: 'BACKUP: completed ' object: '${backupfile}')
> addToLog.
> > %
> > commit
> >
> > logout
> > quit
> > "
> >
> >
> > if [ $? -eq 0 ]
> > then
> > echo "`date` Successful backup ... starting validation of backup" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > echo "------------------------------------------------------------" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $GEMSTONE/bin/copydbf ${backupfile} /dev/null >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
> > if [ $? -eq 0 ]
> > then
> > echo "`date` Successful validation " >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > echo
> "------------------------------------------------------------" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > echo
> "------------------------------------------------------------" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> >
> >
> > echo "`date` Delete old backups" >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 "
> >
> > run
> > | autoGeneratedBackups backupsToKeep |
> > autoGeneratedBackups := (FACompatibilityUtils current
> directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries
> > select: [ :anEntry | ((anEntry faBasename
> beginsWith: '${GEMSTONE_NAME}_backup_')
> > and: [ anEntry faBasename endsWith:
> '.dbf.gz' ])
> > and: [anEntry faBasename size =
> '${GEMSTONE_NAME}_backup_${date}.dbf.gz' size ]
> > ].
> > (FileStream oldFileNamed:
> '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log')
> > setToEnd;
> > nextPutAll: ('There are ', ((FACompatibilityUtils current
> directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries) size
> asString, ' entries in backup folder'); crlf;
> > nextPutAll: ('There are ', autoGeneratedBackups size
> asString, ' autogenerated backups'); crlf;
> > flush.
> > autoGeneratedBackups := autoGeneratedBackups asSortedCollection:
> [:entryA :entryB | entryA faBasename < entryB faBasename].
> > backupsToKeep := autoGeneratedBackups last: (7 min:
> autoGeneratedBackups size).
> > (autoGeneratedBackups reject: [ :each | backupsToKeep includes:
> each ]) do: [:anEntry |
> > (FileStream oldFileNamed:
> '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log')
> > setToEnd; nextPutAll: ('File to be deleted: ',
> anEntry faBasename); crlf; flush.
> > anEntry faParentDirectory deleteFileNamed: anEntry
> faBasename.
> > ].
> > System performOnServer: 'ln -sf ''', backupsToKeep first
> fullName , ''' ''', backupsToKeep last faParentDirectory pathName,
> '/OldestBackup'''.
> > %
> > commit
> > logout
> > quit
> > "
> >
>
> I don't understund what the system do in this steps.
>
It's just because my bash skills are a bit limited and I prefer to script
in smalltalk :)
Basically, what it does, is to take the directory of backups (
'${GEMSTONE_DATADIR}/backups/'), take all those files which are on the
pattern ${GEMSTONE_NAME}_backup_YYYYMMDD_HHMMSS.dbf.gz, then sorts them,
then keep only the newest 7 and delete the rest.
>
> >
> > echo "`date` Delete old tranlogs file and only keep the needed
> ones for the oldest backup `readlink
> > ${GEMSTONE_DATADIR}/backups/OldestBackup` " >>
> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log
> > sh $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/delete-old-tranlogs.sh -d
> ${GEMSTONE_DATADIR} -f
>
> Can you sharing the delete-old-tranlogs.sh code?
>
This is part of Norbert Hartl code but I do not remember if I modified it
too:
https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh
My version (maybe I did not changed it) is:
#!/bin/bash
while getopts ":d:g:f:r" opt; do
case $opt in
d)
DATA_DIR=$OPTARG
;;
f)
FILE=$OPTARG
;;
g)
export GEMSTONE=$OPTARG
;;
r)
REMOVE_TRANLOGS=1
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
done
function help {
echo "usage: $0 -d [directory] -g [gemstonedir] -r"
echo "
-d [directory] data directory of stone (containing extent0.dbf)
-f [file] file to take information about needed tranlog from
(extent or backup)
-g [gemstonedir] directory of gemstone installation (default:
/opt/gemstone/product)
-r really remove tranlogs. Without this switch they are
only shown
"
exit
}
if [ "$DATA_DIR" == "" ];
then
help;
fi
if [ ! -f "$FILE" ];
then
echo "did not find $FILE"
exit
fi
if [ "$GEMSTONE" == "" ];
then
COPYDBF_EXE=`which copydbf`
else
COPYDBF_EXE="$GEMSTONE/bin/copydbf"
fi
if [ ! -x "$COPYDBF_EXE" ];
then
echo "cannot find copydbf binary. please use -e argument or adjust your
path variable"
exit
fi
LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d ' '
-f 13`
if [ "$LAST_NEEDED_TRANLOG" == "" ];
then
echo "could not find information about last needed tranlog"
exit
fi
if [[ ! "$LAST_NEEDED_TRANLOG" =~ tranlog[0-9][0-9]*.dbf$ ]];
then
echo "parse error in output from copydby. Result is not a tranlog
filename"
exit
fi
SORTED_LIST_OF_FILES=`ls -1 $DATA_DIR/tranlog* | xargs -n1 basename | sort
-k2 -tg -n`
LINES_IN_LIST=`echo "$SORTED_LIST_OF_FILES" | wc -l`
if [ "$LINES_IN_LIST" -eq 1 ];
then
echo "only one tranlog file found. nothing to delete"
exit
fi
LINE_OF_LAST_NEEDED_TRANLOG=`echo "$SORTED_LIST_OF_FILES" | grep -n
$LAST_NEEDED_TRANLOG | cut -d ':' -f 1`
if [ "$LINE_OF_LAST_NEEDED_TRANLOG" == "" ];
then
echo "could not find last needed tranlog in list of available tranlogs."
exit
fi
if [ "$LINE_OF_LAST_NEEDED_TRANLOG" -lt 2 ];
then
echo "nothing to do"
exit
fi
LINE_OF_LAST_OBSOLETE_FILE=$(($LINE_OF_LAST_NEEDED_TRANLOG -1 ))
for obsolete in `echo "$SORTED_LIST_OF_FILES" | head -n
$LINE_OF_LAST_OBSOLETE_FILE`;
do
if [ ! -z "$REMOVE_TRANLOGS" ];
then
echo "removing $obsolete"
rm "$DATA_DIR/$obsolete"
else
echo "file $obsolete is obsolete"
fi
done
>
> > ${GEMSTONE_DATADIR}/backups/OldestBackup -g /opt/gemstone/product
> -r >>
> > $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1
> >
> > echo "`date` ---------------------- Backup finihed
> -----------------------"
> >
>
> Thanks for any considerations.
>
> Dario
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
--
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150609/87e6a116/attachment-0001.html>
More information about the Glass
mailing list