[Glass] Backups, GC and removing translogs

Tue Aug 12 11:40:42 PDT 2014

On Mon, Aug 11, 2014 at 6:15 PM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:

> Mariano,
>
> I think this sounds fine overall ... just a few minor comments in-lined
>
>
Hi Dale, thanks, I answer below.

>
> On Mon, Aug 11, 2014 at 1:33 PM, Mariano Martinez Peck <
> marianopeck at gmail.com> wrote:
>
>> Hi guys,
>>
>> I want to know what do you think about the following scenario. I want to
>> start making some cleaning for my gemstones. After reading several emails
>> etc I came to the following steps that I would like to run DAILY:
>>
>> 1) Stop all seaside gems (including maintainance etc)
>>
>> 2) Perform MFC and reclaim:
>>
>> [
>> System beginTransaction.
>>  SystemRepository markForCollection.
>> SystemRepository reclaimAll.
>> System commitTransaction.
>>
>> ] on: Halt,  Warning do: [:ex | ex resume]
>>
>> This step is not mandatory for me since I do run GLASS maintainance VM
>> which does a MFC every hour...but...it might be a good idea anyway.
>>
>
> If you are going to run an mfc once a day, then I would suggest that you
> can probably get away with eliminating the hourly mfc and run on epoch gc
> instead. The mfc  introduces excessive disk i/o as  every page (with live
> objects) is dragged through the SPC and at the end, you will need to pull
> your "standard" working set of pages in off of disk..
>

Yes, I was thinking the same. Once I have this GC+backup+clean stuff
working, I will disable hourly MFC from GLASS'  maintainance VM.

>
>> 3) Perform backup :
>>
>> SystemRepository startNewLog.
>> SystemRepository fullBackupCompressedTo:
>> '/opt/somewhere/backup-dateXXX.dbf.gz'.
>>
>
> It's a good idea to validate your backup after the backup is completed.
> Take a look at the backup script that I use up on SS3[1] for an example
> using copydbf. This at least avoids issues with file level corruption. I
> believe that some folks do a restore from backup of the newly created
> backup file along with some validation which covers any logical corruption.
> Whether or not you go this far really depends upon how critical the backups
> are ...
>

This is a good idea. Thanks. I will try to add this soon.

>
> For super critical backups you will want to arrange for off-site storage
> of backups and tranlogs.
>
>
I, I was thinking about that too. Mostly...tranlogs. Because right now,
with my scheme, if I have a disk failure, I loose the last 24 hours edits
because translogs are only written in same disk and only backup at night.
But I guess this is a second step for me.

> In the backup script you will see that I force the creation of a new
> tranlog right before the backup ...
>

Yes, but I am already doing that, right? (see lines above). I mean, I send
#startNewLog before #fullBackupCompressedTo:

> Frankly I forget why I put that trick in there:), but the idea is that you
> will avoid having monster tranlogs lying around ...
>
> [1]
> https://github.com/glassdb/webEditionHome/blob/master/bin/startGemStoneBackup.sh#L65-83
>
>>
>> 4) Clean:
>>
>> Now...from what I understand, this script:
>> https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh
>>  does:
>>
>> LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d '
>> ' -f 13`
>>
>> I have just checked copydbf and indeed, seems to be answer the same as
>> #oldestLogFileIdForRecovery. So it seems safe to use this tool, correct?
>>
>
> I don't know offhand, but if Norbert put it in his script, then I trust
> it:)
>
>
>>  However...from what I understand..the critical part there is the $FILE
>> which should be the last backup (in my example the latest in
>> /opt/somewhere/). Is this correct?
>>
>
> If you are keeping several backups around you need to keep the tranlogs
> that apply to the oldest backup that you are keeping... Using the latest
> backup file will indeed give you the list of old and new tranlogs needed
> for the latest, but there may be a tranlog or two that are still needed for
> the oldest backup that you are keeping.
>
>
Here is my main question now. Originally, I thought: I want to a daily
backup of the last 7 days (or whatever number). And then, I would need all
the tranlogs needed for the oldest day, let's say the 7th. So, originally I
thought my backup script would move those 7 backup files and all needed
tranlogs to my backup disk. And always keep that: 7 backups and needed
tranlogs for last day.
But now I wonder....I would be moving the tranlogs at the same time I move
the backups (once a day, at night). Hence, I think it doesn't make sense.
Why? Because why would I want an older backup + tranlogs, if I already have
the next days backup as well. Is there any advantage of keeping the
tranlogs that I am not seeing?  Maybe to take an older backup and do cherry
picking of the transactions?

Of course, a different thing would be if I directly write the tranlogs in
my backup disk as well. Because there I could have even the last 24 hours
edits.

Thanks in advance,

-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140812/eeba63d2/attachment-0001.html>