[Glass] Backups, GC and removing translogs

[Dale Henrichs]

On Tue, Aug 12, 2014 at 11:40 AM, Mariano Martinez Peck <
marianopeck at gmail.com> wrote:

>
>
>
> On Mon, Aug 11, 2014 at 6:15 PM, Dale Henrichs <
> dale.henrichs at gemtalksystems.com> wrote:
>
>> Mariano,
>>
>> I think this sounds fine overall ... just a few minor comments in-lined
>>
>>
> Hi Dale, thanks, I answer below.
>
>
>
>>
>> On Mon, Aug 11, 2014 at 1:33 PM, Mariano Martinez Peck <
>> marianopeck at gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I want to know what do you think about the following scenario. I want to
>>> start making some cleaning for my gemstones. After reading several emails
>>> etc I came to the following steps that I would like to run DAILY:
>>>
>>> 1) Stop all seaside gems (including maintainance etc)
>>>
>>> 2) Perform MFC and reclaim:
>>>
>>> [
>>> System beginTransaction.
>>>  SystemRepository markForCollection.
>>> SystemRepository reclaimAll.
>>> System commitTransaction.
>>>
>>> ] on: Halt,  Warning do: [:ex | ex resume]
>>>
>>> This step is not mandatory for me since I do run GLASS maintainance VM
>>> which does a MFC every hour...but...it might be a good idea anyway.
>>>
>>
>> If you are going to run an mfc once a day, then I would suggest that you
>> can probably get away with eliminating the hourly mfc and run on epoch gc
>> instead. The mfc  introduces excessive disk i/o as  every page (with live
>> objects) is dragged through the SPC and at the end, you will need to pull
>> your "standard" working set of pages in off of disk..
>>
>
> Yes, I was thinking the same. Once I have this GC+backup+clean stuff
> working, I will disable hourly MFC from GLASS'  maintainance VM.
>
>
>
>>
>>> 3) Perform backup :
>>>
>>> SystemRepository startNewLog.
>>> SystemRepository fullBackupCompressedTo:
>>> '/opt/somewhere/backup-dateXXX.dbf.gz'.
>>>
>>
>> It's a good idea to validate your backup after the backup is completed.
>> Take a look at the backup script that I use up on SS3[1] for an example
>> using copydbf. This at least avoids issues with file level corruption. I
>> believe that some folks do a restore from backup of the newly created
>> backup file along with some validation which covers any logical corruption.
>> Whether or not you go this far really depends upon how critical the backups
>> are ...
>>
>
> This is a good idea. Thanks. I will try to add this soon.
>
>
>>
>> For super critical backups you will want to arrange for off-site storage
>> of backups and tranlogs.
>>
>>
> I, I was thinking about that too. Mostly...tranlogs. Because right now,
> with my scheme, if I have a disk failure, I loose the last 24 hours edits
> because translogs are only written in same disk and only backup at night.
> But I guess this is a second step for me.
>
>
>
>> In the backup script you will see that I force the creation of a new
>> tranlog right before the backup ...
>>
>
> Yes, but I am already doing that, right? (see lines above). I mean, I send
> #startNewLog before #fullBackupCompressedTo:
>

Okay, I missed that:)


>
>
>
>> Frankly I forget why I put that trick in there:), but the idea is that
>> you will avoid having monster tranlogs lying around ...
>>
>> [1]
>> https://github.com/glassdb/webEditionHome/blob/master/bin/startGemStoneBackup.sh#L65-83
>>
>>>
>>> 4) Clean:
>>>
>>> Now...from what I understand, this script:
>>> https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh
>>>  does:
>>>
>>> LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d
>>> ' ' -f 13`
>>>
>>> I have just checked copydbf and indeed, seems to be answer the same as
>>> #oldestLogFileIdForRecovery. So it seems safe to use this tool, correct?
>>>
>>
>> I don't know offhand, but if Norbert put it in his script, then I trust
>> it:)
>>
>>
>>>  However...from what I understand..the critical part there is the $FILE
>>> which should be the last backup (in my example the latest in
>>> /opt/somewhere/). Is this correct?
>>>
>>
>>  If you are keeping several backups around you need to keep the tranlogs
>> that apply to the oldest backup that you are keeping... Using the latest
>> backup file will indeed give you the list of old and new tranlogs needed
>> for the latest, but there may be a tranlog or two that are still needed for
>> the oldest backup that you are keeping.
>>
>>
> Here is my main question now. Originally, I thought: I want to a daily
> backup of the last 7 days (or whatever number). And then, I would need all
> the tranlogs needed for the oldest day, let's say the 7th. So, originally I
> thought my backup script would move those 7 backup files and all needed
> tranlogs to my backup disk. And always keep that: 7 backups and needed
> tranlogs for last day.
> But now I wonder....I would be moving the tranlogs at the same time I move
> the backups (once a day, at night). Hence, I think it doesn't make sense.
> Why? Because why would I want an older backup + tranlogs, if I already have
> the next days backup as well. Is there any advantage of keeping the
> tranlogs that I am not seeing?  Maybe to take an older backup and do cherry
> picking of the transactions?
>

Very good question. So technically you really only need the latest backup
and tranlogs. If you lose the disks with the extents or find the extent
file are somehow corrupted you will just take your latest backup and
tranlogs restore and your back in the game.

What happens if it turns out the backup file is corrupted? If you don't
have any previous backup files you have no chance to restore your system to
a good state, depending upon the corruption involved in an extreme
emergency  the corruption in the extent might be manually reparable, but we
don't want to go there:)

A couple days worth of backups and tranlogs will give you additional
insurance against a single corrupted backup file.

We have had commercial customers discover that they have introduced logical
corruption in their repositories due to some sort of bug in the code and
having historical backups and tranlogs have helped them understand and
repair the corruption.

On the other hand if you have validated that you can "successfully" restore
from the backup file and and you keep copies of the backups and tranlogs in
a couple of different physical locations you probably don't need to keep a
whole bunch of backups and tranlogs around.

If you have available disk space, it doesn't hurt to have too many backups:)


> Of course, a different thing would be if I directly write the tranlogs in
> my backup disk as well. Because there I could have even the last 24 hours
> edits.
>
>
I think that it is prudent to backup tranlogs to separate physical drives.
If you lose a disk drive with 23 hours of tranlogs on it and no other
copies means that you have lost a days worth of transactions...

Dale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20140812/308e64b7/attachment.html>