[Glass] Purge tranlogs?
Dale K. Henrichs
dale.henrichs at gemtalksystems.com
Sat Dec 21 19:58:38 PST 2013
----- Original Message -----
| From: "Norbert Hartl" <norbert at hartl.name>
| To: "Dale K. Henrichs" <dale.henrichs at gemtalksystems.com>
| Cc: "GemStone Seaside beta discussion"
| <glass at lists.gemtalksystems.com>, "Mariano Martinez Peck"
| <marianopeck at gmail.com>
| Sent: Saturday, December 21, 2013 2:51:12 AM
| Subject: Re: [Glass] Purge tranlogs?
| Am 21.12.2013 um 02:36 schrieb Dale K. Henrichs <
| dale.henrichs at gemtalksystems.com >:
| | ----- Original Message -----
|
| | | From: "Norbert Hartl" < norbert at hartl.name >
| |
|
| | | To: "Dale K. Henrichs" < dale.henrichs at gemtalksystems.com >
| |
|
| | | Cc: "GemStone Seaside beta discussion" <
| | | glass at lists.gemtalksystems.com >, "Mariano Martinez Peck" <
| | | marianopeck at gmail.com >
| |
|
| | | Sent: Friday, December 20, 2013 3:07:35 PM
| |
|
| | | Subject: Re: [Glass] Purge tranlogs?
| |
|
| | | May I assume you start a new tranlog every two hours before
| | | copying
| | | them to a new machine. I personally wouldn’t transfer a file that
| | | is
| | | still being written even if there is no random access regarding
| | | file
| | | position.
| |
|
| | No I don't start a new tranlog, so I'm willing to potentially lose
| | some data at the cost of simplicity ... I use rsync to do the copy
| | and it repairs the tranlogs at the next boundary …
|
| As it is append only the difference is if the tools can cope with
| that corruption at the end or not. Good to know that this works. It
| is a lot less painful then to do copy tasks
I'm pretty sure that recovering from tranlogs is pretty bullet proof ... our ACID guarantee is that what is written to disk is good. Commit is complete after tranlog record makes it to disk, so partial records should be tolerated if recovery needed ... copying every two hours means I'm willing to tolerate some lost records in the case where a machine and disks fry, so partial records are not critical (to me)...
I guess this could be tested by taking copies of a tranlog that successfully restored from backup and copy the tranlog shaving off a byte or two at the end ...
| | | Furthermore what probably isn’t clear by now. A fullbackup as the
| | | name says contains everything you need. Having the tranlogs just
| | | closes the gap between backup time and now, right?
| |
|
| | once you make a full backup you only need to keep around the
| | tranlogs
| | that were "open at the time of the backup" plus all tranlogs
| | created
| | after the backup. By "open at the time of the backup", I am
| | referring to the fact that to properly restore from a backup you
| | may
| | need tranlog records that predate the start of the backup and those
| | records may multiple tranlog files earlier. There is a system call
| | (that I don't recall right now) that will tell you the oldest
| | tranlog needed for recovery ... all of the tranlogs previous to
| | that
| | one are not needed. If you want to be complete, then you are
| | probably going to want to keep multiple backups and tranlogs going
| | back in time …
|
| I do
| SystemRepository startNewLog.
| SystemRepository fullBackupCompressedTo:
| '/opt/application/$1/backup/backup.dbf.gz'
| So the tranlog being open while backing up is the new one. So I
| should good to go only having the new tranlog?
Potentially more than one log file could be needed. The expression `SystemRepository oldestLogFileIdForRecovery ` gives you the correct answer.
| | and while we are on the subject it is worth validating that your
| | freshly taken backup is not corrupted and that is done by using
| | copydbf to copy the file ... copydbf will fail if the file has not
| | been correctly written …
|
| Good point. I’ll add that.Thanks.
| Norbert
| | | So I’m a little bit lazy. I have arranged schedules via cron. 30
| | | minutes to midnight I start the cleanup handler for my business
| | | model. This removes old and obsolete stuff (producing dead
| | | objects).
| | | 5 minutes later I start an MFC and let that settle for 15 minutes
| | | because it is just marking and after that the gc thread kicks in
| | | removing the objects for real (removing dead objects). Then I
| | | start
| | | a full backup of the stone and compress it. A little while later
| | | the
| | | backup process comes visiting, collects the backups, encrypts and
| | | transfers them to a backup machine of the provider. That’s all.
| | | Imperfect but fine for me :)
| |
|
| | Other than differences in a couple of details, this is what I've
| | done
| | for ss3 ...
|
| | Dale
|
| | | Norbert
| |
|
| | | Am 20.12.2013 um 23:40 schrieb Dale K. Henrichs <
| | | dale.henrichs at gemtalksystems.com >:
| |
|
| | | | The issue here is what is your backup strategy ...the
| | | | definition
| | | | of
| | | | an unneeded tranlog is defined by the last backup that you are
| | | | keeping ... how many backups are your keeping and where are you
| | | | keeping your backups ... it's a very good idea to do these
| | | | things...for ss3 i copy tranlogs to a separate host every two
| | | | hours
| | | | and delete tranlogs that are 2 days old ... so I have some
| | | | redundancy, but I am keeping my backups and tranlogs on a
| | | | different
| | | | machine .... there are relatively easy formulas, but there are
| | | | a
| | | | number of different solutions and there are tradeoffs as well
| | | | ...
| | |
| |
|
| | | | My preference is for folks who are using GLASS to describe the
| | | | different schemes that they are using and through the community
| | | | develop the sets of scripts that work for different groups of
| | | | communities ...
| | |
| |
|
| | | | I don't want to dictate how you set things up in production,
| | | | but
| | | | I
| | | | am
| | | | willing to dictate how you should set things to get started
| | | | with
| | | | GemStone:)
| | |
| |
|
| | | | Dale
| | |
| |
|
| | | | ----- Original Message -----
| | |
| |
|
| | | | | From: "Norbert Hartl" < norbert at hartl.name >
| | | |
| | |
| |
|
| | | | | To: "Mariano Martinez Peck" < marianopeck at gmail.com >
| | | |
| | |
| |
|
| | | | | Cc: "GemStone Seaside beta discussion" <
| | | | | glass at lists.gemtalksystems.com >
| | | |
| | |
| |
|
| | | | | Sent: Friday, December 20, 2013 1:19:03 PM
| | | |
| | |
| |
|
| | | | | Subject: Re: [Glass] Purge tranlogs?
| | | |
| | |
| |
|
| | | | | Am 18.12.2013 um 19:28 schrieb Mariano Martinez Peck <
| | | | | marianopeck at gmail.com >:
| | | |
| | |
| |
|
| | | | | | On Sat, Dec 14, 2013 at 11:49 AM, Norbert Hartl <
| | | | | | norbert at hartl.name
| | | | | | > wrote:
| | | | |
| | | |
| | |
| |
|
| | | | | | | Am 13.12.2013 um 21:53 schrieb Mariano Martinez Peck <
| | | | | | | marianopeck at gmail.com >:
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | | | Hi,
| | | | | | |
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | | | Is that possible? They are taking some space in my
| | | | | | | | laptop...
| | | | | | |
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | | | I periodically make some backups from GemTools.
| | | | | | |
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | | You might find that script useful
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | | https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh
| | | | | |
| | | | |
| | | |
| | |
| |
|
| | | | | | Thanks Norbert,
| | | | |
| | | |
| | |
| |
|
| | | | | | That was pretty easy to use and clear. I could release my
| | | | | | last
| | | | | | 15GB
| | | | | | of my HDD ;)
| | | | |
| | | |
| | |
| |
|
| | | | | | I wonder if such a script could be included in
| | | | | | /opt/gemstone/product/seaside/bin ?
| | | | |
| | | |
| | |
| |
|
| | | | | | I wonder the side effects of removing the obsolete
| | | | | | translogs.
| | | | | | I
| | | | | | guess
| | | | | | not that much but I am not 100% clear of what that means
| | | | | | exactly.
| | | | | | Need to read more about it.
| | | | |
| | | |
| | |
| |
|
| | | | | I talked to James in Edinburgh about this and the conclusion
| | | | | is
| | | | | that
| | | | | there is a chance this goes wrong but the odds are very low.
| | | | | If
| | | | | you
| | | | | want to be save I think you need to do an explicit snapshot
| | | | | before
| | | | | invoking the script. James can tell it how it is exactly.
| | | |
| | |
| |
|
| | | | | Norbert
| | | |
| | |
| |
|
| | | | | _______________________________________________
| | | |
| | |
| |
|
| | | | | Glass mailing list
| | | |
| | |
| |
|
| | | | | Glass at lists.gemtalksystems.com
| | | |
| | |
| |
|
| | | | | http://lists.gemtalksystems.com/mailman/listinfo/glass
| | | |
| | |
| |
|
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20131221/c85f4417/attachment.html>
More information about the Glass
mailing list