[Glass] Transaction Logs so large ....

[Paul Baumann via Glass]

I recall using several approaches, I'm not sure which I stayed with. It was
nine years ago, and I've also addressed this for GBS clients. A server
subclass was one, because there was no better hook. I recall there being a
facility that could be registered for transaction changes. I recall noting
an expiration time some seconds from current. I recall Norm had added (or
perhaps optimized) a way to determine view age at my request. I recall the
subclass being to add an instvar and because #continueTransaction needed
special handling (like it did not notify as the others). Those were ways to
know view age in a general way, and hopefully obsolete way for more recent
releases. I recall a "busy while" global being updated and checked to defer
aborts by a background process (which might have been GBS specific code).
Sorry, I'd have to see the code to help with specifics. What you've said
sounds familiar though.

Active management of view age relieved serious performance risks (to
company operations) that had previously existed with reliance on
reactionary SigAbort. Avoid long-held database views. We went from serious
backlog alarms triggered several times a day to eventually none at all and
quick recovery for any commit record backlog.

Paul Baumann

On Oct 24, 2016 10:43 AM, "Marten Feldtmann via Glass" <
glass at lists.gemtalksystems.com> wrote:

> That's very interesting.
>
> This means, that I should introduce my own subclass of ZnServer and its
> #serveConnectionOn: (or the #noteAcceptWaitTimedOut) should be rewritten to
> make an empty transaction (with #beginTransaction and #abortTransaction) on
> a periodical base (15 seconds ?).
>
> Marten
>
> Paul Baumann <plbaumann at gmail.com> hat am 24. Oktober 2016 um 15:41
> geschrieben:
>
> Manual begin mode can still refer to an old view of data, it is not the
> same as transactionless mode where the view is allowed to automatically
> adjust to show newer changes. You still would benefit from periodic aborts
> to avoid the signals to abort. Transactionless mode does not get SigAbort.
>
> Paul Baumann
>
> On Oct 24, 2016 9:13 AM, "Marten Feldtmann" <m at feldtmann.online> wrote:
>
> Concerning transactions: Well, all Topaz scripts (starting the Zinc
> server) are started in #manualBegin transactionMode. If not in that state
> the handling of SigAbort handlers is more difficult.
>
> Other than that: after an incoming request a BeginTransaction is done, the
> work is done and a commit or abort is then done ... then the gem is again
> out of an transaction - and then it should be no candidate for SigAbort
> handler (I'm not sure of that last statement).
>
> Conflicts - yes initially lots of. After that we introduce retries on the
> server side and now the conflicts are not very often - though the external
> programmers should always consider this ( ... and its not done).
>
> Other conflicts can not be solved in this way and need a different API
> behaviour (normally implemented with RcQueue and worker topaz processes on
> the server side). The offered API then returns a TASK id and the programmer
> has to watch for its task ... long-term logic tasks can be programmed in a
> conflict-free way then on the server side.
>
> For logic conflicts we have a revision counter in the objects.
>
> (We use an API-HTTP-REST-RPC oriented way of programming the Gemstone/S
> database when interacting via C#, Python or Javascript with the database).
>
> Paul Baumann <plbaumann at gmail.com> hat am 24. Oktober 2016 um 14:37
> geschrieben:
>
> > "more commits than what a gem can react to"
>
> Difficulties like that can happen with continueTransaction, not aborts nor
> beginTransaction. That said, continueTransaction is usually what a gem is
> coded to try first when it receives a SigAbort.
>
> Write code so that idle gems abort/begin every so often (like 15 seconds)
> without having to be signaled. Aborts are low cost. Do an abort when the
> request comes in (before processing it). Busy gems that are well done
> should never be in transaction long enough to get a SigAbort.
>
> SigAborts only (normally) go to the gems holding the oldest transaction
> and time is normally allowed for them to respond. Other gems may be holding
> a transaction view started a millisecond after those and they would not
> (normally) be signaled. A trick is to write code to proactively signal more
> gems than normally would be so that a 45 second allowed response time is
> able to gain more than a millisecond before the next oldest get signaled.
> Again though, good code doesn't need to be signaled.
>
> Do you get many commit conflicts? What is your strategy for handling them?
> Load balancing efforts can sometimes cause more work resolving conflicts. A
> common strategy is to have many gems feed work to an RCQueue that a single
> "manager" gem processes without conflicts. Of course then changes are not
> immediately visible to the requesting gem and more otherwise only-transient
> objects become persistent.
>
> Paul Baumann
>
> On Oct 24, 2016 7:39 AM, "Otto Behrens" <otto at finworks.biz> wrote:
>
> > If you already know this stuff then your question would be related to
> > checkpoint commits not happening in a timely manner. How that can happen
> and
> > how to improve that is a much deeper discussion, but is usually remedied
> by
> > changing gems to not stay with old views of the database. That can mean
> more
> > frequent aborts for example, and starting a transaction a short time
> before
> > making changes to commit. Just one gem staying in-transaction for a long
> > time can hold up checkpoint commits. A developer logged into (and
> > in-transaction) a very active database can hold up checkpoints causing a
> > backlog of transactions not yet applied to extent files.
>
> I understand that if there are gems holding onto old views with other
> gems committing a lot, it may happen that more garbage than usual will
> end up in the extent and the gems will take longer to commit because
> they have more work to determine the write set union etc. I suppose if
> things start to overflow one could create bigger and bigger
> transaction logs because a gem would be committing temporary objects
> to the repository.
>
> If one install proper SigAbort handlers for gems then the gems should
> be catching them and aborting, unless there are way more commits than
> what the gems can react to via sig aborts? Is it in this case that the
> condition may occur?
>
> In our application, we start a number of gems that service seaside
> requests. These gems (as well as the other gems we run in the
> background), all have sig abort handlers. The load balancing also
> helps to ensure that the seaside based application distributes work to
> idle gems. So in my view, gems should be cooperating well.
>
> In spite of this, we still see 10GB of tranlogs created in a few hours
> under low load. (The db is around 23GB).
>
> I suppose the only way to really analyse what's going on is to run
> statmonitor and understand the output. From my previous attempts
> though, it appears as if the Seaside infrastructure is really heavy on
> transaction logs.
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20161024/fe2395e9/attachment-0001.html>