[Glass] Transaction Logs so large ....

[Paul Baumann via Glass]

Manual begin mode can still refer to an old view of data, it is not the
same as transactionless mode where the view is allowed to automatically
adjust to show newer changes. You still would benefit from periodic aborts
to avoid the signals to abort. Transactionless mode does not get SigAbort.

Paul Baumann

On Oct 24, 2016 9:13 AM, "Marten Feldtmann" <m at feldtmann.online> wrote:

> Concerning transactions: Well, all Topaz scripts (starting the Zinc
> server) are started in #manualBegin transactionMode. If not in that state
> the handling of SigAbort handlers is more difficult.
>
> Other than that: after an incoming request a BeginTransaction is done, the
> work is done and a commit or abort is then done ... then the gem is again
> out of an transaction - and then it should be no candidate for SigAbort
> handler (I'm not sure of that last statement).
>
> Conflicts - yes initially lots of. After that we introduce retries on the
> server side and now the conflicts are not very often - though the external
> programmers should always consider this ( ... and its not done).
>
> Other conflicts can not be solved in this way and need a different API
> behaviour (normally implemented with RcQueue and worker topaz processes on
> the server side). The offered API then returns a TASK id and the programmer
> has to watch for its task ... long-term logic tasks can be programmed in a
> conflict-free way then on the server side.
>
> For logic conflicts we have a revision counter in the objects.
>
> (We use an API-HTTP-REST-RPC oriented way of programming the Gemstone/S
> database when interacting via C#, Python or Javascript with the database).
>
> Paul Baumann <plbaumann at gmail.com> hat am 24. Oktober 2016 um 14:37
> geschrieben:
>
> > "more commits than what a gem can react to"
>
> Difficulties like that can happen with continueTransaction, not aborts nor
> beginTransaction. That said, continueTransaction is usually what a gem is
> coded to try first when it receives a SigAbort.
>
> Write code so that idle gems abort/begin every so often (like 15 seconds)
> without having to be signaled. Aborts are low cost. Do an abort when the
> request comes in (before processing it). Busy gems that are well done
> should never be in transaction long enough to get a SigAbort.
>
> SigAborts only (normally) go to the gems holding the oldest transaction
> and time is normally allowed for them to respond. Other gems may be holding
> a transaction view started a millisecond after those and they would not
> (normally) be signaled. A trick is to write code to proactively signal more
> gems than normally would be so that a 45 second allowed response time is
> able to gain more than a millisecond before the next oldest get signaled.
> Again though, good code doesn't need to be signaled.
>
> Do you get many commit conflicts? What is your strategy for handling them?
> Load balancing efforts can sometimes cause more work resolving conflicts. A
> common strategy is to have many gems feed work to an RCQueue that a single
> "manager" gem processes without conflicts. Of course then changes are not
> immediately visible to the requesting gem and more otherwise only-transient
> objects become persistent.
>
> Paul Baumann
>
> On Oct 24, 2016 7:39 AM, "Otto Behrens" <otto at finworks.biz> wrote:
>
> > If you already know this stuff then your question would be related to
> > checkpoint commits not happening in a timely manner. How that can happen
> and
> > how to improve that is a much deeper discussion, but is usually remedied
> by
> > changing gems to not stay with old views of the database. That can mean
> more
> > frequent aborts for example, and starting a transaction a short time
> before
> > making changes to commit. Just one gem staying in-transaction for a long
> > time can hold up checkpoint commits. A developer logged into (and
> > in-transaction) a very active database can hold up checkpoints causing a
> > backlog of transactions not yet applied to extent files.
>
> I understand that if there are gems holding onto old views with other
> gems committing a lot, it may happen that more garbage than usual will
> end up in the extent and the gems will take longer to commit because
> they have more work to determine the write set union etc. I suppose if
> things start to overflow one could create bigger and bigger
> transaction logs because a gem would be committing temporary objects
> to the repository.
>
> If one install proper SigAbort handlers for gems then the gems should
> be catching them and aborting, unless there are way more commits than
> what the gems can react to via sig aborts? Is it in this case that the
> condition may occur?
>
> In our application, we start a number of gems that service seaside
> requests. These gems (as well as the other gems we run in the
> background), all have sig abort handlers. The load balancing also
> helps to ensure that the seaside based application distributes work to
> idle gems. So in my view, gems should be cooperating well.
>
> In spite of this, we still see 10GB of tranlogs created in a few hours
> under low load. (The db is around 23GB).
>
> I suppose the only way to really analyse what's going on is to run
> statmonitor and understand the output. From my previous attempts
> though, it appears as if the Seaside infrastructure is really heavy on
> transaction logs.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20161024/d43671df/attachment.html>