[Glass] Strange behaviour ... too many data and commit failed not due to memory, but conflicts

Wed Feb 22 15:49:51 PST 2017

GLASS mailing list wrote
> <!DOCTYPE html>
> 
>     
> Today I noticed a behaviour I found strange. In one of our project we had
> to import address data and my general test case was around100000 addresses
> and the import into Gemstone/S went without problems (using ONE
> transaction) - it was a single task (no other applications running agsint
> the database). It never failed. It simply worked. Now I talked with a
> customer and he told me about projects with 300.000 addresses and
> additional columns (etc). Much more data ... 

Marten,

Independent of the write conflict, there is the question of designing for
this problem space. You appear to have an open-ended problem, both in terms
of number of rows but also in terms of number of columns. You should build a
solution that supports this characteristic. 

If there is a practical limit to the number of columns, you can focus on how
many rows to process in a batch. Otherwise, you would want to design around
the number of cells. You want batches large enough that a significant amount
of work is done in each, but not so large that any kind of error wastes too
much time retrying the batch. Keeping the length of time down protects
against commit record backlogs as well as write conflicts. It cannot
eliminate the possibility, but it does reduce the chances as well as making
recovery more effective.

> Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER
> able to import that data set in ONE transaction. The commit ALWAYS failed.
> The commit failed but not due to memory problems, but due to conflict
> problems ! Again no other task was running against this database (all the
> other REST topaz scripts had nothing to do). Then I wanted to know, why
> the commit always failed and I managed to get this output:
> failureRead-Write Conflicts... Write-Write Conflicts...   
> 9749490177 Write-Dependency Conflicts... Write-ReadLock Conflicts...
> Write-WriteLock Conflicts... Rc-Write-Write Conflicts...
> Synchronized-Commit Conflicts... When I import the same data - but doing a
> commit after e.g. 50000 addresses, the import is again stable as usual.
> The test cases uses lots of RcCounter instances (and those values are
> changed with each address) - but other like that I see no unusual stuff. I
> know, I know - no real solid information, but this behaviour seemed
> strange to me ... But perhaps someone has a hint ???? Marten 
>  
> 
> _______________________________________________
> Glass mailing list

> Glass at .gemtalksystems

> http://lists.gemtalksystems.com/mailman/listinfo/glass

--
View this message in context: http://forum.world.st/Strange-behaviour-too-many-data-and-commit-failed-not-due-to-memory-but-conflicts-tp4935505p4935524.html
Sent from the GLASS mailing list archive at Nabble.com.