[Glass] recommended way to use SIXX with a large file in 3.1?

[Norbert Hartl]

Am 24.09.2013 um 20:33 schrieb Paul DeBruicker <pdebruic at gmail.com>:

> Hi-
> 
> How should I load a 200MB sixx file into 3.1?  I could make it smaller
> by using the compact format., but haven't yet
> 
> 
> I'm working on getting SIXX running with the latest XMLParser because
> its loading XMLParser version 1.0.5 and I have code that uses 1.1.8.
> 
> Once that's done is it just a matter of wrapping the load in a
> #commitOnAlmostOutOfMemoryDuring: ?
> 
I don't think so. The trick with the commit is basically good for reading in the string because gemstone can commit pages of a multiple pages string and keep the reference. This way you chunk the data and commit it partially. How good this will work with the dom parser I don't know. I gave up on Sixx. The last time I tried I tweaked in a way that it works but it took ages to read in the file. Using a pull parser would remove some of the problems a dom parser introduces. But to really optimize it I think some has to change Sixx in order not to do a depth-first traversal of the graph. 
After that I used STON to serialize and deserialize and that worked for more than 200MB of data. Sven, Dale and me were working on STON to get rid of stack overflow problems. So I can say that you probably get much farther with STON than with Sixx. 
It would be interesting to see how Fuel could perform in that area. For a gemstone to gemstone exchange it is probably not to hard to port Fuel to gemstone. But then someone has to do it. And Dale seems to be very busy.

That's all of my cents,

Norbert