[Glass] Sixx problematic
Dario Trussardi via Glass
glass at lists.gemtalksystems.com
Tue Mar 10 14:38:48 PDT 2015
Ciao Mariano,
> Hi Dario,
>
> I was planning to write a whole detailed post about how I finally manage to serialize and materialize using SIXX, with UTF8 and without running out of memory.
>
> Let me paste the code here.... it contains a LOT of comments because I wanted to explain the reasons. Paste it in GemStone so that you can read it better.
>
>
> serialize: anObject toSixxFile: aFilename
> | stream codec persistentKey persistentDict file |
>
> " First, let's be sure to use UTF8 encoding"
> codec := GRCodec forEncoding: 'utf8'.
> " Note that we do the whole serializion to a in memory stream rather than directly to file."
> stream := codec encoderFor: String new writeStream.
>
> "The persistentKey can be whatever...just something unique for the time the serialization is running"
> persistentKey := ('SIXX_ROOT_ARRAY', UUID new asString) asSymbol.
> " We need a persistent place to hold sixx temporal persisting roots. While you can put these roots directly in UserGlobals
> I don't do that because if I fire multiple serializations in paralel, I will get Write-Write conflict in UserGlobals (multiple gems doing at:ifAbsentPut:)
> Therefore, I define only one entry in UserGlobals and I store there a Reduced Conflict instance: a RcKeyValueDictionary. That way, even with multiple gems
> doing exports / imports, I won't get a Write-Write conflict (at least for the UserGlobals at:ifAbsentPut: case). "
> persistentDict := UserGlobals at: #LargeSIXXExportsAndImportRoots ifAbsentPut: [ RcKeyValueDictionary new ].
>
> [ "With #commitOnAlmostOutOfMemoryDuring: we force commits (and hence new persistent objects could go to disk now freeing memory) if we are about to run out of memory"
> MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
> persistentDict at: persistentKey put: Array new.
> "This is the SIXX hook that allow us to specifiy a persistent root. "
> anObject sixxOn: stream persistentRoot: (persistentDict at: persistentKey)
> ].
> "Just be jure to remove entry for the temp root because we do not need it anymore"
> ] ensure: [ persistentDict removeKey: persistentKey ifAbsent: [] ].
>
> "Once the whole serialization was made into a memory stream it's time to write it to a file. Why we didn't write directly to a file from the beginning?
> The reason is that GsFile, GsSocket and a few other cases have state that is associated to the session or gem being run. In addition, SIXX stores the stream
> in the persistent root array. Therefore, the GsFile could technically go to disk if the Gem needs more temp memory and then brough back. When this happens
> the GsFile instance will be quite broken. "
> stream flush.
> file := (GsFile open: aFilename mode: 'w' onClient: false).
> " Instead of using a UTF8 stream for the serialization, another possibility was to user a straight String stream at the beginning and
> now at the end, convert the whole contents into a UTF encoded string. While it may be faster, I thought this approach would require two large arrays in memory
> and since we are trying not to run out of memory, I thoguht the current solution is better. But I might be wrong. "
> file nextPutAll: stream contents.
> file flush.
>
>
>
>
>
>
>
> materializeFromSixxFile: aFilename
> " Please read the comments of #serialize:toSixxFile: since most of them applies to the materialization as well"
> | codec stream object context persistentKey persistentDict |
> persistentKey := ('SIXX_ROOT_ARRAY', Object new identityHash asString) asSymbol.
> codec := GRCodec forEncoding: 'utf8'.
> " Here, again, we read the whole file and get an in-memory readStream to avoid the problem of the GsFile being persisted"
> stream := codec decoderFor: (FileStream oldFileNamed: aFilename) binary contents readStream.
> persistentDict := UserGlobals at: #LargeSIXXExportsAndImportRoots ifAbsentPut: [ RcKeyValueDictionary new ].
> [
> MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
> persistentDict at: persistentKey put: Array new.
> context := SixxContext forRead.
> object := Object readSixxFrom: stream
> context: context
> persistentRoot: (persistentDict at: persistentKey)
> ].
> ] ensure: [ persistentDict removeKey: persistentKey ].
> ^ object
>
>
>
>
> Did it work?
I have do it, it work in my tODE stones.
I can save and load sixx file without error.
Thanks,
Dario
>
>
>
>
> On Tue, Mar 10, 2015 at 3:06 PM, Dario Trussardi via Glass <glass at lists.gemtalksystems.com> wrote:
> Ciao,
>
> I have a devkit tODE stone create with createTodeStone devkit 3.1.0.6
>
> into it i load:
>
> GsDeployer
> bulkMigrate: [
> {#( 'XMLSupport' '1.2.2' 'http://seaside.gemtalksystems.com/ss/MetacelloRepository').
> #('Seaside3' '3.0.13' 'http://smalltalkhub.com/mc/Seaside/MetacelloConfigurations/main').
> #('ZincHTTPComponents' '1.1' 'http://www.squeaksource.com/ZincHTTPComponents').
> #('Magritte3' '3.0' 'http://www.squeaksource.com/MetacelloRepository').
> #('Magritte3AddOns' '3.0.0' 'http://www.squeaksource.com/MetacelloRepository').
> #('Pier3' '3.0.0' 'http://www.squeaksource.com/MetacelloRepository').
> #('Pier3AddOns' '3.0.3' 'http://www.squeaksource.com/MetacelloRepository')}
> do: [ :ar |
> | projectName version repository |
> projectName := ar at: 1.
> version := ar at: 2.
> repository := ar at: 3.
> Metacello new
> configuration: projectName;
> repository: repository;
> get.
> Metacello new
> configuration: projectName;
> version: version;
> repository: repository;
> onUpgrade: [ :ex :existing :new |
> existing locked
> ifTrue: [ ex disallow ]
> ifFalse: [ ex allow ] ];
> onConflict: [ :ex | ex disallow ];
> load.
> Metacello new
> configuration: projectName;
> version: version;
> repository: repository;
> lock ] ]
>
> and:
>
> GsDeployer bulkMigrate: [
> Metacello new
> baseline: 'SIXX';
> repository: 'github://glassdb/SIXX:master/repository';
> load ].
>
> A this point the SixxExamples example2 works fine.
>
>
>
>
>
> Now i can save a sixx file relative to my data application anEnv link with:
>
> writeObject: anEnv toFile: aFilename inFolder: aDirectoryPath
>
> | fileStream sws |
>
> fileStream := ( FileDirectory on: aDirectoryPath) newFileNamed: aFilename.
>
> sws := SixxWriteStream on: fileStream.
>
> sws nextPut: anEnv.
> sws close.
>
>
>
>
>
> After when i do the command: ^Object readSixxFrom: fileStream contents.
>
> to read the sixx file create above the system answer the error:
>
> a SixxXmlParseError occurred (error 2710) - G/S[Scandella3106 devkit:1]
>
> AbstractException >> _signalWith: (envId 0)
> AbstractException >> signal (envId 0)
> SixxPortableUtil class >> signalException: (envId 0)
> [] in SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> AbstractException >> _executeHandler: (envId 0)
> AbstractException >> _signalWith: (envId 0)
> AbstractException >> signal: (envId 0)
> AbstractException class >> signal: (envId 0)
> XMLTokenizer >> parseError: (envId 0)
> XMLTokenizer >> errorExpected: (envId 0)
> XMLTokenizer >> nextLiteral (envId 0)
> XMLTokenizer >> nextPCDataDelimitedBy:putOn: (envId 0)
> [] in XMLTokenizer >> nextPCData (envId 0)
> XMLNestedStreamWriter >> writeWith: (envId 0)
> XMLTokenizer >> nextPCData (envId 0)
> XMLTokenizer >> nextToken (envId 0)
> XMLParser >> parseToken (envId 0)
> XMLParser >> parseDocument (envId 0)
> SAXHandler >> parseDocument (envId 0)
> XMLDOMParser >> parseDocument (envId 0)
> SAXHandler class >> parseDocumentFrom:useNamespaces:persistentRoot: (envId 0)
> XMLDOMParser class >> parseDocumentFrom:useNamespaces:persistentRoot: (envId 0)
> SAXHandler class >> parseDocumentFrom:persistentRoot: (envId 0)
> SixxYaxoXmlParserAdapter class >> parseXml:persistentRoot: (envId 0)
> [] in SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> ExecBlock >> on:do: (envId 0)
> SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> Behavior >> readSixxFrom:context:persistentRoot: (envId 0)
> Behavior >> readSixxFrom: (envId 0)
>
> Considerations ?
>
> Thanks,
>
> Dario
>
> P.S. I have another 3.1.0.6 stone where i have:
>
> ConfigurationOfGsSIXX project currentVersion >=0.3-c.1 [ConfigurationOfGsSIXX]
>
> ConfigurationOfXMLSupport project currentVersion >=1.2.2.1 [ConfigurationOfXMLSupport]
>
> Into it all works fine.
>
> The sixx file created into it can reading into the devkit tODE stone without problem
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150310/a12da977/attachment.html>
More information about the Glass
mailing list