[Glass] Sixx problematic
Mariano Martinez Peck via Glass
glass at lists.gemtalksystems.com
Tue Mar 10 11:22:29 PDT 2015
Hi Dario,
I was planning to write a whole detailed post about how I finally manage to
serialize and materialize using SIXX, with UTF8 and without running out of
memory.
Let me paste the code here.... it contains a LOT of comments because I
wanted to explain the reasons. Paste it in GemStone so that you can read it
better.
serialize: anObject toSixxFile: aFilename
| stream codec persistentKey persistentDict file |
" First, let's be sure to use UTF8 encoding"
codec := GRCodec forEncoding: 'utf8'.
" Note that we do the whole serializion to a in memory stream rather than
directly to file."
stream := codec encoderFor: String new writeStream.
"The persistentKey can be whatever...just something unique for the time the
serialization is running"
persistentKey := ('SIXX_ROOT_ARRAY', UUID new asString) asSymbol.
" We need a persistent place to hold sixx temporal persisting roots. While
you can put these roots directly in UserGlobals
I don't do that because if I fire multiple serializations in paralel, I
will get Write-Write conflict in UserGlobals (multiple gems doing
at:ifAbsentPut:)
Therefore, I define only one entry in UserGlobals and I store there a
Reduced Conflict instance: a RcKeyValueDictionary. That way, even with
multiple gems
doing exports / imports, I won't get a Write-Write conflict (at least for
the UserGlobals at:ifAbsentPut: case). "
persistentDict := UserGlobals at: #LargeSIXXExportsAndImportRoots
ifAbsentPut: [ RcKeyValueDictionary new ].
[ "With #commitOnAlmostOutOfMemoryDuring: we force commits (and hence new
persistent objects could go to disk now freeing memory) if we are about to
run out of memory"
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
persistentDict at: persistentKey put: Array new.
"This is the SIXX hook that allow us to specifiy a persistent root. "
anObject sixxOn: stream persistentRoot: (persistentDict at:
persistentKey)
].
"Just be jure to remove entry for the temp root because we do not need it
anymore"
] ensure: [ persistentDict removeKey: persistentKey ifAbsent: [] ].
"Once the whole serialization was made into a memory stream it's time to
write it to a file. Why we didn't write directly to a file from the
beginning?
The reason is that GsFile, GsSocket and a few other cases have state that
is associated to the session or gem being run. In addition, SIXX stores the
stream
in the persistent root array. Therefore, the GsFile could technically go to
disk if the Gem needs more temp memory and then brough back. When this
happens
the GsFile instance will be quite broken. "
stream flush.
file := (GsFile open: aFilename mode: 'w' onClient: false).
" Instead of using a UTF8 stream for the serialization, another possibility
was to user a straight String stream at the beginning and
now at the end, convert the whole contents into a UTF encoded string. While
it may be faster, I thought this approach would require two large arrays in
memory
and since we are trying not to run out of memory, I thoguht the current
solution is better. But I might be wrong. "
file nextPutAll: stream contents.
file flush.
materializeFromSixxFile: aFilename
" Please read the comments of #serialize:toSixxFile: since most of them
applies to the materialization as well"
| codec stream object context persistentKey persistentDict |
persistentKey := ('SIXX_ROOT_ARRAY', Object new identityHash asString)
asSymbol.
codec := GRCodec forEncoding: 'utf8'.
" Here, again, we read the whole file and get an in-memory readStream to
avoid the problem of the GsFile being persisted"
stream := codec decoderFor: (FileStream oldFileNamed: aFilename) binary
contents readStream.
persistentDict := UserGlobals at: #LargeSIXXExportsAndImportRoots
ifAbsentPut: [ RcKeyValueDictionary new ].
[
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
persistentDict at: persistentKey put: Array new.
context := SixxContext forRead.
object := Object readSixxFrom: stream
context: context
persistentRoot: (persistentDict at: persistentKey)
].
] ensure: [ persistentDict removeKey: persistentKey ].
^ object
Did it work?
On Tue, Mar 10, 2015 at 3:06 PM, Dario Trussardi via Glass <
glass at lists.gemtalksystems.com> wrote:
> Ciao,
>
> I have a devkit tODE stone create with createTodeStone devkit
> 3.1.0.6
>
> into it i load:
>
> GsDeployer
> bulkMigrate: [
> {#( 'XMLSupport' '1.2.2' '
> http://seaside.gemtalksystems.com/ss/MetacelloRepository').
> #('Seaside3' '3.0.13' '
> http://smalltalkhub.com/mc/Seaside/MetacelloConfigurations/main').
> #('ZincHTTPComponents' '1.1' '
> http://www.squeaksource.com/ZincHTTPComponents').
> #('Magritte3' '3.0' 'http://www.squeaksource.com/MetacelloRepository'
> ).
> #('Magritte3AddOns' '3.0.0' '
> http://www.squeaksource.com/MetacelloRepository').
> #('Pier3' '3.0.0' 'http://www.squeaksource.com/MetacelloRepository').
> #('Pier3AddOns' '3.0.3' '
> http://www.squeaksource.com/MetacelloRepository')}
> do: [ :ar |
> | projectName version repository |
> projectName := ar at: 1.
> version := ar at: 2.
> repository := ar at: 3.
> Metacello new
> configuration: projectName;
> repository: repository;
> get.
> Metacello new
> configuration: projectName;
> version: version;
> repository: repository;
> onUpgrade: [ :ex :existing :new |
> existing locked
> ifTrue: [ ex disallow ]
> ifFalse: [ ex allow ] ];
> onConflict: [ :ex | ex disallow ];
> load.
> Metacello new
> configuration: projectName;
> version: version;
> repository: repository;
> lock ] ]
>
> and:
>
> GsDeployer bulkMigrate: [
> Metacello new
> baseline: 'SIXX';
> repository: 'github://glassdb/SIXX:master/repository';
> load ].
>
> A this point the SixxExamples example2 works fine.
>
>
>
>
>
> Now i can save a sixx file relative to my data application anEnv
> link with:
>
> writeObject: anEnv toFile: aFilename inFolder: aDirectoryPath
>
> | fileStream sws |
>
> fileStream := ( FileDirectory on: aDirectoryPath) newFileNamed:
> aFilename.
>
> sws := SixxWriteStream on: fileStream.
>
> sws nextPut: anEnv.
> sws close.
>
>
>
>
>
> After when i do the command: ^Object readSixxFrom: fileStream contents.
>
> to read the sixx file create above the system answer the error:
>
> a SixxXmlParseError occurred (error 2710) - G/S[Scandella3106 devkit:1]
>
> AbstractException >> _signalWith: (envId 0)
> AbstractException >> signal (envId 0)
> SixxPortableUtil class >> signalException: (envId 0)
> [] in SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> AbstractException >> _executeHandler: (envId 0)
> AbstractException >> _signalWith: (envId 0)
> AbstractException >> signal: (envId 0)
> AbstractException class >> signal: (envId 0)
> XMLTokenizer >> parseError: (envId 0)
> XMLTokenizer >> errorExpected: (envId 0)
> XMLTokenizer >> nextLiteral (envId 0)
> XMLTokenizer >> nextPCDataDelimitedBy:putOn: (envId 0)
> [] in XMLTokenizer >> nextPCData (envId 0)
> XMLNestedStreamWriter >> writeWith: (envId 0)
> XMLTokenizer >> nextPCData (envId 0)
> XMLTokenizer >> nextToken (envId 0)
> XMLParser >> parseToken (envId 0)
> XMLParser >> parseDocument (envId 0)
> SAXHandler >> parseDocument (envId 0)
> XMLDOMParser >> parseDocument (envId 0)
> SAXHandler class >> parseDocumentFrom:useNamespaces:persistentRoot: (envId
> 0)
> XMLDOMParser class >> parseDocumentFrom:useNamespaces:persistentRoot:
> (envId 0)
> SAXHandler class >> parseDocumentFrom:persistentRoot: (envId 0)
> SixxYaxoXmlParserAdapter class >> parseXml:persistentRoot: (envId 0)
> [] in SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> ExecBlock >> on:do: (envId 0)
> SixxXmlUtil class >> parseXml:persistentRoot: (envId 0)
> Behavior >> readSixxFrom:context:persistentRoot: (envId 0)
> Behavior >> readSixxFrom: (envId 0)
>
> Considerations ?
>
> Thanks,
>
> Dario
>
> P.S. I have another 3.1.0.6 stone where i have:
>
> ConfigurationOfGsSIXX project currentVersion >=0.3-c.1
> [ConfigurationOfGsSIXX]
>
> ConfigurationOfXMLSupport project currentVersion >=1.2.2.1
> [ConfigurationOfXMLSupport]
>
> Into it all works fine.
>
> The sixx file created into it can reading into the devkit
> tODE stone without problem
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
--
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150310/a9c7f1bc/attachment-0001.html>
More information about the Glass
mailing list