[Glass] Another metacello loading puzzle: xml

[Dale Henrichs via Glass]

On 1/15/18 3:11 AM, Iwan Vosloo via Glass wrote:
> Hi Dale,
>
>>> Either way, this example (working, if done using all the 
>>> complications in our scripts) contradicts the documentation [6] if 
>>> my reasoning is correct.
>> I guess I'm confused right now ... the log new log does not look like 
>> "working" to me as there is an error at the very end and I have not 
>> seen a stack ...
> [...]
>> If the load process is actually working, then I'm not exactly sure 
>> why I spent an hour or so on my weekend answering this email message 
>> only to find out that you are apparently saying "never mind at the 
>> end" ...
>
> Sorry, no, that's not what I meant, I did not express it clearly. (And 
> I do appreciate your help!)
Okay ... I understand ... email is often imperfect especially for 
complicated problems ..
>
> What is working is: using the WonkaLoader, ie, in that code, running 
> ./bootstrap_image.sh -n (a)
>
> The way I understand WonkaLoader code is that *in principle* it does 
> do a version of my "almost-pseudocode" below (b), except (a) adds ways 
> to deal with conflicts and a bunch of other stuff. (And, in (b) I am 
> trying to use a filetree repo)
>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: aLocalGitClone;
>>>   cacheRepository: yyy;
>>>   get.
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: aLocalGitClone;
>>>   cacheRepository: yyy;
>>>   load: 'default'.
>>>
>>> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that 
>>> order)
>>>
>>> Then it does:
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: ourOwnFileTreeRepo;
>>>   cacheRepository: yyy;
>>>   get.
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: ourOwnFileTreeRepo;
>>>   cacheRepository: yyy;
>>>   load: 'CI'.
>
>
> My reasoning is that, if (a) works, and I understand that it really 
> does (b) with some complications added, I should be able to do (b).
>
> But just doing (b) straight up does not work (that's the log you saw) 
> and I have not exposed its breakage for you properly last week - I ran 
> out of time. I am busy delving deeper into it today and will report 
> what I manage to get going.
Probably the best way for me to help is for me to start with a stack and 
a log file ... from the stack I tell where in the code the error is 
coming from ... from the log file I should be able to tell which part of 
the process is being run ... and then that will connect with the scripts .

I don't expect to be able to be an expert on how the overall WonkaLoad 
goes ... but with a stack I will be able to pinpoint where things are 
going wrong.
>
>
>
>
> About the WideString issue and missing  writeWordLike: you say:
>> This is not a correct solution ... the byte patterns differ between  
>> > WideString and QuadByteString ... if it is working it is likely to be 
> > luck ... or the corruption is only showing up in a comment ... or 
> the > code compiles without error, but will fail when run ... really 
> the only > solution in your case is to "edit the Monticello .mcz file" 
> and remove > your patch ... it is much safer to have the load fail 
> than debug the > potential downstream problems...
>
> I understand, I did not try to build a proper solution - I just copied 
> a Pharo implementation over to see what happens. Point is: is it not 
> possible to provide a proper implementation for writeWordLike: (since 
> there is code in GS that calls it)?
Monticello is based on serializing a collection of MCDefinitions using 
DataStream which serializes objects to a stream. Like most serializers 
the objects are copied to a byteStream recording class names and fields 
in a platform-specific fashion. With some luck and a little bit of 
hacking that made it possible for GemStone to be able to deserialize 
Monticello packages.

The String implementation is common across all platforms, but when it 
comes to multi-byte characters each dialect does things a bit 
differently and in the case of Squeak/Pharo the WideString 
implementation is completely different from the GemStone multibyte 
implementations (DoubleTypeString, QuadByteString, Unicode16, and 
Unicode32) and I've never even attempted to create a proper converter.

By the time multi-byte characters became common in pacakges, we had 
moved to FileTree which has proper Utf8 support and a cross-dialect 
serialization model ...

It is probably possible to provide a converter between WideString and 
the GemStone character collection hierarchy, but I do not intend to 
spend a lot of time on Monticello ... it will be supported "forever" but 
I hope to make it an optional addon in GsDevKit.

Over time, the kernel of GsDevKit will rely less and less on the pharo 
compatibility layer as the chunks of functionality are moved into the 
kernel. Monticello itself requires quite a bit of pharo compatibility 
and I have no interest in rewriting Monticello to run in the kernel -- 
tonel and filetree are better disk-based package formats as they are SCM 
neutral and Utf8 aware and we have already ported tonel, filetree and 
Metacello to the kernel so Monticello will be supported for folks who 
still rely on http: .mcz repositories and Monticello will run continue 
to run on top of the pharo compatibility layer (which will of course be 
kept and expanded --- support for specific pharo versions should be 
possible then ...
>
> This also means that whatever I *think* is working using our existing 
> stuff might be breaking in weird ways, so I have to actually go back 
> and sort this out better.
You only have to worry about code that is loaded with writeWordLike: and 
if the issue is in the ancestry, then the corruption would affect the 
loaded code ... however, you may run into trouble saving packages that 
end up being non-String objects ...
>
>
>> Here's some sample code:
> (thanks)
>
>
>> ... are you using Swazoo for your web server? 
>
> I think I can ditch Swazoo, it apparently used to be used in some 
> scenarios, but if I hit those I'll switch them to zinc.
>
>
>>> being able to fetch the correct dependencies at their correct 
>>> versions, from wherever, recursively is exactly the point of using 
>>> metacello in the first place. Having to manage that ourselves 
>>> defeats the point of using Metacello in the first place..
>>> (Besides, many of our dependencies are just Monticello packages 
>>> anyways)
>> Things are not quite that simple ... and I would say that if you 
>> downloading individual package versions, then I would claim that a 
>> git pull is much simpler than hand editting ConfigurationOf files to 
>> change the version of the package you are using ...
>>
>> If you are not specifically managing the versions of packages and 
>> projects (i.e., not hand editing ConfigurationOf files) then you are 
>> at the mercy of the project developers as to which version of the 
>> project you pick up when you update and have no control (other than 
>> hand editting ConfigurationOf files) over which version you are 
>> actually using in your product ...
>>
>> I would argue that with git clones you know exactly which version of 
>> each of the projects you are using ... and you are in complete 
>> control over when you want to use a new version ...
>
> I understand your motivation for git clones. I suspect there's stuff I 
> do not understand about "being at the mercy of project developers".  I 
> am also just used to similar tools in other ecosystems that do not 
> rely on me having to clone git repos (apt, npm, maven, pip) - these do 
> the recursive fetch (although some also have issues regarding 
> conflicting dependencies). My "being used to" these tools, and from 
> Metacello docs I thought this is what Metacello does too.
Automatically cloning git repos is something that is coming ... it will 
be a replacing for the old github: to tar file download ... my concern 
is that if local git clones are not used, then contributing to a project 
that is managed by git on github becomes much more difficult ... if the 
git clones are loaded into the image from the start, then it is natural 
to be able to push any changes/bugfixes back to the originating project ...

Isn't it true that for apt, npm, maven, pit, etc., that you are talking 
about deployment when you say that you don't need git clones? If you 
want to contribute to any of those projects you would need to use a git 
clone, right?

With Smalltalk the development environment is always part of the 
deployment there isn't really a distinction between deployment and 
development ... we often develop in our deployed app ..
>
> What do you mean by hand-editing ConfigurationOf files? Do you mean 
> editing our own, which includes project versions? If we are specific 
> in that, won't we always pick up the same version files? Isn't this 
> just like changing the debian/rules of your own deb package source?
If you are specific, you always get the same version files. If you want 
to move to a different version then you have to edit a ConfigurationOf 
file to change the version of each project ... and I actually think that 
pulling a different branch/tag/sha is actually easier than the process 
required to change the ConfigurationOf file ...
>
>>
>> With all of the said, you are working with a legacy production 
>> loading scheme and if you are going to pay the price of converting 
>> the load system, I would recommend that you wait until GemStone 3.5 
>> (planned, not committed) comes out with Rowan support ... at which 
>> time package loading, Metacello and git support will be incorporated 
>> in the GemStone kernel and be supported by GemTalk Systems instead of 
>> on community support, while still maintaining open source status for 
>> those tools...
>
> Thanks, I hear you. I am trying to understand the tools and my options 
> at the moment, so this certainly is one option to keep in mind.
>
>
> Regarding the rest of your comments: I am working on this today again. 
> I am hoping to delve deeper into it and will check in better logs and 
> report back...
Don't forget to include stacks ... the stack is just as important as the 
error message ... at least for me;)

Dale