[Glass] Another metacello loading puzzle: xml

[Dale Henrichs via Glass]

On 1/9/18 4:02 AM, Iwan Vosloo via Glass wrote:
> Hi guys,
>
> On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:
> > Just a small comment from the outside: that problem seems to be
> > known: 
> http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
> > Maybe Iwan can check if he has instances of QuadByteString too...
> > that might be the reason...
>
> On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
>> Getting rid of the QuadByteStrings in the mcz file is one solution the 
> > other solution is to save the package in Filetree format, since
> > Filetree uses utf8.
>
>
>
> Thanks Mariano and Dale,
>
> Yes, I eventually discovered the QuadByteString issue... I tried using 
> filetree initially, but given the code that's there (that WonkaLoader 
> thing) it is not straightforward to use filetree (I suspect because it 
> uses FileTree to fetch FileTree as well, there may be a 
> chicken-and-egg scenario going on...but I did not investigate).
>
> Instead I figured the QuadByteString issue must be a bug, since 
> writeWordLike: is being called from code present in Gemstone, yet it 
> is not implemented in Gemstone.
>
> I provided an implementation (should I log a bug somewhere?), and 
> using that it actually loads using our scripts. Which is nice because 
> it gives me a starting point of sorts from where I can refactor and 
> simplify.
This is not a correct solution ... the byte patterns differ between 
WideString and QuadByteString ... if it is working it is likely to be 
luck ... or the corruption is only showing up in a comment ... or the 
code compiles without error, but will fail when run ... really the only 
solution in your case is to "edit the Monticello .mcz file" and remove 
your patch ... it is much safer to have the load fail than debug the 
potential downstream problems...

To edit a Monticello .mcz file with WideStrings, you should load the mcz 
file into a pharo (or squeak) image ... from there your best bet would 
seem to be to inspect the package structure and get the snapshot 
definitions and find look for MCMethodDefinitions whose #source is a 
kind of WideString also look for a class definition #comment that might 
contain a WideString ... if you find a comment or method with a 
WideString, then edit the method(s) and comment(s) to eliminate the 
WideString character(s) save a new version of the .mcz file and then 
edit the appropriate ConfigurationOf to point to the new .mcz version 
and try the load again ...

If you are still getting an error or you didn't find any methods or 
comments with WideStrings, then it is likely that a commit comment in 
the .mcz contains the offending character (and there may be more than 
one comment with the problem) or the problem might be another of the 
fields ... all it takes is one WideString to spoil the entire package.

Here's some sample code:

| basePackageName wc comments methods versions badDefs |
basePackageName := 'Metacello-Base'.
wc := MCWorkingCopy allManagers detect: [:each | each packageName = 
basePackageName].
comments := OrderedCollection new..
methods := OrderedCollection new.
wc snapshot definitions do: [:def |
     def isClassDefinition
         ifTrue: [ (def comment isKindOf: WideString) ifTrue: [ comments 
add: def]].
     def isMethodDefinition
         ifTrue: [ (def source isKindOf: WideString) ifTrue: [ comments 
add: def]]].
badDefs := OrderedCollection new.
comments isEmpty ifFalse: [ badDefs add: comments ].
methods isEmpty ifFalse: [ badDefs add: methods ].
badDefs isEmpty ifFalse: [ ^ badDefs ].
"now scan wc ancestry for WideStrings"
versions := OrderedCollection new.
wc ancestry allAncestorsDo: [:aMCVersionInfo |
     (aMCVersionInfo message isKindOf: WideString) ifTrue: [ versions 
add: aMCVersionInfo ] ].
versions
>
> Of course there's strong incentive to just leave it at that from the 
> people who have to pay the $$ for this... But I would have liked to be 
> able to load it all (via/not via network) using only what is provided 
> by GsUpgrader and Metacello without all the various things in our 
> WonkaLoader class....
Yeah the FinWorks WonkaLoader likely predates Metacello let alone 
GsUpgrader and GsDevKit_home...
>
> It is painful to have to get all the right branches of all the right 
> projects from all the right github repositories locally just to be 
> able to load...
Hmmm, it seems that there is some pain associated with the current 
scheme and once you do clone the projects and get them to the right 
version then you will be able to load without network access all of the 
time and when it is a good time for you to update the version of a 
project you are using, you can do it on your own schedule when you are 
prepared for the porting effort ...

A quick scan o f the http: projects that you are using several different 
http: repositories ... Other than Seaside which turns into a github 
download for GemStone anyway and Metacello (which doesn't need http: 
access if you use bootstrapGsDevKit[2]) , the other http: repositories 
I've identified are:

   http://seaside.gemtalksystems.com/ss/hyper
   http://smalltalkhub.com/mc/hernan/XMLPullParser/main
   http://smalltalkhub.com/mc/PharoExtras/XMLParser/main
http://smalltalkhub.com/mc/PharoExtras/OrderPreservingDictionary/main
   http://www.smalltalkhub.com/mc/PharoExtras/BitmapCharacterSet/main
   http://smalltalkhub.com/mc/Swazoo/Swazoo/main
   http://seaside.gemtalksystems.com/ss/swazoo2

hyper also uses a local clone for bootstrapGsDevKit[2], but the others 
have not been moved to Github yet ... are you using Swazoo for your web 
server? If not then you are down to only 4 different projects based on 
http: and I'm thinking that the XML projects need to move to GitHub soon 
and have already heard from Monty that he would be willing to do that 
... The other two OrderPreservingDictionary and  BitmapCharacterSet 
presumably aren't changing that frequently to become difficult to manage 
as a git clone ....

  As for the github: projects that aren't already covered by 
bootstrapGsDevKit[2] it seems that only SeasideSt/Seaside is being used ...

I don't know, but from where I am sitting, it would be easier to manage 
git repositories than try to figure out why your scripts aren't working:)

Oh well...

[1] 
https://raw.githubusercontent.com/finworks/BaselineOfWonkaRuntime/master/logs/transcript.2018-01-01.log
[2] 
https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit 

> being able to fetch the correct dependencies at their correct 
> versions, from wherever, recursively is exactly the point of using 
> metacello in the first place. Having to manage that ourselves defeats 
> the point of using Metacello in the first place..
> (Besides, many of our dependencies are just Monticello packages anyways)
Things are not quite that simple ... and I would say that if you 
downloading individual package versions, then I would claim that a git 
pull is much simpler than hand editting ConfigurationOf files to change 
the version of the package you are using ...

If you are not specifically managing the versions of packages and 
projects (i.e., not hand editing ConfigurationOf files) then you are at 
the mercy of the project developers as to which version of the project 
you pick up when you update and have no control (other than hand 
editting ConfigurationOf files) over which version you are actually 
using in your product ...

I would argue that with git clones you know exactly which version of 
each of the projects you are using ... and you are in complete control 
over when you want to use a new version ...

With all of the said, you are working with a legacy production loading 
scheme and if you are going to pay the price of converting the load 
system, I would recommend that you wait until GemStone 3.5 (planned, not 
committed) comes out with Rowan support ... at which time package 
loading, Metacello and git support will be incorporated in the GemStone 
kernel and be supported by GemTalk Systems instead of on community 
support, while still maintaining open source status for those tools...
>
> If you want to play along, my answers to Dale:
>
>
> On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> > I highly recommend to the bootstrapGsDevKit approach, but if you don't
> > want to take that route, then you should try splitting your load into
> > two phases 1) fetch and load GLASS 2) fetch and load your application.
>
> That is actually exactly what our code does. When you call WonkaLoader 
> class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: 
> (see the call in topaz/makeCIImage.topaz [1] ):
>  - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, 
> GLASS1 (in that order, from locally cloned git repositories); and
>  - then it loads group 'CI' of BaselineOfWonkaRuntime from 
> wonkaRoot/monticello repository (this is where our own code sits).
Okay that makes sense, but I can't tell that from the loading Transcript 
... dumping into the Transcript the steps that are performed by Wonka 
(and the Metacello or other load expression used) would help me 
understand where the transitions are rather than trying to interpret 
things from a pure Metacello log ...
>
> But, all the loading that takes place - whether loading a "bootstrap 
> dependency" (the GLASS/Metacello stuff) or any other package to load - 
> is done in the same method: WonkaLoader>>loadGroup [2], using the same 
> code to create a Metacello [3] and [4].
Okay ... skipping ahead a bit I don't see a Smalltalk stack and without 
the wonka loader markers in the new log[3], I don't really know when or 
what happened so save reading the various code snippets until I can get 
my hands on a stack ...

[3] 
https://github.com/finworks/BaselineOfWonkaRuntime/commit/6316b77d63189f5c47407403d6a9f7f1e37f048e
>
> In the first phase, (loading bootstrap dependencies), it sets up a new 
> Monticello independently for each of those "bootstrap" projects in 
> turn, on their own (cloned github) repository. ( See 
> BaselineOfWOnkaRuntime class >> 
> loadBootstrapBaseline:repository:modes: [5] )
>
> In the second phase, it uses the same underlying method to set up a 
> new Metacello on our own repository.
>
> What is shared in all of the above is how conflicts etc are handled; 
> and the given repositoryOverrides are always ignored if you do it 
> overNetwork. Further, if you DO load overNetwork, it sets the 
> cacheRepository, and always to the same platform-dependent location 
> (which we check into our git repo).
>
>
> What I understand our not-so-pretty code does in short (ignoring 
> handling of conflicts etc) is:
>
> Metacello new
>   baseline: xxx;
>   repository: aLocalGitClone;
>   cacheRepository: yyy;
>   get.
>
> Metacello new
>   baseline: xxx;
>   repository: aLocalGitClone;
>   cacheRepository: yyy;
>   load: 'default'.
>
> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that 
> order)
>
> Then it does:
>
> Metacello new
>   baseline: xxx;
>   repository: ourOwnFileTreeRepo;
>   cacheRepository: yyy;
>   get.
>
> Metacello new
>   baseline: xxx;
>   repository: ourOwnFileTreeRepo;
>   cacheRepository: yyy;
>   load: 'CI'.
>
>
> Either way, this example (working, if done using all the complications 
> in our scripts) contradicts the documentation [6] if my reasoning is 
> correct.
I guess I'm confused right now ... the log new log does not look like 
"working" to me as there is an error at the very end and I have not seen 
a stack ...
>
>
> I have hastily put a version of the above (without any WonkaLoader 
> stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, 
> with its transcript in logs/transcript.2018-01-09.log  but have not 
> applied my mind to that yet. Obviously this naive version still fails.
it's not obvious to me and without a stack I guess I cannot comment with 
any authority.

If the load process is actually working, then I'm not exactly sure why I 
spent an hour or so on my weekend answering this email message only to 
find out that you are apparently saying "never mind at the end" ...

Dale....