[Glass] Another metacello loading puzzle: xml

[monty via Glass]

Dale, if you want to mirror these projects on github (like you do SS), that would be fine. The STHub PharoExtras projects are:

XMLParser
BitmapCharacterSet
OrderPreservingDictionary
XMLWriter
XPath
XMLParserHTML
XMLParserStAX

They have no other dependencies. (XMLPullParser is unrelated and not supported by me; I recommend XMLParserStAX instead.)

Upgrading within a stone isn't supported because to run on GS, these projects rely on GS compatibility .mczs and some Configuration postLoad doits (to explicitly initialize normally lazily initialized class state), and there are issues with Monticello and extension packages on GS where loading a new version of the core package deletes the extensions, and unless the extension package has a new version also, Metacello won't reload it.

I wrote to you about this in 2016:

> (simulating upgrading to a new version that uses the same GS compat .mcz). run the tests again and many fail, because extension methods under "*XML-Parser-GemstoneCompatibility" (eg in Collection or GsFile) disappeared when you reloaded the "XML-Parser" .mcz. This shouldn't happen. I think it's due to this bug (which was fixed on Pharo): http://lists.gforge.inria.fr/pipermail/pharo-bugtracker/2015-May/071136.html

(Note: none of my extension packages overwrite any existing methods; they only add new methods and classes.)

> Sent: Sunday, January 14, 2018 at 8:55 PM
> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
> To: glass at lists.gemtalksystems.com
> Subject: Re: [Glass] Another metacello loading puzzle: xml
>
> 
> 
> On 1/9/18 4:02 AM, Iwan Vosloo via Glass wrote:
> > Hi guys,
> >
> > On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:
> > > Just a small comment from the outside: that problem seems to be
> > > known: 
> > http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
> > > Maybe Iwan can check if he has instances of QuadByteString too...
> > > that might be the reason...
> >
> > On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> >> Getting rid of the QuadByteStrings in the mcz file is one solution the 
> > > other solution is to save the package in Filetree format, since
> > > Filetree uses utf8.
> >
> >
> >
> > Thanks Mariano and Dale,
> >
> > Yes, I eventually discovered the QuadByteString issue... I tried using 
> > filetree initially, but given the code that's there (that WonkaLoader 
> > thing) it is not straightforward to use filetree (I suspect because it 
> > uses FileTree to fetch FileTree as well, there may be a 
> > chicken-and-egg scenario going on...but I did not investigate).
> >
> > Instead I figured the QuadByteString issue must be a bug, since 
> > writeWordLike: is being called from code present in Gemstone, yet it 
> > is not implemented in Gemstone.
> >
> > I provided an implementation (should I log a bug somewhere?), and 
> > using that it actually loads using our scripts. Which is nice because 
> > it gives me a starting point of sorts from where I can refactor and 
> > simplify.
> This is not a correct solution ... the byte patterns differ between 
> WideString and QuadByteString ... if it is working it is likely to be 
> luck ... or the corruption is only showing up in a comment ... or the 
> code compiles without error, but will fail when run ... really the only 
> solution in your case is to "edit the Monticello .mcz file" and remove 
> your patch ... it is much safer to have the load fail than debug the 
> potential downstream problems...
> 
> To edit a Monticello .mcz file with WideStrings, you should load the mcz 
> file into a pharo (or squeak) image ... from there your best bet would 
> seem to be to inspect the package structure and get the snapshot 
> definitions and find look for MCMethodDefinitions whose #source is a 
> kind of WideString also look for a class definition #comment that might 
> contain a WideString ... if you find a comment or method with a 
> WideString, then edit the method(s) and comment(s) to eliminate the 
> WideString character(s) save a new version of the .mcz file and then 
> edit the appropriate ConfigurationOf to point to the new .mcz version 
> and try the load again ...
> 
> If you are still getting an error or you didn't find any methods or 
> comments with WideStrings, then it is likely that a commit comment in 
> the .mcz contains the offending character (and there may be more than 
> one comment with the problem) or the problem might be another of the 
> fields ... all it takes is one WideString to spoil the entire package.
> 
> Here's some sample code:
> 
> | basePackageName wc comments methods versions badDefs |
> basePackageName := 'Metacello-Base'.
> wc := MCWorkingCopy allManagers detect: [:each | each packageName = 
> basePackageName].
> comments := OrderedCollection new..
> methods := OrderedCollection new.
> wc snapshot definitions do: [:def |
>      def isClassDefinition
>          ifTrue: [ (def comment isKindOf: WideString) ifTrue: [ comments 
> add: def]].
>      def isMethodDefinition
>          ifTrue: [ (def source isKindOf: WideString) ifTrue: [ comments 
> add: def]]].
> badDefs := OrderedCollection new.
> comments isEmpty ifFalse: [ badDefs add: comments ].
> methods isEmpty ifFalse: [ badDefs add: methods ].
> badDefs isEmpty ifFalse: [ ^ badDefs ].
> "now scan wc ancestry for WideStrings"
> versions := OrderedCollection new.
> wc ancestry allAncestorsDo: [:aMCVersionInfo |
>      (aMCVersionInfo message isKindOf: WideString) ifTrue: [ versions 
> add: aMCVersionInfo ] ].
> versions
> >
> > Of course there's strong incentive to just leave it at that from the 
> > people who have to pay the $$ for this... But I would have liked to be 
> > able to load it all (via/not via network) using only what is provided 
> > by GsUpgrader and Metacello without all the various things in our 
> > WonkaLoader class....
> Yeah the FinWorks WonkaLoader likely predates Metacello let alone 
> GsUpgrader and GsDevKit_home...
> >
> > It is painful to have to get all the right branches of all the right 
> > projects from all the right github repositories locally just to be 
> > able to load...
> Hmmm, it seems that there is some pain associated with the current 
> scheme and once you do clone the projects and get them to the right 
> version then you will be able to load without network access all of the 
> time and when it is a good time for you to update the version of a 
> project you are using, you can do it on your own schedule when you are 
> prepared for the porting effort ...
> 
> A quick scan o f the http: projects that you are using several different 
> http: repositories ... Other than Seaside which turns into a github 
> download for GemStone anyway and Metacello (which doesn't need http: 
> access if you use bootstrapGsDevKit[2]) , the other http: repositories 
> I've identified are:
> 
>    http://seaside.gemtalksystems.com/ss/hyper
>    http://smalltalkhub.com/mc/hernan/XMLPullParser/main
>    http://smalltalkhub.com/mc/PharoExtras/XMLParser/main
> http://smalltalkhub.com/mc/PharoExtras/OrderPreservingDictionary/main
>    http://www.smalltalkhub.com/mc/PharoExtras/BitmapCharacterSet/main
>    http://smalltalkhub.com/mc/Swazoo/Swazoo/main
>    http://seaside.gemtalksystems.com/ss/swazoo2
> 
> hyper also uses a local clone for bootstrapGsDevKit[2], but the others 
> have not been moved to Github yet ... are you using Swazoo for your web 
> server? If not then you are down to only 4 different projects based on 
> http: and I'm thinking that the XML projects need to move to GitHub soon 
> and have already heard from Monty that he would be willing to do that 
> ... The other two OrderPreservingDictionary and  BitmapCharacterSet 
> presumably aren't changing that frequently to become difficult to manage 
> as a git clone ....
> 
>   As for the github: projects that aren't already covered by 
> bootstrapGsDevKit[2] it seems that only SeasideSt/Seaside is being used ...
> 
> I don't know, but from where I am sitting, it would be easier to manage 
> git repositories than try to figure out why your scripts aren't working:)
> 
> Oh well...
> 
> [1] 
> https://raw.githubusercontent.com/finworks/BaselineOfWonkaRuntime/master/logs/transcript.2018-01-01.log
> [2] 
> https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit 
> 
> > being able to fetch the correct dependencies at their correct 
> > versions, from wherever, recursively is exactly the point of using 
> > metacello in the first place. Having to manage that ourselves defeats 
> > the point of using Metacello in the first place..
> > (Besides, many of our dependencies are just Monticello packages anyways)
> Things are not quite that simple ... and I would say that if you 
> downloading individual package versions, then I would claim that a git 
> pull is much simpler than hand editting ConfigurationOf files to change 
> the version of the package you are using ...
> 
> If you are not specifically managing the versions of packages and 
> projects (i.e., not hand editing ConfigurationOf files) then you are at 
> the mercy of the project developers as to which version of the project 
> you pick up when you update and have no control (other than hand 
> editting ConfigurationOf files) over which version you are actually 
> using in your product ...
> 
> I would argue that with git clones you know exactly which version of 
> each of the projects you are using ... and you are in complete control 
> over when you want to use a new version ...
> 
> With all of the said, you are working with a legacy production loading 
> scheme and if you are going to pay the price of converting the load 
> system, I would recommend that you wait until GemStone 3.5 (planned, not 
> committed) comes out with Rowan support ... at which time package 
> loading, Metacello and git support will be incorporated in the GemStone 
> kernel and be supported by GemTalk Systems instead of on community 
> support, while still maintaining open source status for those tools...
> >
> > If you want to play along, my answers to Dale:
> >
> >
> > On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> > > I highly recommend to the bootstrapGsDevKit approach, but if you don't
> > > want to take that route, then you should try splitting your load into
> > > two phases 1) fetch and load GLASS 2) fetch and load your application.
> >
> > That is actually exactly what our code does. When you call WonkaLoader 
> > class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: 
> > (see the call in topaz/makeCIImage.topaz [1] ):
> >  - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, 
> > GLASS1 (in that order, from locally cloned git repositories); and
> >  - then it loads group 'CI' of BaselineOfWonkaRuntime from 
> > wonkaRoot/monticello repository (this is where our own code sits).
> Okay that makes sense, but I can't tell that from the loading Transcript 
> ... dumping into the Transcript the steps that are performed by Wonka 
> (and the Metacello or other load expression used) would help me 
> understand where the transitions are rather than trying to interpret 
> things from a pure Metacello log ...
> >
> > But, all the loading that takes place - whether loading a "bootstrap 
> > dependency" (the GLASS/Metacello stuff) or any other package to load - 
> > is done in the same method: WonkaLoader>>loadGroup [2], using the same 
> > code to create a Metacello [3] and [4].
> Okay ... skipping ahead a bit I don't see a Smalltalk stack and without 
> the wonka loader markers in the new log[3], I don't really know when or 
> what happened so save reading the various code snippets until I can get 
> my hands on a stack ...
> 
> [3] 
> https://github.com/finworks/BaselineOfWonkaRuntime/commit/6316b77d63189f5c47407403d6a9f7f1e37f048e
> >
> > In the first phase, (loading bootstrap dependencies), it sets up a new 
> > Monticello independently for each of those "bootstrap" projects in 
> > turn, on their own (cloned github) repository. ( See 
> > BaselineOfWOnkaRuntime class >> 
> > loadBootstrapBaseline:repository:modes: [5] )
> >
> > In the second phase, it uses the same underlying method to set up a 
> > new Metacello on our own repository.
> >
> > What is shared in all of the above is how conflicts etc are handled; 
> > and the given repositoryOverrides are always ignored if you do it 
> > overNetwork. Further, if you DO load overNetwork, it sets the 
> > cacheRepository, and always to the same platform-dependent location 
> > (which we check into our git repo).
> >
> >
> > What I understand our not-so-pretty code does in short (ignoring 
> > handling of conflicts etc) is:
> >
> > Metacello new
> >   baseline: xxx;
> >   repository: aLocalGitClone;
> >   cacheRepository: yyy;
> >   get.
> >
> > Metacello new
> >   baseline: xxx;
> >   repository: aLocalGitClone;
> >   cacheRepository: yyy;
> >   load: 'default'.
> >
> > For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that 
> > order)
> >
> > Then it does:
> >
> > Metacello new
> >   baseline: xxx;
> >   repository: ourOwnFileTreeRepo;
> >   cacheRepository: yyy;
> >   get.
> >
> > Metacello new
> >   baseline: xxx;
> >   repository: ourOwnFileTreeRepo;
> >   cacheRepository: yyy;
> >   load: 'CI'.
> >
> >
> > Either way, this example (working, if done using all the complications 
> > in our scripts) contradicts the documentation [6] if my reasoning is 
> > correct.
> I guess I'm confused right now ... the log new log does not look like 
> "working" to me as there is an error at the very end and I have not seen 
> a stack ...
> >
> >
> > I have hastily put a version of the above (without any WonkaLoader 
> > stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, 
> > with its transcript in logs/transcript.2018-01-09.log  but have not 
> > applied my mind to that yet. Obviously this naive version still fails.
> it's not obvious to me and without a stack I guess I cannot comment with 
> any authority.
> 
> If the load process is actually working, then I'm not exactly sure why I 
> spent an hour or so on my weekend answering this email message only to 
> find out that you are apparently saying "never mind at the end" ...
> 
> Dale....
> 
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>