[Glass] Another metacello loading puzzle: xml

Tue Jan 9 04:02:08 PST 2018

Hi guys,

On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:
 > Just a small comment from the outside: that problem seems to be
 > known: http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
 > Maybe Iwan can check if he has instances of QuadByteString too...
 > that might be the reason...

On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> Getting rid of the QuadByteStrings in the mcz file is one solution the 
 > other solution is to save the package in Filetree format, since
 > Filetree uses utf8.

Thanks Mariano and Dale,

Yes, I eventually discovered the QuadByteString issue... I tried using 
filetree initially, but given the code that's there (that WonkaLoader 
thing) it is not straightforward to use filetree (I suspect because it 
uses FileTree to fetch FileTree as well, there may be a chicken-and-egg 
scenario going on...but I did not investigate).

Instead I figured the QuadByteString issue must be a bug, since 
writeWordLike: is being called from code present in Gemstone, yet it is 
not implemented in Gemstone.

I provided an implementation (should I log a bug somewhere?), and using 
that it actually loads using our scripts. Which is nice because it gives 
me a starting point of sorts from where I can refactor and simplify.

Of course there's strong incentive to just leave it at that from the 
people who have to pay the $$ for this... But I would have liked to be 
able to load it all (via/not via network) using only what is provided by 
GsUpgrader and Metacello without all the various things in our 
WonkaLoader class....

It is painful to have to get all the right branches of all the right 
projects from all the right github repositories locally just to be able 
to load...being able to fetch the correct dependencies at their correct 
versions, from wherever, recursively is exactly the point of using 
metacello in the first place. Having to manage that ourselves defeats 
the point of using Metacello in the first place... (Besides, many of our 
dependencies are just Monticello packages anyways)

If you want to play along, my answers to Dale:

On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
 > I highly recommend to the bootstrapGsDevKit approach, but if you don't
 > want to take that route, then you should try splitting your load into
 > two phases 1) fetch and load GLASS 2) fetch and load your application.

That is actually exactly what our code does. When you call WonkaLoader 
class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: (see 
the call in topaz/makeCIImage.topaz [1] ):
  - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, 
GLASS1 (in that order, from locally cloned git repositories); and
  - then it loads group 'CI' of BaselineOfWonkaRuntime from 
wonkaRoot/monticello repository (this is where our own code sits).

But, all the loading that takes place - whether loading a "bootstrap 
dependency" (the GLASS/Metacello stuff) or any other package to load - 
is done in the same method: WonkaLoader>>loadGroup [2], using the same 
code to create a Metacello [3] and [4].

In the first phase, (loading bootstrap dependencies), it sets up a new 
Monticello independently for each of those "bootstrap" projects in turn, 
on their own (cloned github) repository. ( See BaselineOfWOnkaRuntime 
class >> loadBootstrapBaseline:repository:modes: [5] )

In the second phase, it uses the same underlying method to set up a new 
Metacello on our own repository.

What is shared in all of the above is how conflicts etc are handled; and 
the given repositoryOverrides are always ignored if you do it 
overNetwork. Further, if you DO load overNetwork, it sets the 
cacheRepository, and always to the same platform-dependent location 
(which we check into our git repo).

What I understand our not-so-pretty code does in short (ignoring 
handling of conflicts etc) is:

Metacello new
   baseline: xxx;
   repository: aLocalGitClone;
   cacheRepository: yyy;
   get.

Metacello new
   baseline: xxx;
   repository: aLocalGitClone;
   cacheRepository: yyy;
   load: 'default'.

For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that order)

Then it does:

Metacello new
   baseline: xxx;
   repository: ourOwnFileTreeRepo;
   cacheRepository: yyy;
   get.

Metacello new
   baseline: xxx;
   repository: ourOwnFileTreeRepo;
   cacheRepository: yyy;
   load: 'CI'.

Either way, this example (working, if done using all the complications 
in our scripts) contradicts the documentation [6] if my reasoning is 
correct.

I have hastily put a version of the above (without any WonkaLoader 
stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, with 
its transcript in logs/transcript.2018-01-09.log  but have not applied 
my mind to that yet. Obviously this naive version still fails.

> and it looks like you are running with a pure fetch load and you are 
> doing of fetch of the not only GLASS/GsDevKit, but your entire application.

Not quite. WonkaLoader is used to load a group from 
BaselineOfWonkaRuntime, and the baseline only contains dependencies to 
stuff we need, not any of our actual own code (besides what is contained 
in BaselineOfWonkaRuntime package itself).

...and the two-phase load does happen, as I explained above.

(I added some detail about our project layout at the bottom of the 
README.rst file in github)

> My second thought is that I've recently been working on getting 
> GsDevKit/GLASS to be bootstrapped using only git repositories (i.e., no 
> network) and this script[2] (within the context of GsDevKit_home) does 
> just that. It should be straightforward to adapt the script to a 
> non-GsDevKit_home
> 
> So if you are struggling with getting the networkless load to work it 
> may make sense for you to use the bootstrapGsDevKit script ... and then 
> work on getting all of the other monticello-based projects that you are 
> using up on github, so that your entire application load can be done 
> without hitting the network ...
> 
> If you take this route then we can ignore the #'writeWordLike:' and also 
> ignore the cacheRepository:/repositoryOverrides issues.

We are currently using git repos, at least for the GLASS bits, but we 
would prefer not to. We'd prefer going with 
cacheRepository:/repositoryOverrides and rely on Metacello to fetch 
whatever is needed without us having to know the details....as I 
explained above. Managing all those dependencies is what we understand 
Metacello's purpose in life is... Cant we help bash it until it works 
for this use case?

There is a bit more to it: the way we handle load conflicts... but I'll 
leave that for a later discussion perhaps.

[1] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImage.topaz

[2] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/loadGroup..st

[3] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/initializeBaseName.repository.modes.repositoryOverrides..st

[4] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/metacelloForLoading.st

[5] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadBootstrapBaseline.repository.modes..st

[6] 
https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching

[7] 
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImageUsingStandardTools.topaz

-- 

Regards
- Iwan