[Glass] Possible Bug: String>>#= treats nulls as a terminator

[monty via Glass]

I was writing tests for stream converter classes that do encoding/decoding from various encodings. But any use of Strings to store binary data is a use case. ByteArray is more appropriate, but GsFile is still byte-character based by default, even when you open files in binary mode (which I assume just disables line ending normalization on Windows).

> Sent: Saturday, January 27, 2018 at 12:18 PM
> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
> To: glass at lists.gemtalksystems.com
> Subject: Re: [Glass] Possible Bug: String>>#= treats nulls as a terminator
>
> Monty,
> 
> Good points ... this "unexpected" behavior of Unicode strings with 
> respect to control characters has been hard for us to grapple with 
> internally as well, but this is unicode being unicode. I did notice that 
> with the exception of code point 173, all of the code points you list 
> are indeed control characters according the Unicode character table[1].
> 
> Code point 173 is a "Soft Hypen"[2] and doesn't really seem to fit the 
> description of a control character, so I'm now curious if we might have 
> a bug here, either in our implementation, the implementation of libICU 
> or my understanding:)
> 
> I'm curious how you ran across this behavior? The control characters 
> wouldn't seem to be a normal part of strings intended for display ...
> 
> I'm asking because if there is a use case for providing the old literal 
> byte comparison operators we can make them available.
> 
> Dale
> 
> [1] https://unicode-table.com/en/#control-character
> [2] https://unicode-table.com/en/00AD/
> 
> On 01/27/2018 01:57 AM, monty via Glass wrote:
> > My example and thread title were wrong. It skips null *and* various control chars entirely when comparing:
> > (0 to: 255) select: [:each |
> > 	(String with: $a with: $b) =
> > 		(String with: $a with: each asCharacter with: $b)]
> >
> > which yields:
> > anArray( 0, 1, 2, 3, 4, 5, 6, 7, 8, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 127, 128, 129, 130, 131, 132, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 173)
> >
> > The GS Prog Guide (p. 77) says the ICU lib handles string comparisons internally, and it seems to ignore these characters for the sake of normalization.
> >
> > But that means it's possible for two Strings to be #= while having different #sizes and indexable characters, and that comparisons between Strings containing binary data aren't reliable, and that other String methods aren't consistent with #=:
> > | one two |
> > one := String with: $a with: 0 asCharacter with: $b.
> > two := String with: $a with: $b.
> > one = two
> > 	and: [(one at: 1 equals: two) not
> > 		and: [(two at: 1 equals: one) not]]
> >
> > And since GsFile #next and #contents are character based:
> > (GsFile open: 'bin.one' mode: 'wb' onClient: false)
> > 	nextPutAll: #[100 25 200];
> > 	close.
> > (GsFile open: 'bin.two' mode: 'wb' onClient: false)
> > 	nextPutAll: #[100 200];
> > 	close.
> > (GsFile open: 'bin.one' mode: 'rb' onClient: false) contents =
> > 	(GsFile open: 'bin.two' mode: 'rb' onClient: false) contents.
> >
> > Consider this more as a "heads-up" for users than a bug report, since this is apparently the intended, documented behavior.
> >
> >> Sent: Friday, January 26, 2018 at 2:20 AM
> >> From: "monty via Glass" <glass at lists.gemtalksystems.com>
> >> To: glass at lists.gemtalksystems.com
> >> Subject: [Glass] Possible Bug: String>>#= treats nulls as a terminator
> >>
> >> Is this correct?
> >>
> >> (String with: 12 asCharacter with: 0 asCharacter) =
> >>      (String with: 12 asCharacter with: 0 asCharacter with: 32 asCharacter)
> >>
> >> Other string methods, like #copyAfter:, don't treat null the same way.
> >> _______________________________________________
> >> Glass mailing list
> >> Glass at lists.gemtalksystems.com
> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>
> > _______________________________________________
> > Glass mailing list
> > Glass at lists.gemtalksystems.com
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> 
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>