[Glass] Possible Bug: String>>#= treats nulls as a terminator

Mon Jan 29 23:34:42 PST 2018

The real problem is String>>#=. It's bizarre that two SequenceableCollections can be #= yet have different #sizes and that for every shared index i, it's not necessarily true that "(one at: i) = (two at: i)":
| one two |

one := String with: $a with: 25 asCharacter with: $b.
two := one copyWithout: one second. 
one = two
	and: [one asArray ~= two asArray
		and: [
			(1 to: (one size min: two size)) anySatisfy: [:i |
				(one at: i) ~= (two at: i)]]].

Java and C# model strings as immutable indexed collections of UTF-16 16-bit code units (meaning surrogate pair-encoded code points require two units), and no normalization is done during comparisons. Instead there are special methods, like Normalize(), that convert a string into a chosen normalized form, and normalized comparisons can then be done on the converted strings. Ignoring the choice of UTF-16, this seems like a better, safer approach if you're still committed to treating strings as indexable character collections.

But I'm not sure how you can fix String or GsFile without breaking backwards compatibility.

> Sent: Monday, January 29, 2018 at 11:44 AM
> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
> To: glass at lists.gemtalksystems.com
> Subject: Re: [Glass] Possible Bug: String>>#= treats nulls as a terminator
>
> 
> 
> On 01/29/2018 01:16 AM, monty via Glass wrote:
> > I was writing tests for stream converter classes that do encoding/decoding from various encodings. But any use of Strings to store binary data is a use case. ByteArray is more appropriate, but GsFile is still byte-character based by default, even when you open files in binary mode (which I assume just disables line ending normalization on Windows).
> This seems like a GemStone bug at the end of the day ... ByteArray and 
> Utf8 are the two classes that _should_ be used, but if GsFile is not 
> handling them well, then that is an issue for us ... I will check this 
> out ...
> 
> Thanks,
> 
> Dale
> 
> >
> >> Sent: Saturday, January 27, 2018 at 12:18 PM
> >> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
> >> To: glass at lists.gemtalksystems.com
> >> Subject: Re: [Glass] Possible Bug: String>>#= treats nulls as a terminator
> >>
> >> Monty,
> >>
> >> Good points ... this "unexpected" behavior of Unicode strings with
> >> respect to control characters has been hard for us to grapple with
> >> internally as well, but this is unicode being unicode. I did notice that
> >> with the exception of code point 173, all of the code points you list
> >> are indeed control characters according the Unicode character table[1].
> >>
> >> Code point 173 is a "Soft Hypen"[2] and doesn't really seem to fit the
> >> description of a control character, so I'm now curious if we might have
> >> a bug here, either in our implementation, the implementation of libICU
> >> or my understanding:)
> >>
> >> I'm curious how you ran across this behavior? The control characters
> >> wouldn't seem to be a normal part of strings intended for display ...
> >>
> >> I'm asking because if there is a use case for providing the old literal
> >> byte comparison operators we can make them available.
> >>
> >> Dale
> >>
> >> [1] https://unicode-table.com/en/#control-character
> >> [2] https://unicode-table.com/en/00AD/
> >>
> >> On 01/27/2018 01:57 AM, monty via Glass wrote:
> >>> My example and thread title were wrong. It skips null *and* various control chars entirely when comparing:
> >>> (0 to: 255) select: [:each |
> >>> 	(String with: $a with: $b) =
> >>> 		(String with: $a with: each asCharacter with: $b)]
> >>>
> >>> which yields:
> >>> anArray( 0, 1, 2, 3, 4, 5, 6, 7, 8, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 127, 128, 129, 130, 131, 132, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 173)
> >>>
> >>> The GS Prog Guide (p. 77) says the ICU lib handles string comparisons internally, and it seems to ignore these characters for the sake of normalization.
> >>>
> >>> But that means it's possible for two Strings to be #= while having different #sizes and indexable characters, and that comparisons between Strings containing binary data aren't reliable, and that other String methods aren't consistent with #=:
> >>> | one two |
> >>> one := String with: $a with: 0 asCharacter with: $b.
> >>> two := String with: $a with: $b.
> >>> one = two
> >>> 	and: [(one at: 1 equals: two) not
> >>> 		and: [(two at: 1 equals: one) not]]
> >>>
> >>> And since GsFile #next and #contents are character based:
> >>> (GsFile open: 'bin.one' mode: 'wb' onClient: false)
> >>> 	nextPutAll: #[100 25 200];
> >>> 	close.
> >>> (GsFile open: 'bin.two' mode: 'wb' onClient: false)
> >>> 	nextPutAll: #[100 200];
> >>> 	close.
> >>> (GsFile open: 'bin.one' mode: 'rb' onClient: false) contents =
> >>> 	(GsFile open: 'bin.two' mode: 'rb' onClient: false) contents.
> >>>
> >>> Consider this more as a "heads-up" for users than a bug report, since this is apparently the intended, documented behavior.
> >>>
> >>>> Sent: Friday, January 26, 2018 at 2:20 AM
> >>>> From: "monty via Glass" <glass at lists.gemtalksystems.com>
> >>>> To: glass at lists.gemtalksystems.com
> >>>> Subject: [Glass] Possible Bug: String>>#= treats nulls as a terminator
> >>>>
> >>>> Is this correct?
> >>>>
> >>>> (String with: 12 asCharacter with: 0 asCharacter) =
> >>>>       (String with: 12 asCharacter with: 0 asCharacter with: 32 asCharacter)
> >>>>
> >>>> Other string methods, like #copyAfter:, don't treat null the same way.
> >>>> _______________________________________________
> >>>> Glass mailing list
> >>>> Glass at lists.gemtalksystems.com
> >>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>>>
> >>> _______________________________________________
> >>> Glass mailing list
> >>> Glass at lists.gemtalksystems.com
> >>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >> _______________________________________________
> >> Glass mailing list
> >> Glass at lists.gemtalksystems.com
> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>
> > _______________________________________________
> > Glass mailing list
> > Glass at lists.gemtalksystems.com
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> 
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
> 