[Glass] Possible Bug: String>>#= treats nulls as a terminator

Tue Jan 30 00:48:20 PST 2018

Hi Monty,

> On 30.01.2018, at 08:34, monty via Glass <glass at lists.gemtalksystems.com> wrote:
> 
> The real problem is String>>#=. It's bizarre that two SequenceableCollections can be #= yet have different #sizes and that for every shared index i, it's not necessarily true that "(one at: i) = (two at: i)":

This is, however in line with unicode …
See this very on-point discussion of the matter:

https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/

Best regards
	-Tobias

> | one two |
> 
> one := String with: $a with: 25 asCharacter with: $b.
> two := one copyWithout: one second. 
> one = two
> 	and: [one asArray ~= two asArray
> 		and: [
> 			(1 to: (one size min: two size)) anySatisfy: [:i |
> 				(one at: i) ~= (two at: i)]]].
> 
> Java and C# model strings as immutable indexed collections of UTF-16 16-bit code units (meaning surrogate pair-encoded code points require two units), and no normalization is done during comparisons. Instead there are special methods, like Normalize(), that convert a string into a chosen normalized form, and normalized comparisons can then be done on the converted strings. Ignoring the choice of UTF-16, this seems like a better, safer approach if you're still committed to treating strings as indexable character collections.
> 
> But I'm not sure how you can fix String or GsFile without breaking backwards compatibility.
> 
>> Sent: Monday, January 29, 2018 at 11:44 AM
>> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
>> To: glass at lists.gemtalksystems.com
>> Subject: Re: [Glass] Possible Bug: String>>#= treats nulls as a terminator
>> 
>> 
>> 
>> On 01/29/2018 01:16 AM, monty via Glass wrote:
>>> I was writing tests for stream converter classes that do encoding/decoding from various encodings. But any use of Strings to store binary data is a use case. ByteArray is more appropriate, but GsFile is still byte-character based by default, even when you open files in binary mode (which I assume just disables line ending normalization on Windows).
>> This seems like a GemStone bug at the end of the day ... ByteArray and 
>> Utf8 are the two classes that _should_ be used, but if GsFile is not 
>> handling them well, then that is an issue for us ... I will check this 
>> out ...
>> 
>> Thanks,
>> 
>> Dale
>> 
>>> 
>>>> Sent: Saturday, January 27, 2018 at 12:18 PM
>>>> From: "Dale Henrichs via Glass" <glass at lists.gemtalksystems.com>
>>>> To: glass at lists.gemtalksystems.com
>>>> Subject: Re: [Glass] Possible Bug: String>>#= treats nulls as a terminator
>>>> 
>>>> Monty,
>>>> 
>>>> Good points ... this "unexpected" behavior of Unicode strings with
>>>> respect to control characters has been hard for us to grapple with
>>>> internally as well, but this is unicode being unicode. I did notice that
>>>> with the exception of code point 173, all of the code points you list
>>>> are indeed control characters according the Unicode character table[1].
>>>> 
>>>> Code point 173 is a "Soft Hypen"[2] and doesn't really seem to fit the
>>>> description of a control character, so I'm now curious if we might have
>>>> a bug here, either in our implementation, the implementation of libICU
>>>> or my understanding:)
>>>> 
>>>> I'm curious how you ran across this behavior? The control characters
>>>> wouldn't seem to be a normal part of strings intended for display ...
>>>> 
>>>> I'm asking because if there is a use case for providing the old literal
>>>> byte comparison operators we can make them available.
>>>> 
>>>> Dale
>>>> 
>>>> [1] https://unicode-table.com/en/#control-character
>>>> [2] https://unicode-table.com/en/00AD/
>>>> 
>>>> On 01/27/2018 01:57 AM, monty via Glass wrote:
>>>>> My example and thread title were wrong. It skips null *and* various control chars entirely when comparing:
>>>>> (0 to: 255) select: [:each |
>>>>> 	(String with: $a with: $b) =
>>>>> 		(String with: $a with: each asCharacter with: $b)]
>>>>> 
>>>>> which yields:
>>>>> anArray( 0, 1, 2, 3, 4, 5, 6, 7, 8, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 127, 128, 129, 130, 131, 132, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 173)
>>>>> 
>>>>> The GS Prog Guide (p. 77) says the ICU lib handles string comparisons internally, and it seems to ignore these characters for the sake of normalization.
>>>>> 
>>>>> But that means it's possible for two Strings to be #= while having different #sizes and indexable characters, and that comparisons between Strings containing binary data aren't reliable, and that other String methods aren't consistent with #=:
>>>>> | one two |
>>>>> one := String with: $a with: 0 asCharacter with: $b.
>>>>> two := String with: $a with: $b.
>>>>> one = two
>>>>> 	and: [(one at: 1 equals: two) not
>>>>> 		and: [(two at: 1 equals: one) not]]
>>>>> 
>>>>> And since GsFile #next and #contents are character based:
>>>>> (GsFile open: 'bin.one' mode: 'wb' onClient: false)
>>>>> 	nextPutAll: #[100 25 200];
>>>>> 	close.
>>>>> (GsFile open: 'bin.two' mode: 'wb' onClient: false)
>>>>> 	nextPutAll: #[100 200];
>>>>> 	close.
>>>>> (GsFile open: 'bin.one' mode: 'rb' onClient: false) contents =
>>>>> 	(GsFile open: 'bin.two' mode: 'rb' onClient: false) contents.
>>>>> 
>>>>> Consider this more as a "heads-up" for users than a bug report, since this is apparently the intended, documented behavior.
>>>>> 
>>>>>> Sent: Friday, January 26, 2018 at 2:20 AM
>>>>>> From: "monty via Glass" <glass at lists.gemtalksystems.com>
>>>>>> To: glass at lists.gemtalksystems.com
>>>>>> Subject: [Glass] Possible Bug: String>>#= treats nulls as a terminator
>>>>>> 
>>>>>> Is this correct?
>>>>>> 
>>>>>> (String with: 12 asCharacter with: 0 asCharacter) =
>>>>>>      (String with: 12 asCharacter with: 0 asCharacter with: 32 asCharacter)
>>>>>> 
>>>>>> Other string methods, like #copyAfter:, don't treat null the same way.
>>>>>> _______________________________________________
>>>>>> Glass mailing list
>>>>>> Glass at lists.gemtalksystems.com
>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>> 
>>>>> _______________________________________________
>>>>> Glass mailing list
>>>>> Glass at lists.gemtalksystems.com
>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>> 
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> 
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> 
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass