[Glass] Can I avoid ReversedRangeIndexReadStream >> atEnd ?

[Mariano Martinez Peck via Glass]

Hi Dale,

I am reviving this old thread. So...with the legacy indexes I had something
like this:

stream := query reversedReadStream.
"We are using #_atEndForStream: and #getNext because of performance
reasons. For more details see email 'Can I avoid
ReversedRangeIndexReadStream >> atEnd ?'
In GemStone 3.4 we may be able to pass a false below... "
            ^ (stream _atEndForStream: false)
                ifTrue: [ nil ]
                ifFalse: [ stream getNext ]


As I was sure there was no nils in the collection and that way, it was much
faster (else, #atEnd was too slow).

I just upgraded to use the new btreeplus indexes but I am not sure if I
should do something in particular for above speedup. I am using now the
default BtreePlusGsIndexReadStream >> atEnd. Is that correct or there could
be a fastest path when I am sure there is no nils?

Thanks in advance,


On Thu, Oct 26, 2017 at 9:56 AM, Mariano Martinez Peck <
marianopeck at gmail.com> wrote:

>
>
> On Wed, Oct 25, 2017 at 5:27 PM, Dale Henrichs <
> dale.henrichs at gemtalksystems.com> wrote:
>
>> Mariano,
>>
>> It is possible that there is a bug in reverse read stream that is masked
>> by fact that the comparison code in the loop is returning true at the right
>> time  ... while working on the btree plus implementation I did find some
>> bugs the reversed read stream implementation that might explain this
>> behavior ... the btree plus reverse read stream implementation is very
>> different because in the btree plus implementation the leaf nodes are
>> doubly linked lists and the previous/next leaks are used ...
>>
>> I would suggest that you wait for 3.4 to be released and  use btree plus
>> indexes ... we have an Alpha5 release that is pretty close to the final
>> release that you could take for a spin --- you should definitely see better
>> performance with the btree plus indexes for your use case ...
>>
>
> OK, thanks Dale for the answer. I guess I will wait then. Thanks for the
> offer of Alpha5 but I guess I would rather wait a bit until final release.
>
> Cheers,
>
>
>
>> Dale
>>
>> On 10/25/2017 12:14 PM, Mariano Martinez Peck wrote:
>>
>>
>>
>> On Wed, Oct 25, 2017 at 3:07 PM, Dale Henrichs via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>>> Mariano,
>>>
>>> The costly code is in RangeIndexReadStream>>_atEndForStream: (when
>>> aBool is true) and that code can be safely skipped if the values in the
>>> index will not include nils, or if your index is on a CharacterCollection
>>> and you do not mix Strings and Symbols or if your index is on a Float and
>>> the values do not include NaNs.
>>>
>>
>> Hi Dale,
>>
>> In my case, the collection is of instances of FaSecurityClosingPriceRecord
>> and as you can see in my original email the index is on a date type of
>> instVar.
>> If you see code below:
>>
>>              query := self
>>                 nearestQueryFor: dateToCompare
>>                 on: (self indexingDict at: securityUniqueId).
>>               stream := query reversedReadStream.
>>               ^ (stream _atEndForStream: false)
>>                 ifTrue: [ nil ]
>>                 ifFalse: [ stream getNext ]
>>
>>
>> I can assure you that:
>>
>> 1) Values of the indexes are always dates.
>> 2) I have no FaSecurityClosingPriceRecord instance with nil date
>> 3) I have no nil in the collection passed by to `nearestQueryFor:
>> dateToCompare  on:`
>>
>> Yet....I am getting different results wether I pass `false` or `true` to
>> #_atEndForStream:   :(
>> And even worst, it looks like the correct one is when I pass false...
>>
>> Maybe I am hitting an scenario where `true` is needed that is not in the
>> ones you listed?
>> I can confirm at least that the speedup when using `false` was huge, so I
>> would love to find a way to skip that!
>>
>> Thanks in advance,
>>
>>
>> In 3.4 we do this automatically when you create a btree plus index with
>>> optmiizedComparison index option ... the 3.4 stream operations have been
>>> optimized in other ways as well --- 3.4 is due to be released very soon ...
>>>
>>> Dale
>>>
>>> On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass wrote:
>>>
>>> Hi Dale,
>>>
>>> I am doing a query like this (as per your recommendation):
>>>
>>>
>>> nearestQueryFor: aDate on: aCollection
>>> | query |
>>> query := GsQuery fromFormula: self nearestQuery  on: aCollection.
>>> query bind: 'targetDate' to: aDate.
>>> ^ query
>>>
>>> nearestQuery
>>> ^ nearestQuery ifNil: [
>>> nearestQuery := GsQueryPredicate fromQueryStatement:  'each.date <=
>>> targetDate'.
>>> nearestQuery
>>> ].
>>>
>>>
>>> And when I use it, I do this:
>>>
>>> query := self nearestQueryFor: dateToCompare on: (self indexingDict at:
>>> securityUniqueId).
>>> stream := query reversedReadStream.
>>>             ^ stream atEnd
>>>                 ifTrue: [ nil ]
>>>                 ifFalse: [ stream next ]
>>>
>>> I am profiling some code that uses this a lot and I see that quite some
>>> time is spent in the #atEnd.
>>> I attach a part of the profiling output to see what I mean.
>>>
>>> Do you see something obvious I can speed up?
>>>
>>> Thanks in advance,
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>>
>>> _______________________________________________
>>> Glass mailing listGlass at lists.gemtalksystems.comhttp://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>>>
>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20171208/e089851f/attachment-0001.html>