[Glass] Can I avoid ReversedRangeIndexReadStream >> atEnd ?
Dale Henrichs via Glass
glass at lists.gemtalksystems.com
Mon Dec 11 10:09:36 PST 2017
Mariano,
The btree plus indexes (with optimizedComparison option - recommended)
do not allow nil values ... also there are new classes being used for
Streams (see BtreePlusReadStream) and a number of other other places ---
in converting to the new set of classes I did a fair amount of
optimization and beefed up test coverage considerably ... with that said
I would be interested in seeing any performance comparisons that you
might have for your use case ... I think that there is still room for
improvement, but there are quite a few different dimensions that can be
looked at with tradeoffs for different schemes, so I would prefer to
focus on performance issues for real life use cases, then spend a lot of
time and effort optimizing a use case that is not being used ...
Dale
On 12/8/17 3:16 PM, Mariano Martinez Peck wrote:
> Hi Dale,
>
> I am reviving this old thread. So...with the legacy indexes I had
> something like this:
>
> stream := query reversedReadStream.
> "We are using #_atEndForStream: and #getNext because of performance
> reasons. For more details see email 'Can I avoid
> ReversedRangeIndexReadStream >> atEnd ?'
> In GemStone 3.4 we may be able to pass a false below... "
> ^ (stream _atEndForStream: false)
> ifTrue: [ nil ]
> ifFalse: [ stream getNext ]
>
>
> As I was sure there was no nils in the collection and that way, it was
> much faster (else, #atEnd was too slow).
>
> I just upgraded to use the new btreeplus indexes but I am not sure if
> I should do something in particular for above speedup. I am using now
> the default BtreePlusGsIndexReadStream >> atEnd. Is that correct or
> there could be a fastest path when I am sure there is no nils?
>
> Thanks in advance,
>
>
> On Thu, Oct 26, 2017 at 9:56 AM, Mariano Martinez Peck
> <marianopeck at gmail.com <mailto:marianopeck at gmail.com>> wrote:
>
>
>
> On Wed, Oct 25, 2017 at 5:27 PM, Dale Henrichs
> <dale.henrichs at gemtalksystems.com
> <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>
> Mariano,
>
> It is possible that there is a bug in reverse read stream that
> is masked by fact that the comparison code in the loop is
> returning true at the right time ... while working on the
> btree plus implementation I did find some bugs the reversed
> read stream implementation that might explain this behavior
> ... the btree plus reverse read stream implementation is very
> different because in the btree plus implementation the leaf
> nodes are doubly linked lists and the previous/next leaks are
> used ...
>
> I would suggest that you wait for 3.4 to be released and use
> btree plus indexes ... we have an Alpha5 release that is
> pretty close to the final release that you could take for a
> spin --- you should definitely see better performance with the
> btree plus indexes for your use case ...
>
>
> OK, thanks Dale for the answer. I guess I will wait then. Thanks
> for the offer of Alpha5 but I guess I would rather wait a bit
> until final release.
>
> Cheers,
>
> Dale
>
>
> On 10/25/2017 12:14 PM, Mariano Martinez Peck wrote:
>>
>>
>> On Wed, Oct 25, 2017 at 3:07 PM, Dale Henrichs via Glass
>> <glass at lists.gemtalksystems.com
>> <mailto:glass at lists.gemtalksystems.com>> wrote:
>>
>> Mariano,
>>
>> The costly code is in
>> RangeIndexReadStream>>_atEndForStream: (when aBool is
>> true) and that code can be safely skipped if the values
>> in the index will not include nils, or if your index is
>> on a CharacterCollection and you do not mix Strings and
>> Symbols or if your index is on a Float and the values do
>> not include NaNs.
>>
>>
>> Hi Dale,
>>
>> In my case, the collection is of instances
>> of FaSecurityClosingPriceRecord and as you can see in my
>> original email the index is on a date type of instVar.
>> If you see code below:
>>
>> query := self
>> nearestQueryFor: dateToCompare
>> on: (self indexingDict at: securityUniqueId).
>> stream := query reversedReadStream.
>> ^ (stream _atEndForStream: false)
>> ifTrue: [ nil ]
>> ifFalse: [ stream getNext ]
>>
>> I can assure you that:
>>
>> 1) Values of the indexes are always dates.
>> 2) I have no FaSecurityClosingPriceRecord instance with nil date
>> 3) I have no nil in the collection passed by to
>> `nearestQueryFor: dateToCompare on:`
>>
>> Yet....I am getting different results wether I pass `false`
>> or `true` to #_atEndForStream: :(
>> And even worst, it looks like the correct one is when I pass
>> false...
>>
>> Maybe I am hitting an scenario where `true` is needed that is
>> not in the ones you listed?
>> I can confirm at least that the speedup when using `false`
>> was huge, so I would love to find a way to skip that!
>>
>> Thanks in advance,
>>
>>
>> In 3.4 we do this automatically when you create a btree
>> plus index with optmiizedComparison index option ... the
>> 3.4 stream operations have been optimized in other ways
>> as well --- 3.4 is due to be released very soon ...
>>
>> Dale
>>
>>
>> On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass
>> wrote:
>>> Hi Dale,
>>>
>>> I am doing a query like this (as per your recommendation):
>>>
>>>
>>> nearestQueryFor: aDate on: aCollection
>>> | query |
>>> query := GsQuery fromFormula: self nearestQuery on:
>>> aCollection.
>>> query bind: 'targetDate' to: aDate.
>>> ^ query
>>>
>>> nearestQuery
>>> ^ nearestQuery ifNil: [
>>> nearestQuery := GsQueryPredicate fromQueryStatement:
>>> 'each.date <= targetDate'.
>>> nearestQuery
>>> ].
>>>
>>>
>>> And when I use it, I do this:
>>>
>>> query := self nearestQueryFor: dateToCompare on: (self
>>> indexingDict at: securityUniqueId).
>>> stream := query reversedReadStream.
>>> ^ stream atEnd
>>> ifTrue: [ nil ]
>>> ifFalse: [ stream next ]
>>>
>>> I am profiling some code that uses this a lot and I see
>>> that quite some time is spent in the #atEnd.
>>> I attach a part of the profiling output to see what I mean.
>>>
>>> Do you see something obvious I can speed up?
>>>
>>> Thanks in advance,
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>> <http://marianopeck.wordpress.com>
>>>
>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> <mailto:Glass at lists.gemtalksystems.com>
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> <mailto:Glass at lists.gemtalksystems.com>
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>> <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>> <http://marianopeck.wordpress.com>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20171211/afd73d5e/attachment-0001.html>
More information about the Glass
mailing list