[Glass] Can I avoid ReversedRangeIndexReadStream >> atEnd ?
Mariano Martinez Peck via Glass
glass at lists.gemtalksystems.com
Mon Dec 11 10:19:01 PST 2017
On Mon, Dec 11, 2017 at 3:09 PM, Dale Henrichs <
dale.henrichs at gemtalksystems.com> wrote:
> Mariano,
>
> The btree plus indexes (with optimizedComparison option - recommended) do
> not allow nil values ... also there are new classes being used for Streams
> (see BtreePlusReadStream) and a number of other other places --- in
> converting to the new set of classes I did a fair amount of optimization
> and beefed up test coverage considerably ... with that said I would be
> interested in seeing any performance comparisons that you might have for
> your use case ... I think that there is still room for improvement, but
> there are quite a few different dimensions that can be looked at with
> tradeoffs for different schemes, so I would prefer to focus on performance
> issues for real life use cases, then spend a lot of time and effort
> optimizing a use case that is not being used ...
>
>
Hi Dale,
I forgot to answer myself on Friday. I went ahead with my code and I let
the old #atEnd rather than my hack on _atEndForStream). Later, I profiled
the scenario in which this was a bottleneck before, and from what I can see
now, that is not a bottleneck anymore :)
So...already enjoying the new indexes! Good job!
> Dale
>
> On 12/8/17 3:16 PM, Mariano Martinez Peck wrote:
>
> Hi Dale,
>
> I am reviving this old thread. So...with the legacy indexes I had
> something like this:
>
> stream := query reversedReadStream.
> "We are using #_atEndForStream: and #getNext because of performance
> reasons. For more details see email 'Can I avoid
> ReversedRangeIndexReadStream >> atEnd ?'
> In GemStone 3.4 we may be able to pass a false below... "
> ^ (stream _atEndForStream: false)
> ifTrue: [ nil ]
> ifFalse: [ stream getNext ]
>
>
> As I was sure there was no nils in the collection and that way, it was
> much faster (else, #atEnd was too slow).
>
> I just upgraded to use the new btreeplus indexes but I am not sure if I
> should do something in particular for above speedup. I am using now the
> default BtreePlusGsIndexReadStream >> atEnd. Is that correct or there
> could be a fastest path when I am sure there is no nils?
>
> Thanks in advance,
>
>
> On Thu, Oct 26, 2017 at 9:56 AM, Mariano Martinez Peck <
> marianopeck at gmail.com> wrote:
>
>>
>>
>> On Wed, Oct 25, 2017 at 5:27 PM, Dale Henrichs <
>> dale.henrichs at gemtalksystems.com> wrote:
>>
>>> Mariano,
>>>
>>> It is possible that there is a bug in reverse read stream that is masked
>>> by fact that the comparison code in the loop is returning true at the right
>>> time ... while working on the btree plus implementation I did find some
>>> bugs the reversed read stream implementation that might explain this
>>> behavior ... the btree plus reverse read stream implementation is very
>>> different because in the btree plus implementation the leaf nodes are
>>> doubly linked lists and the previous/next leaks are used ...
>>>
>>> I would suggest that you wait for 3.4 to be released and use btree plus
>>> indexes ... we have an Alpha5 release that is pretty close to the final
>>> release that you could take for a spin --- you should definitely see better
>>> performance with the btree plus indexes for your use case ...
>>>
>>
>> OK, thanks Dale for the answer. I guess I will wait then. Thanks for the
>> offer of Alpha5 but I guess I would rather wait a bit until final release.
>>
>> Cheers,
>>
>>
>>
>>> Dale
>>>
>>> On 10/25/2017 12:14 PM, Mariano Martinez Peck wrote:
>>>
>>>
>>>
>>> On Wed, Oct 25, 2017 at 3:07 PM, Dale Henrichs via Glass <
>>> glass at lists.gemtalksystems.com> wrote:
>>>
>>>> Mariano,
>>>>
>>>> The costly code is in RangeIndexReadStream>>_atEndForStream: (when
>>>> aBool is true) and that code can be safely skipped if the values in the
>>>> index will not include nils, or if your index is on a CharacterCollection
>>>> and you do not mix Strings and Symbols or if your index is on a Float and
>>>> the values do not include NaNs.
>>>>
>>>
>>> Hi Dale,
>>>
>>> In my case, the collection is of instances of FaSecurityClosingPriceRecord
>>> and as you can see in my original email the index is on a date type of
>>> instVar.
>>> If you see code below:
>>>
>>> query := self
>>> nearestQueryFor: dateToCompare
>>> on: (self indexingDict at: securityUniqueId).
>>> stream := query reversedReadStream.
>>> ^ (stream _atEndForStream: false)
>>> ifTrue: [ nil ]
>>> ifFalse: [ stream getNext ]
>>>
>>>
>>> I can assure you that:
>>>
>>> 1) Values of the indexes are always dates.
>>> 2) I have no FaSecurityClosingPriceRecord instance with nil date
>>> 3) I have no nil in the collection passed by to `nearestQueryFor:
>>> dateToCompare on:`
>>>
>>> Yet....I am getting different results wether I pass `false` or `true` to
>>> #_atEndForStream: :(
>>> And even worst, it looks like the correct one is when I pass false...
>>>
>>> Maybe I am hitting an scenario where `true` is needed that is not in the
>>> ones you listed?
>>> I can confirm at least that the speedup when using `false` was huge, so
>>> I would love to find a way to skip that!
>>>
>>> Thanks in advance,
>>>
>>>
>>> In 3.4 we do this automatically when you create a btree plus index with
>>>> optmiizedComparison index option ... the 3.4 stream operations have been
>>>> optimized in other ways as well --- 3.4 is due to be released very soon ...
>>>>
>>>> Dale
>>>>
>>>> On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass wrote:
>>>>
>>>> Hi Dale,
>>>>
>>>> I am doing a query like this (as per your recommendation):
>>>>
>>>>
>>>> nearestQueryFor: aDate on: aCollection
>>>> | query |
>>>> query := GsQuery fromFormula: self nearestQuery on: aCollection.
>>>> query bind: 'targetDate' to: aDate.
>>>> ^ query
>>>>
>>>> nearestQuery
>>>> ^ nearestQuery ifNil: [
>>>> nearestQuery := GsQueryPredicate fromQueryStatement: 'each.date <=
>>>> targetDate'.
>>>> nearestQuery
>>>> ].
>>>>
>>>>
>>>> And when I use it, I do this:
>>>>
>>>> query := self nearestQueryFor: dateToCompare on: (self indexingDict at:
>>>> securityUniqueId).
>>>> stream := query reversedReadStream.
>>>> ^ stream atEnd
>>>> ifTrue: [ nil ]
>>>> ifFalse: [ stream next ]
>>>>
>>>> I am profiling some code that uses this a lot and I see that quite some
>>>> time is spent in the #atEnd.
>>>> I attach a part of the profiling output to see what I mean.
>>>>
>>>> Do you see something obvious I can speed up?
>>>>
>>>> Thanks in advance,
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing listGlass at lists.gemtalksystems.comhttp://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>>
>>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
>
>
--
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20171211/46f64d5d/attachment-0001.html>
More information about the Glass
mailing list