[Glass] Processing large text files / statmonitor tips

[Richard Sargent]

>> So...first question...is there anything I could do in FileStream and
friends that
>> could have a big impact in the performance of reading text files?

Ultimately, it comes down to reading from the fastest file system available
to you. In other words, do not use GsFile's *client-side* capability for
large volumes of data. Likewise, NFS mounted volumes will be slower than
volumes local to the Gem's host.

And, of course, if you can have the files on a local SSD, that should be
the fastest of all.



On Mon, Dec 9, 2013 at 11:32 AM, Mariano Martinez Peck <
marianopeck at gmail.com> wrote:

> Hi guys,
>
> I am doing a quick bench to see how GemStone behaves for some use-cases I
> have in which lots of large csv (or similar text based) files are read and
> processed. Notice that so far this read is NOT a bulk load (I am not
> persisting this data in GemStone..I am just processing it).
>
> I am trying to see how can I make it a bit faster (I am sure I can make
> it).
> So...first question...is there anything I could do in FileStream and
> friends that could have a big impact in the performance of reading text
> files?
>
> I used ProfMonitor to see which places of my code were the bottleneck, and
> of course, the files reading is most of it. So I wonder if there is
> anything I could do in the Gem or Stone configuration parameters?
>
> Finally, I run statmonitor and I have the vsd file. But since I don't know
> much so far, I wanted to ask if there were some key statistics I should
> look at for my usecase.
>
> Thanks a lot in advance,
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>


-- 
Richard Sargent
Business Development Manager
503-766-4719
richard.sargent at gemtalksystems.comGemTalk Systems
15220 NW Greenbrier Parkway #240
Beaverton, OR 97006
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20131209/e7f725d4/attachment.html>