[Glass] rest api returning large result

Sun May 26 08:25:37 PDT 2024

Johan, thank you very much.

Otto Behrens

+27 82 809 2375
[image: FINWorks]
[image: FINWorks] <http://za.linkedin.com/in/waltherbehrens>
www.finworks.biz

Disclaimer & Confidentiality Note: This email is intended solely for the
use of the individual or entity named above as it may contain information
that is confidential and privileged. If you are not the intended recipient,
be advised that any dissemination, distribution or copying of this email is
strictly prohibited. FINWorks cannot be held liable by any person other
than the addressee in respect of any opinions, conclusions, advice or other
information contained in this email.

On Sun, May 26, 2024 at 1:21 PM Johan Brichau via Glass <
glass at lists.gemtalksystems.com> wrote:

> In the meantime, I found the real cause in
> ZnBivalentWriteStream>>#next:putAll:startingAt: in a change that was made
> 11 years ago in commit [1] ((by myself... :roll-eyes:)
>
> The incoming collection argument is converted to a ByteArray on each call
> of this iteration.
> The trouble is that this is the entire collection and not just the part
> that is to be written in the iteration.
> That means that for every 16Kb of data, the entire collection of 55MB (in
> my testcase) is converted to a ByteArray….
>
> I’m working on a fix for this in https://github.com/GsDevKit/zinc/pull/105
> It looks like simply removing the conversion and aligning the code of
> ZnBivalentWriteStream>>#next:putAll:startingAt:  with its original in Pharo
> fixes the problem altogether.
>
> I’m not sure if the change was done inadvertently or if it did fix issues.
> I did some manual tests with Seaside in a GS 3.6.8 and it looked fine.
>
> In any case: fixing this will yield significant performance improvements
> for anyone running Zinc in GemStone…. not just for very large responses
> where the problem is very apparent.
>
> [1]
> https://github.com/GsDevKit/zinc/commit/cbddf8b52589ad2107feba8557e27db5cad5acbd#diff-b940261c40f3e320ede03be3b7ac8c7fe5ff81ec310656aa6b557b069cce1dc2
>
>
> On 25 May 2024, at 10:32, Otto Behrens <otto at finworks.biz> wrote:
>
> Johan, this is magic, thanks a lot. Wow
>
> On Sat, 25 May 2024, 09:11 Johan Brichau via Glass, <
> glass at lists.gemtalksystems.com> wrote:
>
>> Other observations (before I head off doing other worldly things one has
>> to do on Saturdays ;-):
>>
>> - Changing `ZnUtils>>streamingBufferSize` from 16384 to 163840 (i.e. x10)
>> also improves the performance (as expected) to 4s response time for the
>> 55MB file
>> - In Pharo, the same code is used and there it takes 3s for the same
>> file, same code (not quite the same Zinc version though, but the methods I
>> mentioned are still the same)
>>
>> Johan
>>
>> On 25 May 2024, at 08:57, Johan Brichau <johan at yesplan.be> wrote:
>>
>> The problem seems to be in `ZnUtils>>nextPutAll:on:` that splits the
>> collection into chunks to write it to the Socket.
>> When I remove that method and just pass through the writing of the
>> collection as `stream nextPutAll: collection`, I go from 39s to 1s (see
>> below, before and after)
>>
>> Now: understand why that piece of code is there and if it’s needed or not
>> :-)
>>
>> Johan
>>
>> johanbrichau at JohansMacBookAir ~ % time curl http://localhost:8383/test
>> --output bla.txt
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
>>  Current
>>                                  Dload  Upload   Total   Spent    Left
>>  Speed
>> 100 55.2M  100 55.2M    0     0  1465k      0  0:00:38  0:00:38 --:--:--
>> 1476k
>> curl http://localhost:8383/test -output bla.txt  0.04s user 0.14s system
>> 0% cpu 38.657 total
>>
>>
>> johanbrichau at JohansMacBookAir ~ % time curl http://localhost:8383/test
>> --output bla.txt
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
>>  Current
>>                                  Dload  Upload   Total   Spent    Left
>>  Speed
>> 100 55.2M  100 55.2M    0     0  45.0M      0  0:00:01  0:00:01 --:--:--
>> 45.0M
>> curl http://localhost:8383/test --output bla.txt  0.01s user 0.07s
>> system 6% cpu 1.255 total
>>
>>
>> On 23 May 2024, at 23:45, Johan Brichau <johan at yesplan.be> wrote:
>>
>> Hi Otto,
>>
>> On 20 May 2024, at 19:08, Otto Behrens <otto at finworks.biz> wrote:
>>
>> Okay, I’ll give that a try and see what I can get when using the Zinc
>> adaptor in GemStone…
>>
>>
>>
>> It took me a bit longer to actually get started since I wanted to debug
>> this on my Mac using a GsDevKit_stones setup and I had to dig into another
>> rabbit hole first :-)
>> Anyway, I setup the simple handler below and I can confirm also notice
>> that a file of 60MB takes 40s to be served and 28MB takes 10s.
>>
>> In Pharo, the 60MB file takes 3s.
>> I’ll dive into this a bit more tomorrow…..
>>
>> get
>>
>> <get>
>> <path: '/'>
>> <produces: 'application/json'>
>>
>> | file |
>> file := GsFile openReadOnServer: '/Users/johanbrichau/testfile'.
>> ^ file contents
>>
>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> https://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> https://lists.gemtalksystems.com/mailman/listinfo/glass
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/archives/glass/attachments/20240526/5b8f10f9/attachment-0001.htm>