[Glass] rest api returning large result

[Johan Brichau]

In the meantime, I found the real cause in ZnBivalentWriteStream>>#next:putAll:startingAt: in a change that was made 11 years ago in commit [1] ((by myself... :roll-eyes:) 

The incoming collection argument is converted to a ByteArray on each call of this iteration.
The trouble is that this is the entire collection and not just the part that is to be written in the iteration.
That means that for every 16Kb of data, the entire collection of 55MB (in my testcase) is converted to a ByteArray….

I’m working on a fix for this in https://github.com/GsDevKit/zinc/pull/105
It looks like simply removing the conversion and aligning the code of ZnBivalentWriteStream>>#next:putAll:startingAt:  with its original in Pharo fixes the problem altogether.

I’m not sure if the change was done inadvertently or if it did fix issues. I did some manual tests with Seaside in a GS 3.6.8 and it looked fine.

In any case: fixing this will yield significant performance improvements for anyone running Zinc in GemStone…. not just for very large responses where the problem is very apparent. 

[1]  https://github.com/GsDevKit/zinc/commit/cbddf8b52589ad2107feba8557e27db5cad5acbd#diff-b940261c40f3e320ede03be3b7ac8c7fe5ff81ec310656aa6b557b069cce1dc2 

> On 25 May 2024, at 10:32, Otto Behrens <otto at finworks.biz> wrote:
> 
> Johan, this is magic, thanks a lot. Wow
> 
> On Sat, 25 May 2024, 09:11 Johan Brichau via Glass, <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>> Other observations (before I head off doing other worldly things one has to do on Saturdays ;-):
>> 
>> - Changing `ZnUtils>>streamingBufferSize` from 16384 to 163840 (i.e. x10) also improves the performance (as expected) to 4s response time for the 55MB file
>> - In Pharo, the same code is used and there it takes 3s for the same file, same code (not quite the same Zinc version though, but the methods I mentioned are still the same)
>> 
>> Johan
>> 
>>> On 25 May 2024, at 08:57, Johan Brichau <johan at yesplan.be <mailto:johan at yesplan.be>> wrote:
>>> 
>>> The problem seems to be in `ZnUtils>>nextPutAll:on:` that splits the collection into chunks to write it to the Socket.
>>> When I remove that method and just pass through the writing of the collection as `stream nextPutAll: collection`, I go from 39s to 1s (see below, before and after)
>>> 
>>> Now: understand why that piece of code is there and if it’s needed or not :-)
>>> 
>>> Johan
>>> 
>>> johanbrichau at JohansMacBookAir ~ % time curl http://localhost:8383/test --output bla.txt
>>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>                                  Dload  Upload   Total   Spent    Left  Speed
>>> 100 55.2M  100 55.2M    0     0  1465k      0  0:00:38  0:00:38 --:--:-- 1476k
>>> curl http://localhost:8383/test -output bla.txt  0.04s user 0.14s system 0% cpu 38.657 total
>>> 
>>> 
>>> johanbrichau at JohansMacBookAir ~ % time curl http://localhost:8383/test --output bla.txt
>>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>                                  Dload  Upload   Total   Spent    Left  Speed
>>> 100 55.2M  100 55.2M    0     0  45.0M      0  0:00:01  0:00:01 --:--:-- 45.0M
>>> curl http://localhost:8383/test --output bla.txt  0.01s user 0.07s system 6% cpu 1.255 total
>>> 
>>> 
>>>> On 23 May 2024, at 23:45, Johan Brichau <johan at yesplan.be <mailto:johan at yesplan.be>> wrote:
>>>> 
>>>> Hi Otto,
>>>> 
>>>>> On 20 May 2024, at 19:08, Otto Behrens <otto at finworks.biz <mailto:otto at finworks.biz>> wrote:
>>>>> 
>>>>> Okay, I’ll give that a try and see what I can get when using the Zinc adaptor in GemStone…
>>>> 
>>>> 
>>>> It took me a bit longer to actually get started since I wanted to debug this on my Mac using a GsDevKit_stones setup and I had to dig into another rabbit hole first :-)
>>>> Anyway, I setup the simple handler below and I can confirm also notice that a file of 60MB takes 40s to be served and 28MB takes 10s. 
>>>> 
>>>> In Pharo, the 60MB file takes 3s.
>>>> I’ll dive into this a bit more tomorrow…..
>>>> 
>>>> get
>>>> 
>>>> <get>
>>>> <path: '/'>
>>>> <produces: 'application/json'>
>>>> 
>>>> | file |
>>>> file := GsFile openReadOnServer: '/Users/johanbrichau/testfile'.
>>>> ^ file contents
>>>> 
>>> 
>> 
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>> https://lists.gemtalksystems.com/mailman/listinfo/glass

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/archives/glass/attachments/20240526/aa428186/attachment.htm>