[Glass] rest api returning large result

Mon May 20 00:48:57 PDT 2024

We have not managed to fix this yet. What is your opinion on the following
ideas?

1. Otto, you are an idiot. Why would you be sending a 70MB json response
out on a REST API? This is not how you do an API. [Otto: that may well be
true. How should I then get the data across to the user? Is there someone
that can help me with a solution?]
2. Otto, you have not kept up to date with things and you are the only one
in the whole world that's using WAGsZincAdaptor serving as an nginx
upstream. WTF. [Otto: Yes, we are here on the bottom tip of Africa where
the internet is slow and we read even slower, sorry about that. Please help
me with some sites, documents and any other material so that I can start
reading.]
3. Otto, have you heard of the idea of compression? You should know that
JSON will compress to at least a 10th of the original size because it is
text with a lot of repetition. [Otto: yes, I downloaded a zip file once and
could not read it in vim. Is this what I should do: compress the connection
between nginx and the Zinc adaptor? Or should I send the json back as a
compressed zip file?]
4. Otto, you should get to know nginx and its settings and understand all
the stuff nginx spits out when debug logging is on. Better still, download
the C source code; you should still be able to after only Smalltalking for
20 years. [Otto: Are you super human? Have you seen all of that? Please
enlighten me as this will take me years.]

Of course I missed some ideas. Please feel free to add them to the list.

Otto Behrens

+27 82 809 2375
[image: FINWorks]
[image: FINWorks] <http://za.linkedin.com/in/waltherbehrens>
www.finworks.biz

Disclaimer & Confidentiality Note: This email is intended solely for the
use of the individual or entity named above as it may contain information
that is confidential and privileged. If you are not the intended recipient,
be advised that any dissemination, distribution or copying of this email is
strictly prohibited. FINWorks cannot be held liable by any person other
than the addressee in respect of any opinions, conclusions, advice or other
information contained in this email.

On Fri, May 17, 2024 at 7:09 AM Otto Behrens <otto at finworks.biz> wrote:

> Hi,
>
> We are running into a performance problem where our API returns about 70MB
> json content. We run a nginx web server which connects to a WAGsZincAdaptor
> that we start in a topaz session. Do you perhaps have the same kind of
> setup and can you please give me some advice on this?
>
> We found that converting objects to json (using Object >> asJson from
> Seaside-JSON-Core) was not performing great, but was eating loads of memory
> because of WABuilder >> render:. This is not the issue and we improved this
> a bit (by eliminating String streamContents: and streaming more directly).
>
> The problem seems to be that after producing the json content,
> transmitting the response takes a long time.
>
> As an experiment, I read a 16MB file from disk and returned that as the
> result of an API call to eliminate all json producing code. I used curl as
> a client on the same machine as the nginx server, stone and the topaz
> session and it takes 26 seconds. This eliminates most overhead (no network
> latency).
>
> The stack below is what I see most of the time:
>
> 1  SocketStream >> nextPutAll:  @natCode+0x4d  [GsNMethod 169113089]
>               FP: 0x7f2c0fee9930=StackLimit[-218] , callerFP:
> StackLimit[-212]
>   arg 1:0x7f2bee7f0de0 (cls:103425 ByteArray size:16384)
>   rcvr: 0x7f2bff68f670 (cls:144280577 SocketStream size:12)
> 2  ZnBivalentWriteStream >> next:putAll:startingAt:  @natCode+0x2cf
>  [GsNMethod 158727169]
>               FP: 0x7f2c0fee9960=StackLimit[-212] , callerFP:
> StackLimit[-202]
>   arg 3:69337098 (SmallInteger 8667137)
>   arg 2:0x7f2c0064fe50 (cls:74753 String size:16627226)'(large_or_fwd
> (size 16627226))'
>   arg 1:131074 (SmallInteger 16384)
>   rcvr: 0x7f2c00675370 (cls:145086465 ZnBivalentWriteStream size:2)
> 3  ZnUtils class >> nextPutAll:on:  @natCode+0x421  [GsNMethod 175369473]
>               FP: 0x7f2c0fee99b0=StackLimit[-202] , callerFP:
> StackLimit[-196]
>   arg 2:0x7f2c00675370 (cls:145086465 ZnBivalentWriteStream size:2)
>   arg 1:0x7f2c0064fe50 (cls:74753 String size:16627226)'(large_or_fwd
> (size 16627226))'
>   rcvr: 0x7f2c0c335750 oid:143053313 (cls:143054593 ZnUtils class size:19)
> 4  ZnByteArrayEntity >> writeOn:  @natCode+0xdb  [GsNMethod 269993473]
>               FP: 0x7f2c0fee99e0=StackLimit[-196] , callerFP:
> StackLimit[-186]
>   arg 1:0x7f2c00675370 (cls:145086465 ZnBivalentWriteStream size:2)
>   rcvr: 0x7f2c00651f00 (cls:145996545 ZnByteArrayEntity size:3)
> 5  ZnEntityWriter >> writeEntity:  @natCode+0x382  [GsNMethod 269988609]
>               FP: 0x7f2c0fee9a30=StackLimit[-186] , callerFP:
> StackLimit[-180]
>   arg 1:0x7f2c00651f00 (cls:145996545 ZnByteArrayEntity size:3)
>   rcvr: 0x7f2c00675398 (cls:145876737 ZnEntityWriter size:2)
> 6  ZnMessage >> writeOn:  @natCode+0x295  [GsNMethod 158696193]
>               FP: 0x7f2c0fee9a60=StackLimit[-180] , callerFP:
> StackLimit[-174]
>   arg 1:0x7f2c00675370 (cls:145086465 ZnBivalentWriteStream size:2)
>   rcvr: 0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)
> 7  ZnResponse >> writeOn:  @natCode+0x1f0  [GsNMethod 155024025857]
>               FP: 0x7f2c0fee9a90=StackLimit[-174] , callerFP:
> StackLimit[-169]
>   arg 1:0x7f2bff68f670 (cls:144280577 SocketStream size:12)
>   rcvr: 0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)
> 8  ZnSingleThreadedServer >> writeResponse:on:  @natCode+0xa3  [GsNMethod
> 169204737]
>               FP: 0x7f2c0fee9ab8=StackLimit[-169] , callerFP:
> StackLimit[-162]
>   arg 2:0x7f2bff68f670 (cls:144280577 SocketStream size:12)
>   arg 1:0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)
>   rcvr: 0x7f2bff5de528 oid:4763064833 (cls:144532225
> ZnManagingMultiThreadedServer size:9)
>
> Kind regards
>
> Otto Behrens
>
> +27 82 809 2375
> [image: FINWorks]
> [image: FINWorks] <http://za.linkedin.com/in/waltherbehrens>
> www.finworks.biz
>
> Disclaimer & Confidentiality Note: This email is intended solely for the
> use of the individual or entity named above as it may contain information
> that is confidential and privileged. If you are not the intended recipient,
> be advised that any dissemination, distribution or copying of this email is
> strictly prohibited. FINWorks cannot be held liable by any person other
> than the addressee in respect of any opinions, conclusions, advice or other
> information contained in this email.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/archives/glass/attachments/20240520/ba340785/attachment.htm>