<div dir="ltr"><div>See below.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 20, 2024 at 7:18 AM Otto Behrens via Glass <<a href="mailto:glass@lists.gemtalksystems.com">glass@lists.gemtalksystems.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>Ralph, thanks a lot for your response. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>70MB is quite large, but should work fine if the relevant settings
are adjusted (increased GEM memory, allowed nginx response size).
I agree, that 26 seconds for 16MB is a very long time.</p></div></blockquote><div><br></div><div>Yes, we ran out of temporary object space and increased that. We also changed the Rest API code a bit to avoid unnecessary buffer copying. It is not streaming properly yet, but survives with the memory that we have allocated for now.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>On an anstract level, these are my thoughts:<br>
<br>
* Depending on your API it might be an good idea to think about
pagination and split the data into some more api requests with
smaller responses.<br></p></div></blockquote><div>Ok, that may help. How big should a page be? Is 10MB too much?</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
* I would configure nginx to do the response compression (gzip) to
the client.<br></p></div></blockquote><div>We did that, but it made no significant difference. The problem is on the server side.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>* I would repeat your local curl test with a connect to your
Zinc-server port just to take nginx out of the equation.<br></p></div></blockquote><div>That is a great idea. Will do that. We are using client certificates, but for a test I can hard code the fingerprint.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
* If nginx is not causing the trouble, I would try to use
GemStone's ProfMonitor to get some more insights about the run
time behaviour of your smalltalk code.<br></p></div></blockquote><div>The smalltalk code is reasonably expensive, but my tests (using kill -USR1) revealed that it was mostly spending time in SocketStream >> nextPutAll: </div></div></div></blockquote><div><br></div><div>I know there were changes in recent versions of GemStone to provide better stream support for the Seaside use cases. I don't recall the details, but there is at least an AppendStream optimized for string building with primitives for some methods, I think. I just checked: "recent" is actually version 3.4!</div><div><br></div><div>Do you see that class in the profile stacks or some other Stream class?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
<br>
Best regards,<br>
Ralph</p>
<p><br>
</p>
<div>Am 20.05.2024 um 09:48 schrieb Otto
Behrens via Glass:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>We have not managed to fix this yet. What is your opinion
on the following ideas?</div>
<div><br>
</div>
<div>1. Otto, you are an idiot. Why would you be sending a 70MB
json response out on a REST API? This is not how you do an
API. [Otto: that may well be true. How should I then get the
data across to the user? Is there someone that can help me
with a solution?]</div>
<div>2. Otto, you have not kept up to date with things and you
are the only one in the whole world that's using
WAGsZincAdaptor serving as an nginx upstream. WTF. [Otto: Yes,
we are here on the bottom tip of Africa where the internet is
slow and we read even slower, sorry about that. Please help me
with some sites, documents and any other material so that I
can start reading.]</div>
<div>3. Otto, have you heard of the idea of compression? You
should know that JSON will compress to at least a 10th of the
original size because it is text with a lot of repetition.
[Otto: yes, I downloaded a zip file once and could not read it
in vim. Is this what I should do: compress the connection
between nginx and the Zinc adaptor? Or should I send the json
back as a compressed zip file?]</div>
<div>4. Otto, you should get to know nginx and its settings and
understand all the stuff nginx spits out when debug logging is
on. Better still, download the C source code; you should still
be able to after only Smalltalking for 20 years. [Otto: Are
you super human? Have you seen all of that? Please enlighten
me as this will take me years.]</div>
<div><br>
</div>
Of course I missed some ideas. Please feel free to add them to
the list.
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td width="400" valign="bottom">
<p style="margin:0px;padding:0px"><span style="font-size:18px;color:rgb(146,148,151);font-family:Calibri,sans-serif;font-weight:700">Otto
Behrens</span><br>
</p>
<p style="font-size:18px;font-weight:700;color:rgb(146,148,151);font-family:Calibri,sans-serif;margin:0px;padding:0px"><span style="font-size:14px;font-weight:300;margin:0px;padding:0px">+27 82 809
2375</span></p>
</td>
<td width="200" valign="middle"><img src="https://www.finworks.biz/signature/finworks-signature-logo.png" alt="FINWorks" style="display: block; border: 0px; width: 200px; height: 38px; margin: 0px; padding: 0px;" width="200" height="38"></td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="5"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium;border-bottom:1px solid rgb(200,28,36)" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="15"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="20"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="display:inline-block" width="15" valign="top"><a href="http://za.linkedin.com/in/waltherbehrens" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.finworks.biz/signature/finworks-linkedin-logo.png" alt="FINWorks" style="display: inline-block; border: 0px; width: 15px; height: 15px; margin-top: 1.5px; padding: 0px;" width="15" height="15"></a></td>
<td style="display:inline-block" width="250" valign="top"><a href="http://www.finworks.biz/" style="color:rgb(200,28,36);font-family:Calibri,sans-serif;margin-left:10px;margin-top:0px;padding-top:0px;font-size:11pt;display:inline-block" target="_blank">www.finworks.biz</a></td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="10"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td>
<p style="font-size:10px;color:rgb(146,148,151);font-family:Calibri,sans-serif;text-align:justify">Disclaimer
& Confidentiality Note: This email is
intended solely for the use of the
individual or entity named above as it may
contain information that is confidential
and privileged. If you are not the
intended recipient, be advised that any
dissemination, distribution or copying of
this email is strictly prohibited.
FINWorks cannot be held liable by any
person other than the addressee in respect
of any opinions, conclusions, advice or
other information contained in this email.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May 17, 2024 at
7:09 AM Otto Behrens <<a href="mailto:otto@finworks.biz" target="_blank">otto@finworks.biz</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>We are running into a performance problem where our API
returns about 70MB json content. We run a nginx web server
which connects to a WAGsZincAdaptor that we start in a
topaz session. Do you perhaps have the same kind of setup
and can you please give me some advice on this?</div>
<div><br>
</div>
<div>We found that converting objects to json (using Object
>> asJson from Seaside-JSON-Core) was not
performing great, but was eating loads of memory because
of WABuilder >> render:. This is not the issue and
we improved this a bit (by eliminating String
streamContents: and streaming more directly).</div>
<div><br>
</div>
<div>The problem seems to be that after producing the json
content, transmitting the response takes a long time.</div>
<div><br>
</div>
<div>As an experiment, I read a 16MB file from disk and
returned that as the result of an API call to eliminate
all json producing code. I used curl as a client on the
same machine as the nginx server, stone and the topaz
session and it takes 26 seconds. This eliminates most
overhead (no network latency).</div>
<div><br>
</div>
<div>The stack below is what I see most of the time:</div>
<div><br>
</div>
<div>1 SocketStream >> nextPutAll: @natCode+0x4d
[GsNMethod 169113089]<br>
FP: 0x7f2c0fee9930=StackLimit[-218] ,
callerFP: StackLimit[-212]<br>
arg 1:0x7f2bee7f0de0 (cls:103425 ByteArray size:16384)<br>
rcvr: 0x7f2bff68f670 (cls:144280577 SocketStream
size:12)<br>
2 ZnBivalentWriteStream >> next:putAll:startingAt:
@natCode+0x2cf [GsNMethod 158727169]<br>
FP: 0x7f2c0fee9960=StackLimit[-212] ,
callerFP: StackLimit[-202]<br>
arg 3:69337098 (SmallInteger 8667137)<br>
arg 2:0x7f2c0064fe50 (cls:74753 String
size:16627226)'(large_or_fwd (size 16627226))'<br>
arg 1:131074 (SmallInteger 16384)<br>
rcvr: 0x7f2c00675370 (cls:145086465
ZnBivalentWriteStream size:2)<br>
3 ZnUtils class >> nextPutAll:on: @natCode+0x421
[GsNMethod 175369473]<br>
FP: 0x7f2c0fee99b0=StackLimit[-202] ,
callerFP: StackLimit[-196]<br>
arg 2:0x7f2c00675370 (cls:145086465
ZnBivalentWriteStream size:2)<br>
arg 1:0x7f2c0064fe50 (cls:74753 String
size:16627226)'(large_or_fwd (size 16627226))'<br>
rcvr: 0x7f2c0c335750 oid:143053313 (cls:143054593
ZnUtils class size:19)<br>
4 ZnByteArrayEntity >> writeOn: @natCode+0xdb
[GsNMethod 269993473]<br>
FP: 0x7f2c0fee99e0=StackLimit[-196] ,
callerFP: StackLimit[-186]<br>
arg 1:0x7f2c00675370 (cls:145086465
ZnBivalentWriteStream size:2)<br>
rcvr: 0x7f2c00651f00 (cls:145996545 ZnByteArrayEntity
size:3)<br>
5 ZnEntityWriter >> writeEntity: @natCode+0x382
[GsNMethod 269988609]<br>
FP: 0x7f2c0fee9a30=StackLimit[-186] ,
callerFP: StackLimit[-180]<br>
arg 1:0x7f2c00651f00 (cls:145996545 ZnByteArrayEntity
size:3)<br>
rcvr: 0x7f2c00675398 (cls:145876737 ZnEntityWriter
size:2)<br>
6 ZnMessage >> writeOn: @natCode+0x295 [GsNMethod
158696193]<br>
FP: 0x7f2c0fee9a60=StackLimit[-180] ,
callerFP: StackLimit[-174]<br>
arg 1:0x7f2c00675370 (cls:145086465
ZnBivalentWriteStream size:2)<br>
rcvr: 0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)<br>
7 ZnResponse >> writeOn: @natCode+0x1f0
[GsNMethod 155024025857]<br>
FP: 0x7f2c0fee9a90=StackLimit[-174] ,
callerFP: StackLimit[-169]<br>
arg 1:0x7f2bff68f670 (cls:144280577 SocketStream
size:12)<br>
rcvr: 0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)<br>
8 ZnSingleThreadedServer >> writeResponse:on:
@natCode+0xa3 [GsNMethod 169204737]<br>
FP: 0x7f2c0fee9ab8=StackLimit[-169] ,
callerFP: StackLimit[-162]<br>
arg 2:0x7f2bff68f670 (cls:144280577 SocketStream
size:12)<br>
arg 1:0x7f2c0064fe20 (cls:145901313 ZnResponse size:3)<br>
rcvr: 0x7f2bff5de528 oid:4763064833 (cls:144532225
ZnManagingMultiThreadedServer size:9)<br>
</div>
<div><br>
</div>
<div>Kind regards<br clear="all">
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td width="400" valign="bottom">
<p style="margin:0px;padding:0px"><span style="font-size:18px;color:rgb(146,148,151);font-family:Calibri,sans-serif;font-weight:700">Otto
Behrens</span><br>
</p>
<p style="font-size:18px;font-weight:700;color:rgb(146,148,151);font-family:Calibri,sans-serif;margin:0px;padding:0px"><span style="font-size:14px;font-weight:300;margin:0px;padding:0px">+27 82 809
2375</span></p>
</td>
<td width="200" valign="middle"><img src="https://www.finworks.biz/signature/finworks-signature-logo.png" alt="FINWorks" style="display: block; border: 0px; width: 200px; height: 38px; margin: 0px; padding: 0px;" width="200" height="38"></td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="5"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium;border-bottom:1px solid rgb(200,28,36)" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="15"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="20"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="display:inline-block" width="15" valign="top"><a href="http://za.linkedin.com/in/waltherbehrens" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.finworks.biz/signature/finworks-linkedin-logo.png" alt="FINWorks" style="display: inline-block; border: 0px; width: 15px; height: 15px; margin-top: 1.5px; padding: 0px;" width="15" height="15"></a></td>
<td style="display:inline-block" width="250" valign="top"><a href="http://www.finworks.biz/" style="color:rgb(200,28,36);font-family:Calibri,sans-serif;margin-left:10px;margin-top:0px;padding-top:0px;font-size:11pt;display:inline-block" target="_blank">www.finworks.biz</a></td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td height="10"><br>
</td>
</tr>
</tbody>
</table>
<table style="color:rgb(0,0,0);font-family:Times;font-size:medium" width="600" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td>
<p style="font-size:10px;color:rgb(146,148,151);font-family:Calibri,sans-serif;text-align:justify">Disclaimer
& Confidentiality Note: This email
is intended solely for the use of the
individual or entity named above as it
may contain information that is
confidential and privileged. If you
are not the intended recipient, be
advised that any dissemination,
distribution or copying of this email
is strictly prohibited. FINWorks
cannot be held liable by any person
other than the addressee in respect of
any opinions, conclusions, advice or
other information contained in this
email.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Glass mailing list
<a href="mailto:Glass@lists.gemtalksystems.com" target="_blank">Glass@lists.gemtalksystems.com</a>
<a href="https://lists.gemtalksystems.com/mailman/listinfo/glass" target="_blank">https://lists.gemtalksystems.com/mailman/listinfo/glass</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Glass mailing list<br>
<a href="mailto:Glass@lists.gemtalksystems.com" target="_blank">Glass@lists.gemtalksystems.com</a><br>
<a href="https://lists.gemtalksystems.com/mailman/listinfo/glass" rel="noreferrer" target="_blank">https://lists.gemtalksystems.com/mailman/listinfo/glass</a><br>
</blockquote></div></div>
_______________________________________________<br>
Glass mailing list<br>
<a href="mailto:Glass@lists.gemtalksystems.com" target="_blank">Glass@lists.gemtalksystems.com</a><br>
<a href="https://lists.gemtalksystems.com/mailman/listinfo/glass" rel="noreferrer" target="_blank">https://lists.gemtalksystems.com/mailman/listinfo/glass</a><br>
</blockquote></div></div>