<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 17, 2021 at 10:40 AM David Shaffer via GemStone-Smalltalk <<a href="mailto:gemstone-smalltalk@lists.gemtalksystems.com">gemstone-smalltalk@lists.gemtalksystems.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Following-up in case future GemStoners go this direction. The disconnection problem is inherent to docker swarms virtual IP system used in its routing meshes. The assumption is that long-running connections result from bugs or DoS attempts so they are cleared periodically (every 2 hours by default…seems silly…what DoS would be mitigated by something that takes 2 hrs to detect a problem?). To solve the problem launch GemStone in such a way that it isn’t part of the swarm routing mesh (the option is “dnsrr” stands for DNS round robin but I’m not suggesting taking advantage of this…only one GemStone server = 1 DNS entry). If you still want to publish your GemStone port so that it can be accessed outside the swarm, you must use “host” mode (the port is published on only the host that is running GemStone, rather than all hosts in the swarm). I’ll paste an example docker compose/swarm YAML file after my sig. I’ve been running for 12 hours now without problems.</div></blockquote><div><br></div><div>I'm glad you resolved this, but especially pleased with your sharing the solution!</div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>David</div><div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">services:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> gemstone:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> image: <your-gemstone-image></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> ports:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> - target: 40055</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> published: <port that non-swarm participants should use></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> mode: host # published port only available on /this/ host</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> deploy:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> replicas: 1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> endpoint_mode: dnsrr # use DNS Round Robin to avoid VIP disconnects every 2 hours</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> volumes:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> - type: volume</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> source: gemstone-data</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> target: /gemstone-data</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> - type: volume</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> source: gemstone-backup</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> target: /gemstone-backup</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> - type: volume</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> source: gemstone-log</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> target: /gemstone-log</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> shm_size: '1gb'</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><br><blockquote type="cite"><div>On Nov 14, 2021, at 7:24 PM, David Shaffer <<a href="mailto:shaffer@SHAFFER-CONSULTING.COM" target="_blank">shaffer@SHAFFER-CONSULTING.COM</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;">I just tried updating one ivar of an object every commit (a timestamp stored in one of my “root” objects) so now there should be some traffic with every commit (wouldn’t there be traffic even if my commits didn’t have data to push? At the very least VW would need to sync with the gem to get the list of new modified objects?). Anyway, no dice, still dies every 2 hours.<div><br></div><div>I am knee deep in google hits right now…I’ll post back if anything pans out.<br><div><br></div><div>-D<br><div><br><blockquote type="cite"><div>On Nov 14, 2021, at 1:53 PM, James Foster <<a href="mailto:smalltalk@jgfoster.net" target="_blank">smalltalk@jgfoster.net</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;">David,<div><br></div><div>The error “socket read EOF” indicates that the Gem attempted to read from a socket and received an EOF response. <div><br></div><div>Given that the client and the server are in different Docker containers, they are effectively on separate hosts and there is a strong indication that the socket closed between them. Given the timing duration and consistency, my first guess is that a socket is being closed due to inactivity. If you added a commit every minute (say, the last time through the loop), would that change the behavior?</div><div><br></div><div>James<br><div><br><blockquote type="cite"><div>On Nov 14, 2021, at 10:40 AM, David Shaffer <<a href="mailto:shaffer@shaffer-consulting.com" target="_blank">shaffer@shaffer-consulting.com</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;">The host is an AWS EC2 instance running Ubuntu (20.04.2) running on AWS (kernel 5.11.0-1020-aws) with Docker (version 20.10.7, build 20.10.7-0ubuntu5~20.04.2). Gemstone and the VisualWorks client are running in separate Ubuntu containers (“latest” on Dockerhub which is 20.04/focal). Docker is running in “swarm mode” on this host and both client and server are swarm services.<div><br></div><div>The sleep is 1 second and there is not always work to do. In fact, most loop iterations complete without committing any changes.</div><div><div><br></div><div>I’m currently pursuing some sketchy syslog entries on the EC2 host that seem to correspond to the network errors. I’ll share them as soon as I’ve pruned them down a bit. This same setup ran for 6 years <span>(this EC2 instance for 2 years, my transition to swarm mode was about 1 year ago) </span>with GOODS as backend without network-related errors, though.</div><div><br></div><div>-D<br><div><br><blockquote type="cite"><div>On Nov 14, 2021, at 1:29 PM, James Foster <<a href="mailto:smalltalk@jgfoster.net" target="_blank">smalltalk@jgfoster.net</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;">David,<div><br></div><div>Tell us a bit more about your configuration. Are you running Windows, macOS, or Linux? Is the client inside the Docker container (you mentioned the “entire system”)? How long is the sleep? Is there always work to do? </div><div><br></div><div>James</div><div><br><div><br><blockquote type="cite"><div>On Nov 14, 2021, at 10:22 AM, David Shaffer via GemStone-Smalltalk <<a href="mailto:gemstone-smalltalk@lists.gemtalksystems.com" target="_blank">gemstone-smalltalk@lists.gemtalksystems.com</a>> wrote:</div><br><div><div style="overflow-wrap: break-word;"><div>Hey folks:</div><div><br></div><div>I’ve (finally) deployed a server using GemStone 3.6.2, GemBuilder 8.5 and VW 9.0. My server’s main loop is essentially:</div><div><br></div><div>Abort</div><div>Do work</div><div>Commit</div><div>Sleep</div><div><br></div><div>Every two hours (I’m not sure it is exactly two hours but it seems pretty reliable), I get the following during the abort call:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px">GS Server Error - GbsGsErrStnNetLost - The session has lost its connection to the Stone Repository monitor.</blockquote><div><br></div><div><div>The entire system runs on a single host in Docker so it can’t possibly be a network hiccup. The gemnetobject logs are pasted below…they make it seem like a network error but, again, this is highly unlikely. Has anyone else run into this or have any troubleshooting advice?</div></div><div><br></div><div>-David</div><div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">--- 11/13/21 22:56:18.959 UTC Login</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: Gave this process preference for OOM killer: wrote to /proc/460/oom_score_adj value 250</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: User ID: Trader</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: Repository: gs64stone</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: Session ID: 5 login at 11/13/21 22:56:18.964 UTC</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: GCI Client Host: </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: Page server PID: -1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">[Info]: using libicu version 58.2</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------------------</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">GemStone: Error Fatal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Network error - text follows:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">, socket read EOF</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Error Category: 231169 [GemStone] Number: 4137 Arg Count: 1 Context : 20 exception : 20</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Arg 1: 20</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">--- 11/13/21 23:03:16.322 UTC Logging out</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">*****************************************************</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">****** Abnormal Shutdown at 11/13/21 23:03:16.824 UTC</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">*****************************************************</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------------------</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">GemStone: Error Fatal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Network error - text follows:</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">, socket read EOF</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Error Category: 231169 [GemStone] Number: 4137 Arg Count: 1 Context : 20 exception : 20</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(124,105,56);background-color:rgba(255,170,161,0.85)"><span style="font-variant-ligatures:no-common-ligatures">Arg 1: 20</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></div></div>_______________________________________________<br>GemStone-Smalltalk mailing list<br><a href="mailto:GemStone-Smalltalk@lists.gemtalksystems.com" target="_blank">GemStone-Smalltalk@lists.gemtalksystems.com</a><br><a href="https://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk" target="_blank">https://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk</a><br></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div>_______________________________________________<br> GemStone-Smalltalk mailing list<br> <a href="mailto:GemStone-Smalltalk@lists.gemtalksystems.com" target="_blank">GemStone-Smalltalk@lists.gemtalksystems.com</a><br> <a href="https://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk" rel="noreferrer" target="_blank">https://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk</a><br> </blockquote></div></div>