[Glass] Gemstones bidirectional communications

Mon Jan 7 14:20:20 PST 2019

Hi Dario,

Look for a framework that will serialize/marshal objects for replication
between object spaces.

A primitive but common form would be to read and write (archive) to a
relational database. You would define table mapping rules for objects with
relatively simple and predictable graph structure. Maintenance costs
increase and performance drops because you are mapping between an efficient
object database and indexed query tables. An rdb can manage a lot of data
though and is good for some types of queries.

You could also marshal data serialized in a format like xml or your own
"direct encoded" data format. A standard like xml is human readable and
data can be modified if necessary to fit new class definitions. XML is
verbose and slow. A direct stream encoding is faster than xml but you may
need a strategy for ensuring object spaces recognize the encoding. You
could do that by encoding metastate with (or versions referenced by) the
object state or you could do something to ensure code is identical between
object spaces. Direct is common and good when encoding is for transport and
not storage. XML favors storage and data correction.

GS includes a binary serialzation tool that might possibly be an
alternative to the direct encoding. Mappings and schema must be equal,
obviously. Performance would be excellent. Sorry, I never used that for
application object state. Never saw a company use it for application code.

A mix of all the above is a framework that I wrote called State Replication
Protocol (SRP). SRP marshals complex object graphs in a unique binary
encoding that is highly portable. Metastate is encoded or referenced,
allowing complex data graphs to be read and mapped even if class
definitions are absent. SRP can be thought of as an encoding that is more
efficient than XML and a framework that already does what you'd need to
write for an XML framework.

James mentioned GBS. GBS is marshaling plus transparency...for a few
dialects. Transparency means objects feel like you are working with them in
GS itself and GBS handles the complexities of keeping everything in sync
through modifications in either objectspace. It is of course a very complex
product with many challenges for good performance. I had managed GBS code
when I worked there. A good tool, but not intended for transparency between
two GS databases.

No easy solutions. Consider just using a large database and a backup
strategy. Replay tranlogs in a copy of the database if you need quick
failover.

Paul Baumann

On Mon, Jan 7, 2019, 3:37 PM James Foster via Glass <
glass at lists.gemtalksystems.com wrote:

> HI Dario,
>
> On Jan 7, 2019, at 9:01 AM, Trussardi Dario Romano via Glass <
> glass at lists.gemtalksystems.com> wrote:
>
> Ciao James,
>
> thanks.
>
> Hi Dario,
>
> There are several ways of doing this along a continuum of how much both
> sides know that GemStone is at the other end. At one extreme, you could
> have each system export and import “flat” files (XML, JSON, STON,  CSV,
> tab-delimited, etc.). This is how we had disparate systems talk to each
> other in the 1980s. It is simple, reliable, and pretty easy to debug.
>
> Next, you could define a “wire protocol” and have the transfer take place
> over a socket. The protocol could be similar to the file format with the
> exchange happening more quickly, but it does require more “handshaking” and
> effort at synchronization (with file transfer the other application doesn’t
> have to be up for you to put a file in a shared directory).
>
> And at the more GemStone-focused end you could use GsExternalSession to
> have a Gem on one system start a session on the other system. With this you
> can execute strings and get back strings (or more complex objects, but that
> would be a lot of work).
>
>
> What do you mean for:  but that would be a lot of work.
>
> Can you give some consideration about it?
>
>
> It is complicated to manage relationships between two object spaces when
> only “primitive” types (SmallIntegers, Booleans, and Characters) can be
> copied between the two spaces. To transfer anything more complex you need
> to convert the object to a set of primitive types, send it over the wire,
> and recreate it on the other side. For an object made up of simple types
> (numbers, dates, strings), it isn’t too difficult to create a copy of the
> object. But for objects that point to other objects, you face the question
> of how deep to go in the copy. Also you face the question of how do you
> manage modifications on one side or the other. Do they get automatically
> transmitted across or do you get just a one-time copy that remains static.
>
> Is there any example code in this regard?
>
>
> Yes, see GBS (https://gemtalksystems.com/products/gbs-vw/). The main
> lesson to learn is that you don’t want to go there!
>
> From which version is this support active?
>
> I have a Gemstone 3.1.0.6 system but it does not define the class
> GsExternalSession.
>
> ( The Gemstone 3.3.6 define it )
>
>
> Sorry, I don’t have that information memorized. You could search through
> the release notes or do a binary search between the two versions.
>
> The two systems must be of the same version?
>
>
> I believe that, for all practical purposes, it is necessary.
>
> James
>
>
>
> The main thing to keep in mind with any of these approaches is that you
> are managing the relationship, not GemStone. There is no “transparent
> replication," object identity is not preserved, and relying on an OOP could
> be a problem if an object gets garbage-collected and the OOP reused.
>
>
>
>
> If one system is collecting new data and dumping it into another system,
> then this model can work pretty well, but you should think about
> long-running transactions (commit record backlog) for any idle sessions.
>
>
> It is not clear to me what you mean.
>
> Can you explain this last point better?
>
>
> See the videos at
> https://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/,
> especially 12-15.
>
>
> Thanks,
>
> Dario
>
> James
>
> On Jan 4, 2019, at 9:06 AM, Trussardi Dario Romano via Glass <
> glass at lists.gemtalksystems.com> wrote:
>
> Ciao,
>
> to manage the problematic application on which i am working
>
> i would need to have two distinct environments that communicate with each
> other.
>
> Where for environment i mean a Gemstone database.
>
>
> The first one should handle local management (i call it local-database)
> and is active on a local server (where the event happens).
>
>
> The second (i call it the internet-database) should manage support for
> users' smartphones and send summary data to the local-database.
>
> Internet-database is active on a server in remote datacenter.
>
> In this way users (hundreds) with their devices go to load the
> Internet-database not interfering with local-database management.
>
>
>
> Now my question is about how i can handle bidirectional communication
> between the two Gemstone database servers (running a VPN)
>
>
> Thanks for considerations,
>
> Dario
>
> P.S. Have I been sufficiently clear?
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
> _______________________________________________
> Glass mailing list
> Glass at lists.gemtalksystems.com
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>

On Jan 7, 2019 3:37 PM, "James Foster via Glass" <
glass at lists.gemtalksystems.com> wrote:

HI Dario,

On Jan 7, 2019, at 9:01 AM, Trussardi Dario Romano via Glass <
glass at lists.gemtalksystems.com> wrote:

Ciao James,

thanks.

Hi Dario,

There are several ways of doing this along a continuum of how much both
sides know that GemStone is at the other end. At one extreme, you could
have each system export and import “flat” files (XML, JSON, STON,  CSV,
tab-delimited, etc.). This is how we had disparate systems talk to each
other in the 1980s. It is simple, reliable, and pretty easy to debug.

Next, you could define a “wire protocol” and have the transfer take place
over a socket. The protocol could be similar to the file format with the
exchange happening more quickly, but it does require more “handshaking” and
effort at synchronization (with file transfer the other application doesn’t
have to be up for you to put a file in a shared directory).

And at the more GemStone-focused end you could use GsExternalSession to
have a Gem on one system start a session on the other system. With this you
can execute strings and get back strings (or more complex objects, but that
would be a lot of work).

What do you mean for:  but that would be a lot of work.

Can you give some consideration about it?

It is complicated to manage relationships between two object spaces when
only “primitive” types (SmallIntegers, Booleans, and Characters) can be
copied between the two spaces. To transfer anything more complex you need
to convert the object to a set of primitive types, send it over the wire,
and recreate it on the other side. For an object made up of simple types
(numbers, dates, strings), it isn’t too difficult to create a copy of the
object. But for objects that point to other objects, you face the question
of how deep to go in the copy. Also you face the question of how do you
manage modifications on one side or the other. Do they get automatically
transmitted across or do you get just a one-time copy that remains static.

Is there any example code in this regard?

Yes, see GBS (https://gemtalksystems.com/products/gbs-vw/). The main lesson
to learn is that you don’t want to go there!

>From which version is this support active?

I have a Gemstone 3.1.0.6 system but it does not define the class
GsExternalSession.

( The Gemstone 3.3.6 define it )

Sorry, I don’t have that information memorized. You could search through
the release notes or do a binary search between the two versions.

The two systems must be of the same version?

I believe that, for all practical purposes, it is necessary.

James

The main thing to keep in mind with any of these approaches is that you are
managing the relationship, not GemStone. There is no “transparent
replication," object identity is not preserved, and relying on an OOP could
be a problem if an object gets garbage-collected and the OOP reused.

If one system is collecting new data and dumping it into another system,
then this model can work pretty well, but you should think about
long-running transactions (commit record backlog) for any idle sessions.

It is not clear to me what you mean.

Can you explain this last point better?

See the videos at
https://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/,
especially 12-15.

Thanks,

Dario

James

On Jan 4, 2019, at 9:06 AM, Trussardi Dario Romano via Glass <
glass at lists.gemtalksystems.com> wrote:

Ciao,

to manage the problematic application on which i am working

i would need to have two distinct environments that communicate with each
other.

Where for environment i mean a Gemstone database.

The first one should handle local management (i call it local-database) and
is active on a local server (where the event happens).

The second (i call it the internet-database) should manage support for
users' smartphones and send summary data to the local-database.

Internet-database is active on a server in remote datacenter.

In this way users (hundreds) with their devices go to load the
Internet-database not interfering with local-database management.

Now my question is about how i can handle bidirectional communication
between the two Gemstone database servers (running a VPN)

Thanks for considerations,

Dario

P.S. Have I been sufficiently clear?
_______________________________________________
Glass mailing list
Glass at lists.gemtalksystems.com
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
Glass at lists.gemtalksystems.com
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
Glass at lists.gemtalksystems.com
http://lists.gemtalksystems.com/mailman/listinfo/glass
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20190107/e1cb2bc8/attachment-0001.html>