[Glass] startStone after reboot

Bill Erickson via Glass glass at lists.gemtalksystems.com
Tue Feb 24 10:23:08 PST 2015


It's the extra space allocated for stats and tables as I mentioned.



------------------------------------------------------------------------
Bill Erickson
GemTalk Systems Engineering
15220 NW Greenbrier Parkway #240, Beaverton OR 97006
------------------------------------------------------------------------

On Tue, Feb 24, 2015 at 10:20 AM, Mariano Martinez Peck <
marianopeck at gmail.com> wrote:

>
>
> On Tue, Feb 24, 2015 at 2:50 PM, Bill Erickson via Glass <
> glass at lists.gemtalksystems.com> wrote:
>
>> Keep in mind that SHR_PAGE_CACHE_SIZE_KB determines the size for DB
>> pages.  GemStone also adds to it additional space for holding stats and
>> various internal tables, so the actual shared memory region allocated will
>> be a bit larger.
>>
>>
>>
> Ohh I didn't know that. So... if I have a SHR_PAGE_CACHE_SIZE_KB
> of 1000000, that should be 1024000000 bytes. However, "ipcs -m" shows
> me 1080459264.  So..is this a "rounding" difference between KB and Bytes or
> is what you are saying?
>
> Thanks in advance,
>
>
>
>>
>> ------------------------------------------------------------------------
>> Bill Erickson
>> GemTalk Systems Engineering
>> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
>> ------------------------------------------------------------------------
>>
>> On Tue, Feb 24, 2015 at 9:39 AM, Mariano Martinez Peck via Glass <
>> glass at lists.gemtalksystems.com> wrote:
>>
>>>
>>>
>>> On Tue, Feb 24, 2015 at 2:04 PM, Dario Trussardi via Glass <
>>> glass at lists.gemtalksystems.com> wrote:
>>>
>>>>
>>>> Mariano,
>>>>
>>>> Dario,
>>>>
>>>> unix$   sudo vim /etc/sysctl
>>>>
>>>>
>>>> I setup my /etc/sysctl with  this data:
>>>>
>>>>
>>>> #   75% of total RAM. If ARM = 8GB then let's take 6GB.
>>>> kernel.shmmax = 6442450944
>>>>
>>>> #    shmmax / 4096
>>>> kernel.shmall = 1572864
>>>>
>>>>
>>>> And after reboot the system work fine.
>>>>
>>>> Thank.
>>>>
>>>>
>>>>
>>>> Just change those numbers... assign a total of 75% of your total RAM to
>>>> shmmax  and then set shmall accordly.
>>>>
>>>> Of course, that depends on your scenario. If you always use GLASS, then
>>>> you are allowed to 2GB SPC.
>>>>
>>>>
>>>> But for define 2GB SPC what  SHR_PAGE_CACHE_SIZE_KB = xyz
>>>>
>>>> i need to define ?
>>>>
>>>> For now my system.conf  define : SHR_PAGE_CACHE_SIZE_KB = 500000;
>>>>
>>>
>>> That is using only 500MB. Just put:
>>>
>>> SHR_PAGE_CACHE_SIZE_KB=2000000;
>>>
>>>
>>>
>>>> It's not clear for me.
>>>>
>>>>
>>>
>>> I think SHR_PAGE_CACHE_SIZE_KB is pretty clear...it's the size of the
>>> SPC in KB.
>>>
>>> Cheers,
>>>
>>>
>>>> Thanks for any considerations,
>>>>  Dario
>>>>
>>>>
>>>> So...your total needs are:   amount of stones in same machine * SPC (
>>>> whatever SPC you choose ).
>>>>
>>>> Don't forget to reboot before trying to start stone.
>>>>
>>>> Best,
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Feb 24, 2015 at 12:57 PM, Bill Erickson via Glass <
>>>> glass at lists.gemtalksystems.com> wrote:
>>>>
>>>>> Dario,
>>>>>
>>>>> Well, your report says:
>>>>>
>>>>> > dimensione max seg (kbyte) = 32768
>>>>>
>>>>> If you're specifying a shared page cache size > 32MB, then it won't
>>>>> work.  You either need to specify a GemStone shared page cache size < 32MB
>>>>> or reconfigure your linux kernel to allow bigger shared memory regions.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> Bill Erickson
>>>>> GemTalk Systems Engineering
>>>>> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> On Tue, Feb 24, 2015 at 7:46 AM, Dario Trussardi via Glass <
>>>>> glass at lists.gemtalksystems.com> wrote:
>>>>>
>>>>>> Bill,
>>>>>>
>>>>>> Dario,
>>>>>>
>>>>>> If you read the pcmon file carefully you'll see the following entries:
>>>>>>
>>>>>> *****************
>>>>>>
>>>>>> |   GemStone could not retrieve the IPC identifier associated with
>>>>>> the memory |
>>>>>> |   key -989659131.  shmget() error = errno=22,EINVAL, Invalid
>>>>>> argument (programmer
>>>>>> | error).
>>>>>>         |
>>>>>> |
>>>>>>         |
>>>>>>   GemStone could not attach to the shared page cache.
>>>>>>
>>>>>>   [SpcMon trace]: ... cache creation failed ...
>>>>>>   [SpcMon trace]: ... if the errno is (EINVAL) it is likely because
>>>>>> the
>>>>>>                       cache size is less than the operating system
>>>>>> imposed
>>>>>>                       minimum or greater than the operating system
>>>>>> maximum.
>>>>>>
>>>>>> ******************
>>>>>>
>>>>>> As per the comments here, is the setting of the
>>>>>> SHR_PAGE_CACHE_SIZE_KB too large for how the kernel of this machine is
>>>>>> configured for shared memory sizes (kernel.SHMMAX -- see the GS
>>>>>> Installation Guide for details)?
>>>>>>
>>>>>> Also, use the unix command:
>>>>>>
>>>>>> unix> ipcs -m
>>>>>>
>>>>>>
>>>>>> The ipcs -m    command on the server report:    ( anything )
>>>>>>
>>>>>> ------ Segm. Memoria Condivisa --------
>>>>>> chiave     shmid      proprietario perms      byte       nattch
>>>>>> stato
>>>>>>
>>>>>>
>>>>>>
>>>>>> the  ipcs -lm    command report:
>>>>>>
>>>>>> ------ Limiti della memoria condivisa --------
>>>>>> numero massimo di segmenti = 4096
>>>>>> dimensione max seg (kbyte) = 32768
>>>>>> max total shared memory (kbytes) = 8388608
>>>>>> dimensione min seg (byte) = 1
>>>>>>
>>>>>> I need to setup something ?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Dario
>>>>>>
>>>>>>
>>>>>> to see if you have left behind allocated shared memory regions from
>>>>>> earlier runs of GemStone that have left behind their shared page caches
>>>>>> which are using up the available shared memory and leaving none available
>>>>>> for new attempts.  These can be killed off using the unix ipcrm command.
>>>>>> See the relevant unix man pages for details.
>>>>>>
>>>>>> Regards,
>>>>>> BillErickson
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> Bill Erickson
>>>>>> GemTalk Systems Engineering
>>>>>> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> On Tue, Feb 24, 2015 at 2:22 AM, Dario Trussardi via Glass <
>>>>>> glass at lists.gemtalksystems.com> wrote:
>>>>>>
>>>>>>> Ciao.
>>>>>>>
>>>>>>> i create a 'glass'    user  profile with the :   sudo adduser glass
>>>>>>>     command.
>>>>>>>
>>>>>>> But when i startStone  devkit    the system answer the same error.
>>>>>>>
>>>>>>> Ciao,
>>>>>>>
>>>>>>> i have a doubt.
>>>>>>>  I need to create a Glass user profile?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Dario
>>>>>>>
>>>>>>> Dario,
>>>>>>>
>>>>>>> I'm not that familiar with GLASS or the devkit, but know a bit about
>>>>>>> GemStone and can comment at that level -- Dale may chime in with a more
>>>>>>> informed idea later...
>>>>>>>
>>>>>>> The help text gives you some clues where to look:
>>>>>>>
>>>>>>> > An error occurred when initializing the shared cache
>>>>>>>
>>>>>>> Clear enough -- this is what the problem is, now to figure out what
>>>>>>> caused it...
>>>>>>>
>>>>>>> > Help:
>>>>>>>
>>>>>>> >   Operating system kernel configured for shared memory?
>>>>>>>
>>>>>>> Well you got the system started at least once, so it's unlikely you
>>>>>>> haven't configured shared memory properly or it wouldn't have worked the
>>>>>>> first time.
>>>>>>>
>>>>>>> >  Check SharedPageCache Monitor log file (*pcmon.log).
>>>>>>>
>>>>>>>
>>>>>>>  The  *pcmon.log   report:
>>>>>>>
>>>>>>>
>>>>>>>  _____________________________________________________________________________
>>>>>>> |                             GemStone Child Task
>>>>>>>           |
>>>>>>> |
>>>>>>>           |
>>>>>>> |    VERSION: 3.1.0.6, Thu Apr 17 16:12:14 2014
>>>>>>>           |
>>>>>>> |      BUILD: gss64_3_1_0_5_x_branch-33242
>>>>>>>          |
>>>>>>> |  BUILT FOR: x86-64 (Linux)
>>>>>>>          |
>>>>>>> |       MODE: 64 bit
>>>>>>>          |
>>>>>>> | RUNNING ON: 4-CPU scandella x86_64 (Linux 3.13.0-24-generic
>>>>>>> #46-Ubuntu SMP  |
>>>>>>> | Thu Apr 10 19:11:08 UTC 2014) 7789MB
>>>>>>>          |
>>>>>>> | PROCESS ID: 10799     DATE: 02/24/15 10:59:58 CET
>>>>>>>           |
>>>>>>> |   USER IDS: REAL=scandella (1000) EFFECTIVE=scandella (1000)
>>>>>>>          |
>>>>>>> |    COMMAND:
>>>>>>> /opt/oodb/gsDevKitHome/gemstone/products/GemStone64Bit3.1.0.6-x86_64.Linux/sys/startshrpcmon
>>>>>>> | devkit~9af495ccd1149d82 31250 49 0 5000 3906 1 1900 0 432
>>>>>>>           |
>>>>>>>
>>>>>>> |_____________________________________________________________________________|
>>>>>>> SharedPageCache Monitor Summary:
>>>>>>>    the hostname is:    scandella
>>>>>>>    GEMSTONE is:
>>>>>>>  "/opt/oodb/gsDevKitHome/gemstone/stones/devkit/product"
>>>>>>>    shrpcmon arguments are:
>>>>>>>       SharedPageCache Monitor name  devkit~9af495ccd1149d82.
>>>>>>>       Number of pages               31250.
>>>>>>>       Number of processes           49.
>>>>>>>       Cache locked in memory        0.
>>>>>>>       SpinLockCount                 5000.
>>>>>>>       TargetFreeFrameCount          3906.
>>>>>>>       Number of free frame servers  1.
>>>>>>>       Number of shared counters     1900.
>>>>>>>       Port number                   0.
>>>>>>>       Permissions                   432.
>>>>>>>
>>>>>>>  _____________________________________________________________________________
>>>>>>> |             GemStone/S64 Object-Oriented Data Management System
>>>>>>>           |
>>>>>>> |                   Copyright (C) GemTalk Systems 1986-2014
>>>>>>>           |
>>>>>>> |                            All rights reserved.
>>>>>>>           |
>>>>>>>
>>>>>>> +-----------------------------------------------------------------------------+
>>>>>>> |    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor
>>>>>>>          |
>>>>>>> |    VERSION: 3.1.0.6, Thu Apr 17 16:12:14 2014
>>>>>>>           |
>>>>>>> |      BUILD: gss64_3_1_0_5_x_branch-33242
>>>>>>>          |
>>>>>>> |  BUILT FOR: x86-64 (Linux)
>>>>>>>          |
>>>>>>> |       MODE: 64 bit
>>>>>>>          |
>>>>>>> | RUNNING ON: 4-CPU scandella x86_64 (Linux 3.13.0-24-generic
>>>>>>> #46-Ubuntu SMP  |
>>>>>>> | Thu Apr 10 19:11:08 UTC 2014) 7789MB
>>>>>>>          |
>>>>>>> | PROCESS ID: 10799     DATE: 02/24/15 10:59:58 CET
>>>>>>>           |
>>>>>>> |   USER IDS: REAL=scandella (1000) EFFECTIVE=scandella (1000)
>>>>>>>          |
>>>>>>>
>>>>>>> +-----------------------------------------------------------------------------+
>>>>>>> |   GEMSTONE_NRS_ALL = #dir:$GEMSTONE_LOGDIR#log:%N%P.log
>>>>>>>           |
>>>>>>>
>>>>>>> |_____________________________________________________________________________|
>>>>>>> |   GemStone could not retrieve the IPC identifier associated with
>>>>>>> the memory |
>>>>>>> |   key -989659131.  shmget() error = errno=22,EINVAL, Invalid
>>>>>>> argument (programmer
>>>>>>> | error).
>>>>>>>           |
>>>>>>> |
>>>>>>>           |
>>>>>>>   GemStone could not attach to the shared page cache.
>>>>>>>
>>>>>>>   [SpcMon trace]: ... cache creation failed ...
>>>>>>>   [SpcMon trace]: ... if the errno is (EINVAL) it is likely because
>>>>>>> the
>>>>>>>                       cache size is less than the operating system
>>>>>>> imposed
>>>>>>>                       minimum or greater than the operating system
>>>>>>> maximum.
>>>>>>> --- 02/24/15 10:59:58.543 CET :Starting shrpcmonitor shutdown
>>>>>>> --- 02/24/15 10:59:58.543 CET :Waiting for crashed slot recovery
>>>>>>> thread to shutdown...Done.
>>>>>>> --- 02/24/15 10:59:58.543 CET :Waiting for clean slot recovery
>>>>>>> thread to shutdown...Done.
>>>>>>> --- 02/24/15 10:59:58.543 CET :Waiting for stats thread to
>>>>>>> shutdown...Done.
>>>>>>> --- 02/24/15 10:59:58.543 CET :All threads have stopped.
>>>>>>>   The Shared Page Cache Monitor is shutting down.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is your best place to start  -- check this log file for errors
>>>>>>> and it should offer more insight into why the shared page cache could not
>>>>>>> be initialized.  It should point to one of the following possibilities:
>>>>>>>
>>>>>>> >  Check SHR_PAGE_CACHE_NUM_PROCS in stone config file.
>>>>>>>
>>>>>>> Not likely -- it would have prevented the first startup from working.
>>>>>>>
>>>>>>> >  Stale files in /opt/gemstone/locks/<stonename>*?
>>>>>>>
>>>>>>>
>>>>>>> I found the files:  devkit~9af495ccd1149d82     and:
>>>>>>>    gemstone.hostid
>>>>>>>
>>>>>>> Considerations ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Dario
>>>>>>>
>>>>>>> >  ipcs - check for stale resources ?
>>>>>>>
>>>>>>> These two are possibilities -- the log file should give some
>>>>>>> indication.
>>>>>>>
>>>>>>> Good Luck,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>> Bill Erickson
>>>>>>> GemTalk Systems Engineering
>>>>>>> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> On Sat, Feb 21, 2015 at 4:26 AM, Dario Trussardi via Glass <
>>>>>>> glass at lists.gemtalksystems.com> wrote:
>>>>>>>
>>>>>>>> Ciao,
>>>>>>>>
>>>>>>>> i have a Ubuntu system where i load Development Kit Server
>>>>>>>> Installation and
>>>>>>>>
>>>>>>>> installServer devkit 3.1.0.6
>>>>>>>>
>>>>>>>> After installation all works fine and i can login the repository
>>>>>>>> from Gemtools.
>>>>>>>>
>>>>>>>> Now i reboot the system and the startStone devkit   answer:
>>>>>>>>
>>>>>>>> startStone devkit
>>>>>>>>
>>>>>>>> Starting stone: devkit
>>>>>>>>
>>>>>>>> --- 21/02/2015 13:01:31.584 CET ---
>>>>>>>>
>>>>>>>> stopstone[Info]: GemStone version '3.1.0.6'
>>>>>>>>
>>>>>>>> stopstone[Info]: Server 'devkit' is not running.
>>>>>>>>
>>>>>>>> startstone[Info]: GemStone version '3.1.0.6'
>>>>>>>>
>>>>>>>> startstone[Info]: Starting Stone repository monitor 'devkit'.
>>>>>>>>
>>>>>>>> startstone[Info]: GEMSTONE is:
>>>>>>>> '/opt/oodb/gsDevKitHome/gemstone/stones/devkit/product'.
>>>>>>>>
>>>>>>>> startstone[Info]: GEMSTONE_NRS_ALL is:
>>>>>>>> '#dir:$GEMSTONE_LOGDIR#log:%N%P.log'.
>>>>>>>>
>>>>>>>> startstone[Info]:
>>>>>>>>
>>>>>>>>
>>>>>>>> GEMSTONE_SYS_CONF=/opt/oodb/gsDevKitHome/gemstone/stones/devkit/extents/system.conf
>>>>>>>>
>>>>>>>>
>>>>>>>> GEMSTONE_EXE_CONF=/opt/oodb/gsDevKitHome/gemstone/stones/devkit/devkit.conf
>>>>>>>>
>>>>>>>> startstone[Info]: Log file is
>>>>>>>> '/opt/oodb/gsDevKitHome/gemstone/stones/devkit/logs/devkit.log'.
>>>>>>>>
>>>>>>>>
>>>>>>>> startstone[Error]: Stone process (id=1807) has died.
>>>>>>>>
>>>>>>>> startstone[Error]: Examine
>>>>>>>> '/opt/oodb/gsDevKitHome/gemstone/stones/devkit/logs/devkit.log' for more
>>>>>>>> information.  Excerpt follows:
>>>>>>>>
>>>>>>>>  The stone was unable to start a cache page server on host
>>>>>>>> '<stone's host>'.
>>>>>>>>
>>>>>>>> Reason:  connect to cache monitor failed.
>>>>>>>>
>>>>>>>>   Monitor process (1809) did not start.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Help:
>>>>>>>>
>>>>>>>>   Operating system kernel configured for shared memory?
>>>>>>>>
>>>>>>>>   Check SharedPageCache Monitor log file (*pcmon.log).
>>>>>>>>
>>>>>>>>   Check SHR_PAGE_CACHE_NUM_PROCS in stone config file.
>>>>>>>>
>>>>>>>>   Stale files in /opt/gemstone/locks/<stonename>*?
>>>>>>>>
>>>>>>>>   ipcs - check for stale resources ?
>>>>>>>>
>>>>>>>>
>>>>>>>> An error occurred when initializing the shared cache
>>>>>>>>
>>>>>>>>
>>>>>>>>     Stone startup has failed.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The relative devkit.log    report:
>>>>>>>>
>>>>>>>> | Key file:
>>>>>>>> /opt/oodb/gsDevKitHome/gemstone/products/GemStone64Bit3.1.0.6-x86_64.Linux/seaside/etc/gemstone.key
>>>>>>>> | GemStone ID: Free GS/S Web Edition |
>>>>>>>> | LICENSED TO: GemStone Seaside Community |
>>>>>>>> | KEY ORIGIN: 0x5e0a1e8d (a Linux x86) |
>>>>>>>>
>>>>>>>> |_____________________________________________________________________________|
>>>>>>>>
>>>>>>>> ========================================================================
>>>>>>>> Now starting GemStone monitor.
>>>>>>>> Write to /proc/1749/oom_score_adj failed with EACCES , linux user
>>>>>>>> does not have CAP_SYS_RESOURCE
>>>>>>>> No server process protection from OOM killer
>>>>>>>>
>>>>>>>> _____________________________________________________________________________
>>>>>>>> | SESSION CONFIGURATION: The maximum number of concurrent sessions
>>>>>>>> is 41. |
>>>>>>>>
>>>>>>>> |_____________________________________________________________________________|
>>>>>>>>  Attaching the Shared Cache using Stone name: devkit
>>>>>>>>  The stone was unable to start a cache page server on host
>>>>>>>> '<stone's host>'.
>>>>>>>> Reason: connect to cache monitor failed.
>>>>>>>> Monitor process (1751) did not start.
>>>>>>>> Help:
>>>>>>>> Operating system kernel configured for shared memory?
>>>>>>>> Check SharedPageCache Monitor log file (*pcmon.log).
>>>>>>>> Check SHR_PAGE_CACHE_NUM_PROCS in stone config file.
>>>>>>>> Stale files in /opt/gemstone/locks/<stonename>*?
>>>>>>>> ipcs - check for stale resources ?
>>>>>>>> An error occurred when initializing the shared cache
>>>>>>>>  Stone startup has failed.
>>>>>>>>
>>>>>>>> Thanks for any consideration.
>>>>>>>>
>>>>>>>> Dario
>>>>>>>>
>>>>>>>> P.S.  After reboot the system, before any other command,  i do the
>>>>>>>> command :    apt-get install gdisk.
>>>>>>>>
>>>>>>>> Can this create some problem?
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Glass mailing list
>>>>>>>> Glass at lists.gemtalksystems.com
>>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Glass mailing list
>>>>>>> Glass at lists.gemtalksystems.com
>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Glass mailing list
>>>>>>> Glass at lists.gemtalksystems.com
>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Glass mailing list
>>>>>> Glass at lists.gemtalksystems.com
>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Glass mailing list
>>>>>> Glass at lists.gemtalksystems.com
>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Glass mailing list
>>>>> Glass at lists.gemtalksystems.com
>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Glass mailing list
>>>> Glass at lists.gemtalksystems.com
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>>
>>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>> _______________________________________________
>>> Glass mailing list
>>> Glass at lists.gemtalksystems.com
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
>>>
>>
>> _______________________________________________
>> Glass mailing list
>> Glass at lists.gemtalksystems.com
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150224/f08d997d/attachment-0001.html>


More information about the Glass mailing list