[Glass] [GLASS] Seaside - growing extent - normal?
Lawrence Kellogg via Glass
glass at lists.gemtalksystems.com
Wed Apr 1 19:05:43 PDT 2015
> On Mar 31, 2015, at 12:41 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com> wrote:
>
> Larry,
>
> I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...
>
> So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...
>
> So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.
>
Yes, the collection at UserGlobals at: #ExpiryCleanup is empty.
I ran the reapSeasideCache code again.
> Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...
>
> Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...
>
I cleared the caches.
I ran another MFC
>
> At this point there's basically two directions that we can take:
>
> 1. Top down. Start inspecting the data structures in your application and look
> for suspicious collections/objects that could be hanging onto objects above and
> beyond those absolutely needed.
>
> 2. Bottom up. Scan your recent backup and get an instance count report[2] that
> will tell you what class of object is clogging up your data base .... Perhaps you'll
> recognize a big runner or two and know where to look to drop the references.
> If no, we'll have to pick a suspicious class, list the instances of that class, and then
> Repository>>listReferences: to work our way back to a known root and then
> NUKE THE SUCKER:)
Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.
1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView
Larry
>
> Dale
>
> [1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches <https://code.google.com/p/glassdb/wiki/ClearPersistentCaches>
> [2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/ <https://programminggems.wordpress.com/2009/12/15/scanbackup-2/>
> On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:
>>
>>> On Mar 30, 2015, at 6:24 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>
>>> The initial MFC gave you (pre-backup):
>>>
>>> 390,801,691 live objects with 23,382,898 dead
>>>
>>> The second MFC gave you (post-backup):
>>>
>>> 391,007,811 live objects with 107 dead
>>>
>>> Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...
>>>
>>> So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...
>>>
>>> I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...
>>>
>>> I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …
>>
>>
>> Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has:
>>
>> <Mail Attachment.png>
>>
>>
>> Is it necessary to reinitialize the ObjectQueue?
>>
>> Is there some report I can run that will tell me what is holding onto so much space?
>>
>> Best,
>>
>> Larry
>>
>>
>>>
>>> Dale
>>>
>>> On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:
>>>>
>>>>> On Mar 30, 2015, at 12:28 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>>>
>>>>> Okay,
>>>>>
>>>>> I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...
>>>>>
>>>>> So we need to find out what's going on in the new extent ... so do another mfc and send me the results
>>>>
>>>>
>>>> Ok, I made it through another mark for collection and here is the result:
>>>>
>>>> <Mail Attachment.png>
>>>>
>>>>
>>>>
>>>>
>>>> Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much.
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...
>>>>>
>>>>
>>>> Here is the file size report before the mark for collection
>>>>
>>>> Extent #1
>>>> -----------
>>>> Filename = !TCP at localhost#dir:/opt/gemstone/product/seaside/data#log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf <log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf>
>>>>
>>>> File size = 23732.00 Megabytes
>>>> Space available = 3478.58 Megabytes
>>>>
>>>> Totals
>>>> ------
>>>> Repository size = 23732.00 Megabytes
>>>> Free Space = 3478.58 Megabytes
>>>>
>>>> and after
>>>>
>>>> Extent #1
>>>> -----------
>>>> Filename = !TCP at localhost#dir:/opt/gemstone/product/seaside/data#log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf <log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf>
>>>>
>>>> File size = 23732.00 Megabytes
>>>> Space available = 3476.47 Megabytes
>>>>
>>>> Totals
>>>> ------
>>>> Repository size = 23732.00 Megabytes
>>>> Free Space = 3476.47 Megabytes
>>>>
>>>>
>>>>
>>>> I await further instructions.
>>>>
>>>> Best,
>>>>
>>>> Larry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...
>>>>>
>>>>> Dale
>>>>>
>>>>> On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
>>>>>> Hello Dale,
>>>>>>
>>>>>> Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled.
>>>>>> Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?
>>>>>>
>>>>>>
>>>>>> <Mail Attachment.png>
>>>>>>
>>>>>> After the ObjectLog init, and mark, I did a restore into a fresh extent.
>>>>>>
>>>>>> Here is the size of the new extent vs the old, saved extent:
>>>>>>
>>>>>> <Mail Attachment.png>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Larry
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Mar 25, 2015, at 2:15 PM, Dale Henrichs <dale.henrichs at gemtalksystems.com <mailto:dale.henrichs at gemtalksystems.com>> wrote:
>>>>>>>
>>>>>>> Okay here's the sequence of steps that I think you should take:
>>>>>>>
>>>>>>> 1. expire all of your sessions:
>>>>>>>
>>>>>>> | expired |
>>>>>>> Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
>>>>>>> expired := WABasicDevelopment reapSeasideCache.
>>>>>>> expired > 0
>>>>>>> ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
>>>>>>> Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
>>>>>>> System commitTransactions
>>>>>>>
>>>>>>> 2. initalize your object log
>>>>>>>
>>>>>>> 3. run MFC
>>>>>>>
>>>>>>> [
>>>>>>> System abortTransaction.
>>>>>>> SystemRepository markForCollection ]
>>>>>>> on: Warning
>>>>>>> do: [ :ex |
>>>>>>> Transcript
>>>>>>> cr;
>>>>>>> show: ex description.
>>>>>>> ex resume ]
>>>>>>>
>>>>>>> 4. Then do a backup and restore ... you can use GemTools to do the restore,
>>>>>>> but then you should read the SysAdmin docs[1] for instructions to do the restore
>>>>>>> (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
>>>>>>> much be the same, but it's best to look up the docs for your GemStone version[2]
>>>>>>> and follow those instructions)
>>>>>>>
>>>>>>> As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).
>>>>>>>
>>>>>>> Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...
>>>>>>>
>>>>>>> Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...
>>>>>>>
>>>>>>> Dale
>>>>>>>
>>>>>>> [1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm <http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm>
>>>>>>> [2] http://gemtalksystems.com/techsupport/resources/ <http://gemtalksystems.com/techsupport/resources/>
>>>>>>>
>>>>>>> On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
>>>>>>>> Hello Dale,
>>>>>>>>
>>>>>>>> Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL.
>>>>>>>>
>>>>>>>> I’m not running the maintenance VM and I haven’t been doing regular mark for collects.
>>>>>>>>
>>>>>>>> I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup?
>>>>>>>>
>>>>>>>> I’ll also try the ObjectLog trick.
>>>>>>>>
>>>>>>>> I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Larry
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <glass at lists.gemtalksystems.com <mailto:glass at lists.gemtalksystems.com>> wrote:
>>>>>>>>>
>>>>>>>>> Lawrence,
>>>>>>>>>
>>>>>>>>> Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?
>>>>>>>>>
>>>>>>>>> Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...
>>>>>>>>>
>>>>>>>>> Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...
>>>>>>>>>
>>>>>>>>> The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...
>>>>>>>>>
>>>>>>>>> Dale
>>>>>>>>>
>>>>>>>>> On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
>>>>>>>>>> Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.
>>>>>>>>>>
>>>>>>>>>> I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal?
>>>>>>>>>>
>>>>>>>>>> --- 03/07/2015 02:44:14 PM UTC ---
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22528 megabytes.
>>>>>>>>>> Repository has grown to 22528 megabytes.
>>>>>>>>>>
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22544 megabytes.
>>>>>>>>>> Repository has grown to 22544 megabytes.
>>>>>>>>>>
>>>>>>>>>> --- 03/08/2015 03:31:45 PM UTC ---
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22560 megabytes.
>>>>>>>>>> Repository has grown to 22560 megabytes.
>>>>>>>>>>
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22576 megabytes.
>>>>>>>>>> Repository has grown to 22576 megabytes.
>>>>>>>>>>
>>>>>>>>>> --- 03/10/2015 03:19:34 AM UTC ---
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22592 megabytes.
>>>>>>>>>> Repository has grown to 22592 megabytes.
>>>>>>>>>>
>>>>>>>>>> --- 03/10/2015 03:46:39 PM UTC ---
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22608 megabytes.
>>>>>>>>>> Repository has grown to 22608 megabytes.
>>>>>>>>>>
>>>>>>>>>> Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
>>>>>>>>>> has grown to 22624 megabytes.
>>>>>>>>>> Repository has grown to 22624 megabytes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> My extent has now grown to
>>>>>>>>>>
>>>>>>>>>> -rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?
>>>>>>>>>>
>>>>>>>>>> I suppose I should also do a SystemRepository backup, if I can remember the commands.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Larry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Glass mailing list
>>>>>>>>>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>>>>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Glass mailing list
>>>>>>>>> Glass at lists.gemtalksystems.com <mailto:Glass at lists.gemtalksystems.com>
>>>>>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass <http://lists.gemtalksystems.com/mailman/listinfo/glass>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150401/3745ebb5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-04-01 at 10.02.25 PM.png
Type: image/png
Size: 183513 bytes
Desc: not available
URL: <http://lists.gemtalksystems.com/mailman/private/glass/attachments/20150401/3745ebb5/attachment-0001.png>
More information about the Glass
mailing list