Project

Profile

Help

Issue #1008 ยป Issue #4818 - 2015-06-11T15_45_25Z.eml

Ben.Stanley, 06/11/2015 05:45 PM

 
1
Return-Path: <Ben.Stanley@exemail.com.au>
2
Received: from mi014.mc1.hosteurope.de ([80.237.138.241]) by wp245.webpack.hosteurope.de running ExIM with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) id 1Z34fG-0005BQ-QA; Thu, 11 Jun 2015 17:45:10 +0200
3
Received: from pecan2-mail.exetel.com.au ([220.233.0.71] helo=pecan2.exetel.com.au) by mx0.webpack.hosteurope.de (mi014.mc1.hosteurope.de) with esmtp id 1Z34fE-0005Zl-FG for dropbox+pulp+c71e@plan.io; Thu, 11 Jun 2015 17:45:10 +0200
4
Received: from 69.55.70.115.static.exetel.com.au ([115.70.55.69] helo=[192.168.2.117]) by pecan2.exetel.com.au with esmtp (Exim 4.84) (envelope-from <Ben.Stanley@exemail.com.au>) id 1Z34f9-00079D-OM for dropbox+pulp+c71e@plan.io; Fri, 12 Jun 2015 01:45:04 +1000
5
Date: Fri, 12 Jun 2015 01:45:02 +1000
6
From: Ben Stanley <Ben.Stanley@exemail.com.au>
7
To: Pulp <dropbox+pulp+c71e@plan.io>
8
Message-ID: <14de34bbc48.2811.5c72c04e7b2534944f17f24240f7ac0b@exemail.com.au>
9
In-Reply-To: <redmine.journal-4815.20150611151841.04db6e3c7610f3d1@plan.io>
10
References: <redmine.issue-1008.20150527235743@plan.io>
11
 <redmine.journal-4815.20150611151841.04db6e3c7610f3d1@plan.io>
12
Subject: Re: [Pulp - Issue #1008] Package Missing from repo metadata
13
Mime-Version: 1.0
14
Content-Type: multipart/alternative;
15
 boundary=----------14de34bc0b1bab28111f26cbb1;
16
 charset=UTF-8
17
Content-Transfer-Encoding: 7bit
18
Delivery-date: Thu, 11 Jun 2015 17:45:10 +0200
19
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101
20
 Thunderbird/24.6.0 AquaMail/1.5.7.18 (build: 21070083)
21
X-HE-Spam-Level: +
22
X-HE-Spam-Score: 1.4
23
X-HE-Spam-Report: Content analysis details: (1.4 points) pts rule name
24
 description ---- ----------------------
25
 -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE
26
 RBL: Sender listed at http://www.dnswl.org/, no trust [220.233.0.71 listed in
27
 list.dnswl.org] 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence
28
 0.1 HTML_MESSAGE BODY: HTML included in message
29
Envelope-to: dropbox+pulp+c71e@plan.io
30

    
31
This is a multi-part message in MIME format.
32
------------14de34bc0b1bab28111f26cbb1
33
Content-Type: text/plain;
34
 charset=UTF-8;
35
 format=flowed
36
Content-Transfer-Encoding: quoted-printable
37

    
38
This pulp installation started at version 2.5, but the repo in question w=
39
as =
40

    
41
created with pulp 2.6.1, so I don't think that =
42

    
43
https://bugzilla.redhat.com/show_bug.cgi?id=3D1171509 explains it.
44

    
45
I have had problems with large numbers of tasks on this server, and I =
46

    
47
periodically have to cancel tasks. It is possible that canceling a sync m=
48
ay =
49

    
50
have caused the error state.
51

    
52
The way to catch the cause of this problem is to add code to perform a de=
53
ep =
54

    
55
validation that the published repo is equal to the source repo, after a =
56

    
57
sync. This will take extra time to perform, and should be optionally =
58

    
59
enabled, but will determine the conditions that produce the problem. Thin=
60
k =
61

    
62
of it as a paranoid debug mode with double checking built in.
63

    
64

    
65
On 12 June 2015 1:19:07 am Pulp <dropbox+pulp+c71e@plan.io> wrote:
66

    
67
> --- Please write your response above this line ---
68
>
69
> Issue #1008 has been updated by bcourt.
70
>
71
>
72
> Hi Ben,
73
>
74
> Was this Pulp installation ever running Pulp 2.5.0?  There were some =
75

    
76
> specific upgrade instructions for upgrading from 2.5.0 to a newer versi=
77
on =
78

    
79
> (https://pulp.readthedocs.org/en/latest/user-guide/release-notes/2.5.x.=
80
html#upgrade-instructions-for-2-5-0-2-5-1).
81
>
82
> More generally, incremental publishing does process the RPMs that have =
83
been =
84

    
85
> added to the repo since the last publish completed successfully. It =
86

    
87
> specifically skips all the RPMs that have already been published. If no=
88
 =
89

    
90
> RPMs have been added since the last successful publish then the step is=
91
 =
92

    
93
> skipped entirely. If an rpm repo publish fails for any reason it should=
94
 not =
95

    
96
> update the last_published timestamp on the repository. For a onetime up=
97
date =
98

    
99
> of all your repositories to force the next publish to not be incrementa=
100
l =
101

    
102
> the simplest thing is to update all the yum distributors records in the=
103
 =
104

    
105
> mongo database directly to set last_publish to null, =
106

    
107
> (db.repo_distributors.update({"distributor_type_id": =
108

    
109
> "yum_distributor"},{$set: { "last_update": null }},{multi: true}), and =
110
then =
111

    
112
> republish all the repos normally. Alternately if you can tolerate the d=
113
own =
114

    
115
> time for the repos, just delete all the directories in =
116

    
117
> /var/lib/pulp/published/yum/* and then republish your repos.
118
>
119
> Given the behavior of the incremental publish, I'm concerned that you w=
120
ere =
121

    
122
> able to get into a state where things were incorrect. Knowing the code =
123
I =
124

    
125
> would like to understand how that could happen but there is not a good =
126
way =
127

    
128
> to do that after the fact.
129
>
130
> If you would like to disable incremental publishing there is not curren=
131
tly =
132

    
133
> a way to do that. Please open a feature request RFE if you would like i=
134
t.  =
135

    
136
> Thanks!
137
>
138
> ----------------------------------------
139
> Issue #1008: Package Missing from repo metadata
140
> https://pulp.plan.io/issues/1008#change-4815
141
>
142
> * Author: Ben.Stanley
143
> * Status: NEW
144
> * Priority: High
145
> * Assignee: jortel@redhat.com
146
> * Category: pulp-admin
147
> * Severity: 3. High
148
> * Version: 2.6.1
149
> * Target Release:
150
> * OS: RHEL 7
151
> * Backwards Incompatible: No
152
> * Triaged: Yes
153
> * Tags:
154
> * QA Contact:
155
> ----------------------------------------
156
>
157
> I have a problem with syncing
158
> http://elrepo.org/linux/elrepo/el7/x86_64
159
>
160
> It seems that the package
161
> nvidia-x11-drv-304xx-304.125-1.el7.elrepo.x86_64.rpm
162
> is not included in my yum metadata. However, it is published in my loca=
163
l
164
> repo.
165
>
166
> [root stig ]# yum list nvidia-x11-drv-304xx
167
> Loaded plugins: langpacks, product-id, subscription-manager
168
> Available Packages
169
> nvidia-x11-drv-304xx.x86_64
170
> 304.123-2.el7.elrepo                                    elrepo
171
>
172
> Only the older version 304.123-2 of the package is found, despite the n=
173
ewer =
174

    
175
> 304.125-1 being published in the repo.
176
>
177
> When I change my /etc/yum.repos.d/elrepo.repo to point to upstream inst=
178
ead
179
> of my local pulp server, yum finds the correct nvidia-x11-drv-304xx (af=
180
ter
181
> yum clean metadata).
182
>
183
> I have recently run a sync, but it has not fixed the problem. There is
184
> nothing interesting in the log (no exceptions or errors).
185
>
186
> I am running pulp 2.6.1 (package versions below).
187
>
188
> I have found that I can fix this kind of problem by deleting the repo a=
189
nd
190
> re-creating it, but I need to get to the bottom of this so that it stop=
191
s
192
> happening. I have complained about something like this previously in
193
> pulp-2.5.1 at [0] . I created my elrepo in pulp-2.6.1, so it is not
194
> possible to blame an older version of pulp this time.
195
>
196
> I need to get to the bottom of this, and hopefully get this resolved
197
> properly. Otherwise, I look like a goose when my local copy doesn't wor=
198
k.
199
>
200
> I need to know that after I have completed a sync, that everything real=
201
ly
202
> is OK, before I sync to my disconnected system. Performing a complete
203
> internet sync takes a long time, and a disconnected sync is going to be=
204

    
205
> even worse. I want to prove that everything is right before I do the
206
> disconnected sync.
207
>
208
> Questions:
209
> 1) Is there a tool that I can use to compare the metadata of two repos =
210
and
211
> determine equivalence (at least to the point of validating the sync)? S=
212
uch
213
> a tool would allow me to proactively detect such problems, and attempt =
214
to
215
> correct them. It would also provide more test cases for you...
216
>
217
> 2) Is there some way of investigating why this particular file isn't
218
> appearing in the metadata?
219
>
220
>
221
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D rpm -qa | grep pulp=
222
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
223
=3D=3D
224
> python-pulp-common-2.6.1-1.el7.noarch
225
> pulp-rpm-admin-extensions-2.6.1-1.el7.noarch
226
> pulp-server-2.6.1-1.el7.noarch
227
> python-pulp-client-lib-2.6.1-1.el7.noarch
228
> python-pulp-puppet-common-2.6.1-1.el7.noarch
229
> pulp-puppet-plugins-2.6.1-1.el7.noarch
230
> pulp-admin-client-2.6.1-1.el7.noarch
231
> python-isodate-0.5.0-4.pulp.el7.noarch
232
> python-kombu-3.0.24-5.pulp.el7.noarch
233
> pulp-rpm-plugins-2.6.1-1.el7.noarch
234
> python-pulp-bindings-2.6.1-1.el7.noarch
235
> pulp-puppet-admin-extensions-2.6.1-1.el7.noarch
236
> python-pulp-rpm-common-2.6.1-1.el7.noarch
237
> pulp-selinux-2.6.1-1.el7.noarch
238
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
239
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
240
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
241
>
242
> [0] https://pulp.plan.io/issues/942
243
>
244
>
245
> ---Files--------------------------------
246
> elrepo-el7-x86_64-repodata-20150605.bz (287 KB)
247
> pulp_20150609_DEBUG_sync_test.log (606 KB)
248
>
249
>
250
> --
251
> You have received this notification because you have either subscribed =
252
to =
253

    
254
> or are involved in a project on Pulp Planio.
255
> To change your notification preferences, please click here: =
256

    
257
> https://pulp.plan.io/my/account?tour=3Dmail_preferences
258
>
259
> This notification was cheerfully delivered by https://plan.io
260
> =C2=A0
261
>
262

    
263
------------14de34bc0b1bab28111f26cbb1
264
Content-Type: text/html;
265
 charset=UTF-8
266
Content-Transfer-Encoding: quoted-printable
267

    
268
<html>
269
<head>
270
<style>
271
@import url(https://assets.plan.io/stylesheets/fonts.css);
272
body {
273
  font-family: "ProximaNova-Regular", Verdana, sans-serif;
274
  font-size: 1.1em;
275
  color:#333434;
276
}
277
h1, h2, h3 { font-family: "ProximaNova-Bold", "Trebuchet MS", Verdana,
278
sans-serif; margin: 0px; }
279
h1 { font-size: 1.2em; }
280
h2, h3 { font-size: 1.1em; }
281
a, a:link, a:visited, a:hover, a:active { color:#2b7a94; }
282
a.wiki-anchor { display: none; }
283
fieldset.attachments {border-width: 1px 0 0 0;}
284
hr {
285
  width: 100%;
286
  height: 1px;
287
  background: #ccc;
288
  border: 0;
289
}
290
</style>
291

    
292

    
293
</head>
294
<body>
295
<div style=3D"color: black;">
296
<div style=3D"color: black;">
297
<p style=3D"margin: 0 0 1em 0; color: black;">This pulp installation star=
298
ted
299
at version 2.5, but the repo in question was created with pulp 2.6.1, so =
300
I
301
don't think that <a
302
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1171509">https://bu=
303
gzilla.redhat.com/show_bug.cgi?id=3D1171509</a>
304
explains it.</p>
305
<p style=3D"margin: 0 0 1em 0; color: black;">I have had problems with la=
306
rge
307
numbers of tasks on this server, and I periodically have to cancel tasks.=
308

    
309
It is possible that canceling a sync may have caused the error state.</p>=
310

    
311
<p style=3D"margin: 0 0 1em 0; color: black;">The way to catch the cause =
312
of
313
this problem is to add code to perform a deep validation that the publish=
314
ed
315
repo is equal to the source repo, after a sync. This will take extra time=
316

    
317
to perform, and should be optionally enabled, but will determine the
318
conditions that produce the problem. Think of it as a paranoid debug mode=
319

    
320
with double checking built in.</p>
321
</div>
322
<div style=3D"color: black;">
323
<p
324
style=3D"color: black; font-size: 10pt; font-family: Arial, sans-serif; m=
325
argin: 10pt 0;">On
326
12 June 2015 1:19:07 am Pulp &lt;dropbox+pulp+c71e@plan.io&gt; wrote:</p>=
327

    
328
<blockquote type=3D"cite" class=3D"gmail_quote"
329
style=3D"margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-le=
330
ft: 0.75ex;"><table
331
width=3D"100%">
332
<tr><td
333
style=3D"font-family: MarketWeb, Verdana, sans-serif;font-size:0.8em;text=
334
-align:center;width:100%;color:#D7D7D7;"><p>---
335
Please write your response above this line ---</p></td></tr>
336
<tr><td>Issue #1008 has been updated by bcourt.
337

    
338
<ul>
339
</ul>
340

    
341
<p>Hi Ben,</p>
342

    
343

    
344
	<p>Was this Pulp installation ever running Pulp 2.5.0?  There were some
345
specific upgrade instructions for upgrading from 2.5.0 to a newer version=
346

    
347
(<a class=3D"external"
348
href=3D"https://pulp.readthedocs.org/en/latest/user-guide/release-notes/2=
349
.5.x.html#upgrade-instructions-for-2-5-0-2-5-1">https://pulp.readthedocs.=
350
org/en/latest/user-guide/release-notes/2.5.x.html#upgrade-instructions-fo=
351
r-2-5-0-2-5-1</a>).</p>
352

    
353

    
354
	<p>More generally, incremental publishing does process the RPMs that hav=
355
e
356
been added to the repo since the last publish completed successfully. It
357
specifically skips all the RPMs that have already been published. If no
358
RPMs have been added since the last successful publish then the step is
359
skipped entirely. If an rpm repo publish fails for any reason it should n=
360
ot
361
update the last_published timestamp on the repository. For a onetime upda=
362
te
363
of all your repositories to force the next publish to not be incremental
364
the simplest thing is to update all the yum distributors records in the
365
mongo database directly to set last_publish to null,
366
(db.repo_distributors.update({"distributor_type_id":
367
"yum_distributor"},{$set: { "last_update": null }},{multi: true}), and th=
368
en
369
republish all the repos normally. Alternately if you can tolerate the dow=
370
n
371
time for the repos, just delete all the directories in
372
/var/lib/pulp/published/yum/* and then republish your repos.</p>
373

    
374

    
375
	<p>Given the behavior of the incremental publish, I'm concerned that you=
376

    
377
were able to get into a state where things were incorrect. Knowing the co=
378
de
379
I would like to understand how that could happen but there is not a good
380
way to do that after the fact.</p>
381

    
382

    
383
	<p>If you would like to disable incremental publishing there is not
384
currently a way to do that. Please open a feature request RFE if you woul=
385
d
386
like it.  Thanks!</p>
387
<hr />
388
<h1><a href=3D"https://pulp.plan.io/issues/1008#change-4815">Issue #1008:=
389

    
390
Package Missing from repo metadata</a></h1>
391

    
392
<ul><li>Author: Ben.Stanley</li>
393
<li>Status: NEW</li>
394
<li>Priority: High</li>
395
<li>Assignee: jortel@redhat.com</li>
396
<li>Category: pulp-admin</li>
397
<li>Severity: 3. High</li>
398
<li>Version: 2.6.1</li>
399
<li>Target Release: </li>
400
<li>OS: RHEL 7</li>
401
<li>Backwards Incompatible: No</li>
402
<li>Triaged: Yes</li>
403
<li>Tags: </li>
404
<li>QA Contact: </li></ul>
405

    
406
<p>I have a problem with syncing<br /><a class=3D"external"
407
href=3D"http://elrepo.org/linux/elrepo/el7/x86_64">http://elrepo.org/linu=
408
x/elrepo/el7/x86_64</a></p>
409

    
410

    
411
	<p>It seems that the package<br
412
/>nvidia-x11-drv-304xx-304.125-1.el7.elrepo.x86_64.rpm<br />is not includ=
413
ed
414
in my yum metadata. However, it is published in my local<br />repo.</p>
415

    
416

    
417
	<p>[root stig ]# yum list nvidia-x11-drv-304xx<br />Loaded plugins:
418
langpacks, product-id, subscription-manager<br />Available Packages<br
419
/>nvidia-x11-drv-304xx.x86_64                                   <br
420
/>304.123-2.el7.elrepo                                    elrepo</p>
421

    
422

    
423
	<p>Only the older version 304.123-2 of the package is found, despite the=
424

    
425
newer 304.125-1 being published in the repo.</p>
426

    
427

    
428
	<p>When I change my /etc/yum.repos.d/elrepo.repo to point to upstream
429
instead<br />of my local pulp server, yum finds the correct
430
nvidia-x11-drv-304xx (after<br />yum clean metadata).</p>
431

    
432

    
433
	<p>I have recently run a sync, but it has not fixed the problem. There
434
is<br />nothing interesting in the log (no exceptions or errors).</p>
435

    
436

    
437
	<p>I am running pulp 2.6.1 (package versions below).</p>
438

    
439

    
440
	<p>I have found that I can fix this kind of problem by deleting the repo=
441

    
442
and<br />re-creating it, but I need to get to the bottom of this so that =
443
it
444
stops<br />happening. I have complained about something like this
445
previously in<br />pulp-2.5.1 at [0] . I created my elrepo in pulp-2.6.1,=
446

    
447
so it is not<br />possible to blame an older version of pulp this time.</=
448
p>
449

    
450

    
451
	<p>I need to get to the bottom of this, and hopefully get this resolved<=
452
br
453
/>properly. Otherwise, I look like a goose when my local copy doesn't wor=
454
k.</p>
455

    
456

    
457
	<p>I need to know that after I have completed a sync, that everything
458
really<br />is OK, before I sync to my disconnected system. Performing a
459
complete<br />internet sync takes a long time, and a disconnected sync is=
460

    
461
going to be<br />even worse. I want to prove that everything is right
462
before I do the<br />disconnected sync.</p>
463

    
464

    
465
	<p>Questions:<br />1) Is there a tool that I can use to compare the
466
metadata of two repos and<br />determine equivalence (at least to the poi=
467
nt
468
of validating the sync)? Such<br />a tool would allow me to proactively
469
detect such problems, and attempt to<br />correct them. It would also
470
provide more test cases for you...</p>
471

    
472

    
473
	<p>2) Is there some way of investigating why this particular file isn't<=
474
br
475
/>appearing in the metadata?</p>
476

    
477

    
478
	<p>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D rpm -qa | grep pu=
479
lp =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
480
=3D=3D<br
481
/>python-pulp-common-2.6.1-1.el7.noarch<br
482
/>pulp-rpm-admin-extensions-2.6.1-1.el7.noarch<br
483
/>pulp-server-2.6.1-1.el7.noarch<br
484
/>python-pulp-client-lib-2.6.1-1.el7.noarch<br
485
/>python-pulp-puppet-common-2.6.1-1.el7.noarch<br
486
/>pulp-puppet-plugins-2.6.1-1.el7.noarch<br
487
/>pulp-admin-client-2.6.1-1.el7.noarch<br
488
/>python-isodate-0.5.0-4.pulp.el7.noarch<br
489
/>python-kombu-3.0.24-5.pulp.el7.noarch<br
490
/>pulp-rpm-plugins-2.6.1-1.el7.noarch<br
491
/>python-pulp-bindings-2.6.1-1.el7.noarch<br
492
/>pulp-puppet-admin-extensions-2.6.1-1.el7.noarch<br
493
/>python-pulp-rpm-common-2.6.1-1.el7.noarch<br
494
/>pulp-selinux-2.6.1-1.el7.noarch
495
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
496
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
497
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</p>
498

    
499

    
500
	<p>[0] <a class=3D"external"
501
href=3D"https://pulp.plan.io/issues/942">https://pulp.plan.io/issues/942<=
502
/a></p>
503

    
504
  <fieldset class=3D"attachments"><legend>Files</legend>
505
    <a
506
href=3D"https://pulp.plan.io/attachments/download/136/elrepo-el7-x86_64-r=
507
epodata-20150605.bz">elrepo-el7-x86_64-repodata-20150605.bz</a>
508
    (287 KB)<br />
509
    <a
510
href=3D"https://pulp.plan.io/attachments/download/141/pulp_20150609_DEBUG=
511
_sync_test.log">pulp_20150609_DEBUG_sync_test.log</a>
512
    (606 KB)<br />
513
  </fieldset>
514

    
515
<script type=3D"application/ld+json">
516
{
517
  "@context": "http://schema.org",
518
  "@type": "EmailMessage",
519
  "action": {
520
    "@type": "ViewAction",
521
    "url": "https://pulp.plan.io/issues/1008#change-4815",
522
    "name": "View on Planio"
523
  },
524
  "description": "Click here to view this issue update on Planio."
525
}
526
</script>
527

    
528
</td></tr>
529
<tr><td style=3D"font-size:0.8em;width:100%;"><hr/><p>You have received t=
530
his
531
notification because you have either subscribed to or are involved in a
532
project on Pulp Planio.<br />To change your notification preferences,
533
please click here: <a class=3D"external"
534
href=3D"https://pulp.plan.io/my/account?tour=3Dmail_preferences">https://=
535
pulp.plan.io/my/account?tour=3Dmail_preferences</a></p><td></tr>
536
<tr><td
537
style=3D"font-family: MarketWeb, Verdana, sans-serif;font-size:1.2em;text=
538
-align:center;width:100%;color:#D7D7D7;"><br/><div><a
539
href=3D"https://plan.io/" style=3D"color:#D7D7D7;text-decoration:none;">T=
540
his
541
notification was cheerfully delivered by</a></div><td></tr>
542
<tr><td style=3D"text-align:center;width:100%;"><a href=3D"https://plan.i=
543
o/"
544
title=3D"Planio"><img
545
src=3D"https://assets.plan.io/images/planio_logo_gray_204x50.png" height=3D=
546
"25"
547
width=3D"102" alt=3D"Planio" style=3D"vertical-align: middle;"/></a></td>=
548
</tr>
549
</table>
550
</blockquote>
551
</div>
552
</div>
553
</body>
554
</html>
555

    
556
------------14de34bc0b1bab28111f26cbb1--
    (1-1/1)