Project

Profile

Help

Download (19.1 KB)

Issue #1008 ยป Issue #4818 - 2015-06-11T15_45_25Z.eml

Ben.Stanley, 06/11/2015 05:45 PM

 
Return-Path: <Ben.Stanley@exemail.com.au>
Received: from mi014.mc1.hosteurope.de ([80.237.138.241]) by wp245.webpack.hosteurope.de running ExIM with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) id 1Z34fG-0005BQ-QA; Thu, 11 Jun 2015 17:45:10 +0200
Received: from pecan2-mail.exetel.com.au ([220.233.0.71] helo=pecan2.exetel.com.au) by mx0.webpack.hosteurope.de (mi014.mc1.hosteurope.de) with esmtp id 1Z34fE-0005Zl-FG for dropbox+pulp+c71e@plan.io; Thu, 11 Jun 2015 17:45:10 +0200
Received: from 69.55.70.115.static.exetel.com.au ([115.70.55.69] helo=[192.168.2.117]) by pecan2.exetel.com.au with esmtp (Exim 4.84) (envelope-from <Ben.Stanley@exemail.com.au>) id 1Z34f9-00079D-OM for dropbox+pulp+c71e@plan.io; Fri, 12 Jun 2015 01:45:04 +1000
Date: Fri, 12 Jun 2015 01:45:02 +1000
From: Ben Stanley <Ben.Stanley@exemail.com.au>
To: Pulp <dropbox+pulp+c71e@plan.io>
Message-ID: <14de34bbc48.2811.5c72c04e7b2534944f17f24240f7ac0b@exemail.com.au>
In-Reply-To: <redmine.journal-4815.20150611151841.04db6e3c7610f3d1@plan.io>
References: <redmine.issue-1008.20150527235743@plan.io>
<redmine.journal-4815.20150611151841.04db6e3c7610f3d1@plan.io>
Subject: Re: [Pulp - Issue #1008] Package Missing from repo metadata
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary=----------14de34bc0b1bab28111f26cbb1;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Delivery-date: Thu, 11 Jun 2015 17:45:10 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101
Thunderbird/24.6.0 AquaMail/1.5.7.18 (build: 21070083)
X-HE-Spam-Level: +
X-HE-Spam-Score: 1.4
X-HE-Spam-Report: Content analysis details: (1.4 points) pts rule name
description ---- ----------------------
-------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE
RBL: Sender listed at http://www.dnswl.org/, no trust [220.233.0.71 listed in
list.dnswl.org] 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence
0.1 HTML_MESSAGE BODY: HTML included in message
Envelope-to: dropbox+pulp+c71e@plan.io

This is a multi-part message in MIME format.
------------14de34bc0b1bab28111f26cbb1
Content-Type: text/plain;
charset=UTF-8;
format=flowed
Content-Transfer-Encoding: quoted-printable

This pulp installation started at version 2.5, but the repo in question w=
as =

created with pulp 2.6.1, so I don't think that =

https://bugzilla.redhat.com/show_bug.cgi?id=3D1171509 explains it.

I have had problems with large numbers of tasks on this server, and I =

periodically have to cancel tasks. It is possible that canceling a sync m=
ay =

have caused the error state.

The way to catch the cause of this problem is to add code to perform a de=
ep =

validation that the published repo is equal to the source repo, after a =

sync. This will take extra time to perform, and should be optionally =

enabled, but will determine the conditions that produce the problem. Thin=
k =

of it as a paranoid debug mode with double checking built in.


On 12 June 2015 1:19:07 am Pulp <dropbox+pulp+c71e@plan.io> wrote:

> --- Please write your response above this line ---
>
> Issue #1008 has been updated by bcourt.
>
>
> Hi Ben,
>
> Was this Pulp installation ever running Pulp 2.5.0? There were some =

> specific upgrade instructions for upgrading from 2.5.0 to a newer versi=
on =

> (https://pulp.readthedocs.org/en/latest/user-guide/release-notes/2.5.x.=
html#upgrade-instructions-for-2-5-0-2-5-1).
>
> More generally, incremental publishing does process the RPMs that have =
been =

> added to the repo since the last publish completed successfully. It =

> specifically skips all the RPMs that have already been published. If no=
=

> RPMs have been added since the last successful publish then the step is=
=

> skipped entirely. If an rpm repo publish fails for any reason it should=
not =

> update the last_published timestamp on the repository. For a onetime up=
date =

> of all your repositories to force the next publish to not be incrementa=
l =

> the simplest thing is to update all the yum distributors records in the=
=

> mongo database directly to set last_publish to null, =

> (db.repo_distributors.update({"distributor_type_id": =

> "yum_distributor"},{$set: { "last_update": null }},{multi: true}), and =
then =

> republish all the repos normally. Alternately if you can tolerate the d=
own =

> time for the repos, just delete all the directories in =

> /var/lib/pulp/published/yum/* and then republish your repos.
>
> Given the behavior of the incremental publish, I'm concerned that you w=
ere =

> able to get into a state where things were incorrect. Knowing the code =
I =

> would like to understand how that could happen but there is not a good =
way =

> to do that after the fact.
>
> If you would like to disable incremental publishing there is not curren=
tly =

> a way to do that. Please open a feature request RFE if you would like i=
t. =

> Thanks!
>
> ----------------------------------------
> Issue #1008: Package Missing from repo metadata
> https://pulp.plan.io/issues/1008#change-4815
>
> * Author: Ben.Stanley
> * Status: NEW
> * Priority: High
> * Assignee: jortel@redhat.com
> * Category: pulp-admin
> * Severity: 3. High
> * Version: 2.6.1
> * Target Release:
> * OS: RHEL 7
> * Backwards Incompatible: No
> * Triaged: Yes
> * Tags:
> * QA Contact:
> ----------------------------------------
>
> I have a problem with syncing
> http://elrepo.org/linux/elrepo/el7/x86_64
>
> It seems that the package
> nvidia-x11-drv-304xx-304.125-1.el7.elrepo.x86_64.rpm
> is not included in my yum metadata. However, it is published in my loca=
l
> repo.
>
> [root stig ]# yum list nvidia-x11-drv-304xx
> Loaded plugins: langpacks, product-id, subscription-manager
> Available Packages
> nvidia-x11-drv-304xx.x86_64
> 304.123-2.el7.elrepo elrepo
>
> Only the older version 304.123-2 of the package is found, despite the n=
ewer =

> 304.125-1 being published in the repo.
>
> When I change my /etc/yum.repos.d/elrepo.repo to point to upstream inst=
ead
> of my local pulp server, yum finds the correct nvidia-x11-drv-304xx (af=
ter
> yum clean metadata).
>
> I have recently run a sync, but it has not fixed the problem. There is
> nothing interesting in the log (no exceptions or errors).
>
> I am running pulp 2.6.1 (package versions below).
>
> I have found that I can fix this kind of problem by deleting the repo a=
nd
> re-creating it, but I need to get to the bottom of this so that it stop=
s
> happening. I have complained about something like this previously in
> pulp-2.5.1 at [0] . I created my elrepo in pulp-2.6.1, so it is not
> possible to blame an older version of pulp this time.
>
> I need to get to the bottom of this, and hopefully get this resolved
> properly. Otherwise, I look like a goose when my local copy doesn't wor=
k.
>
> I need to know that after I have completed a sync, that everything real=
ly
> is OK, before I sync to my disconnected system. Performing a complete
> internet sync takes a long time, and a disconnected sync is going to be=

> even worse. I want to prove that everything is right before I do the
> disconnected sync.
>
> Questions:
> 1) Is there a tool that I can use to compare the metadata of two repos =
and
> determine equivalence (at least to the point of validating the sync)? S=
uch
> a tool would allow me to proactively detect such problems, and attempt =
to
> correct them. It would also provide more test cases for you...
>
> 2) Is there some way of investigating why this particular file isn't
> appearing in the metadata?
>
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D rpm -qa | grep pulp=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
> python-pulp-common-2.6.1-1.el7.noarch
> pulp-rpm-admin-extensions-2.6.1-1.el7.noarch
> pulp-server-2.6.1-1.el7.noarch
> python-pulp-client-lib-2.6.1-1.el7.noarch
> python-pulp-puppet-common-2.6.1-1.el7.noarch
> pulp-puppet-plugins-2.6.1-1.el7.noarch
> pulp-admin-client-2.6.1-1.el7.noarch
> python-isodate-0.5.0-4.pulp.el7.noarch
> python-kombu-3.0.24-5.pulp.el7.noarch
> pulp-rpm-plugins-2.6.1-1.el7.noarch
> python-pulp-bindings-2.6.1-1.el7.noarch
> pulp-puppet-admin-extensions-2.6.1-1.el7.noarch
> python-pulp-rpm-common-2.6.1-1.el7.noarch
> pulp-selinux-2.6.1-1.el7.noarch
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> [0] https://pulp.plan.io/issues/942
>
>
> ---Files--------------------------------
> elrepo-el7-x86_64-repodata-20150605.bz (287 KB)
> pulp_20150609_DEBUG_sync_test.log (606 KB)
>
>
> --
> You have received this notification because you have either subscribed =
to =

> or are involved in a project on Pulp Planio.
> To change your notification preferences, please click here: =

> https://pulp.plan.io/my/account?tour=3Dmail_preferences
>
> This notification was cheerfully delivered by https://plan.io
> =C2=A0
>

------------14de34bc0b1bab28111f26cbb1
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style>
@import url(https://assets.plan.io/stylesheets/fonts.css);
body {
font-family: "ProximaNova-Regular", Verdana, sans-serif;
font-size: 1.1em;
color:#333434;
}
h1, h2, h3 { font-family: "ProximaNova-Bold", "Trebuchet MS", Verdana,
sans-serif; margin: 0px; }
h1 { font-size: 1.2em; }
h2, h3 { font-size: 1.1em; }
a, a:link, a:visited, a:hover, a:active { color:#2b7a94; }
a.wiki-anchor { display: none; }
fieldset.attachments {border-width: 1px 0 0 0;}
hr {
width: 100%;
height: 1px;
background: #ccc;
border: 0;
}
</style>


</head>
<body>
<div style=3D"color: black;">
<div style=3D"color: black;">
<p style=3D"margin: 0 0 1em 0; color: black;">This pulp installation star=
ted
at version 2.5, but the repo in question was created with pulp 2.6.1, so =
I
don't think that <a
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1171509">https://bu=
gzilla.redhat.com/show_bug.cgi?id=3D1171509</a>
explains it.</p>
<p style=3D"margin: 0 0 1em 0; color: black;">I have had problems with la=
rge
numbers of tasks on this server, and I periodically have to cancel tasks.=

It is possible that canceling a sync may have caused the error state.</p>=

<p style=3D"margin: 0 0 1em 0; color: black;">The way to catch the cause =
of
this problem is to add code to perform a deep validation that the publish=
ed
repo is equal to the source repo, after a sync. This will take extra time=

to perform, and should be optionally enabled, but will determine the
conditions that produce the problem. Think of it as a paranoid debug mode=

with double checking built in.</p>
</div>
<div style=3D"color: black;">
<p
style=3D"color: black; font-size: 10pt; font-family: Arial, sans-serif; m=
argin: 10pt 0;">On
12 June 2015 1:19:07 am Pulp &lt;dropbox+pulp+c71e@plan.io&gt; wrote:</p>=

<blockquote type=3D"cite" class=3D"gmail_quote"
style=3D"margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-le=
ft: 0.75ex;"><table
width=3D"100%">
<tr><td
style=3D"font-family: MarketWeb, Verdana, sans-serif;font-size:0.8em;text=
-align:center;width:100%;color:#D7D7D7;"><p>---
Please write your response above this line ---</p></td></tr>
<tr><td>Issue #1008 has been updated by bcourt.

<ul>
</ul>

<p>Hi Ben,</p>


<p>Was this Pulp installation ever running Pulp 2.5.0? There were some
specific upgrade instructions for upgrading from 2.5.0 to a newer version=

(<a class=3D"external"
href=3D"https://pulp.readthedocs.org/en/latest/user-guide/release-notes/2=
.5.x.html#upgrade-instructions-for-2-5-0-2-5-1">https://pulp.readthedocs.=
org/en/latest/user-guide/release-notes/2.5.x.html#upgrade-instructions-fo=
r-2-5-0-2-5-1</a>).</p>


<p>More generally, incremental publishing does process the RPMs that hav=
e
been added to the repo since the last publish completed successfully. It
specifically skips all the RPMs that have already been published. If no
RPMs have been added since the last successful publish then the step is
skipped entirely. If an rpm repo publish fails for any reason it should n=
ot
update the last_published timestamp on the repository. For a onetime upda=
te
of all your repositories to force the next publish to not be incremental
the simplest thing is to update all the yum distributors records in the
mongo database directly to set last_publish to null,
(db.repo_distributors.update({"distributor_type_id":
"yum_distributor"},{$set: { "last_update": null }},{multi: true}), and th=
en
republish all the repos normally. Alternately if you can tolerate the dow=
n
time for the repos, just delete all the directories in
/var/lib/pulp/published/yum/* and then republish your repos.</p>


<p>Given the behavior of the incremental publish, I'm concerned that you=

were able to get into a state where things were incorrect. Knowing the co=
de
I would like to understand how that could happen but there is not a good
way to do that after the fact.</p>


<p>If you would like to disable incremental publishing there is not
currently a way to do that. Please open a feature request RFE if you woul=
d
like it. Thanks!</p>
<hr />
<h1><a href=3D"https://pulp.plan.io/issues/1008#change-4815">Issue #1008:=

Package Missing from repo metadata</a></h1>

<ul><li>Author: Ben.Stanley</li>
<li>Status: NEW</li>
<li>Priority: High</li>
<li>Assignee: jortel@redhat.com</li>
<li>Category: pulp-admin</li>
<li>Severity: 3. High</li>
<li>Version: 2.6.1</li>
<li>Target Release: </li>
<li>OS: RHEL 7</li>
<li>Backwards Incompatible: No</li>
<li>Triaged: Yes</li>
<li>Tags: </li>
<li>QA Contact: </li></ul>

<p>I have a problem with syncing<br /><a class=3D"external"
href=3D"http://elrepo.org/linux/elrepo/el7/x86_64">http://elrepo.org/linu=
x/elrepo/el7/x86_64</a></p>


<p>It seems that the package<br
/>nvidia-x11-drv-304xx-304.125-1.el7.elrepo.x86_64.rpm<br />is not includ=
ed
in my yum metadata. However, it is published in my local<br />repo.</p>


<p>[root stig ]# yum list nvidia-x11-drv-304xx<br />Loaded plugins:
langpacks, product-id, subscription-manager<br />Available Packages<br
/>nvidia-x11-drv-304xx.x86_64 <br
/>304.123-2.el7.elrepo elrepo</p>


<p>Only the older version 304.123-2 of the package is found, despite the=

newer 304.125-1 being published in the repo.</p>


<p>When I change my /etc/yum.repos.d/elrepo.repo to point to upstream
instead<br />of my local pulp server, yum finds the correct
nvidia-x11-drv-304xx (after<br />yum clean metadata).</p>


<p>I have recently run a sync, but it has not fixed the problem. There
is<br />nothing interesting in the log (no exceptions or errors).</p>


<p>I am running pulp 2.6.1 (package versions below).</p>


<p>I have found that I can fix this kind of problem by deleting the repo=

and<br />re-creating it, but I need to get to the bottom of this so that =
it
stops<br />happening. I have complained about something like this
previously in<br />pulp-2.5.1 at [0] . I created my elrepo in pulp-2.6.1,=

so it is not<br />possible to blame an older version of pulp this time.</=
p>


<p>I need to get to the bottom of this, and hopefully get this resolved<=
br
/>properly. Otherwise, I look like a goose when my local copy doesn't wor=
k.</p>


<p>I need to know that after I have completed a sync, that everything
really<br />is OK, before I sync to my disconnected system. Performing a
complete<br />internet sync takes a long time, and a disconnected sync is=

going to be<br />even worse. I want to prove that everything is right
before I do the<br />disconnected sync.</p>


<p>Questions:<br />1) Is there a tool that I can use to compare the
metadata of two repos and<br />determine equivalence (at least to the poi=
nt
of validating the sync)? Such<br />a tool would allow me to proactively
detect such problems, and attempt to<br />correct them. It would also
provide more test cases for you...</p>


<p>2) Is there some way of investigating why this particular file isn't<=
br
/>appearing in the metadata?</p>


<p>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D rpm -qa | grep pu=
lp =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D<br
/>python-pulp-common-2.6.1-1.el7.noarch<br
/>pulp-rpm-admin-extensions-2.6.1-1.el7.noarch<br
/>pulp-server-2.6.1-1.el7.noarch<br
/>python-pulp-client-lib-2.6.1-1.el7.noarch<br
/>python-pulp-puppet-common-2.6.1-1.el7.noarch<br
/>pulp-puppet-plugins-2.6.1-1.el7.noarch<br
/>pulp-admin-client-2.6.1-1.el7.noarch<br
/>python-isodate-0.5.0-4.pulp.el7.noarch<br
/>python-kombu-3.0.24-5.pulp.el7.noarch<br
/>pulp-rpm-plugins-2.6.1-1.el7.noarch<br
/>python-pulp-bindings-2.6.1-1.el7.noarch<br
/>pulp-puppet-admin-extensions-2.6.1-1.el7.noarch<br
/>python-pulp-rpm-common-2.6.1-1.el7.noarch<br
/>pulp-selinux-2.6.1-1.el7.noarch
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</p>


<p>[0] <a class=3D"external"
href=3D"https://pulp.plan.io/issues/942">https://pulp.plan.io/issues/942<=
/a></p>

<fieldset class=3D"attachments"><legend>Files</legend>
<a
href=3D"https://pulp.plan.io/attachments/download/136/elrepo-el7-x86_64-r=
epodata-20150605.bz">elrepo-el7-x86_64-repodata-20150605.bz</a>
(287 KB)<br />
<a
href=3D"https://pulp.plan.io/attachments/download/141/pulp_20150609_DEBUG=
_sync_test.log">pulp_20150609_DEBUG_sync_test.log</a>
(606 KB)<br />
</fieldset>

<script type=3D"application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"action": {
"@type": "ViewAction",
"url": "https://pulp.plan.io/issues/1008#change-4815",
"name": "View on Planio"
},
"description": "Click here to view this issue update on Planio."
}
</script>

</td></tr>
<tr><td style=3D"font-size:0.8em;width:100%;"><hr/><p>You have received t=
his
notification because you have either subscribed to or are involved in a
project on Pulp Planio.<br />To change your notification preferences,
please click here: <a class=3D"external"
href=3D"https://pulp.plan.io/my/account?tour=3Dmail_preferences">https://=
pulp.plan.io/my/account?tour=3Dmail_preferences</a></p><td></tr>
<tr><td
style=3D"font-family: MarketWeb, Verdana, sans-serif;font-size:1.2em;text=
-align:center;width:100%;color:#D7D7D7;"><br/><div><a
href=3D"https://plan.io/" style=3D"color:#D7D7D7;text-decoration:none;">T=
his
notification was cheerfully delivered by</a></div><td></tr>
<tr><td style=3D"text-align:center;width:100%;"><a href=3D"https://plan.i=
o/"
title=3D"Planio"><img
src=3D"https://assets.plan.io/images/planio_logo_gray_204x50.png" height=3D=
"25"
width=3D"102" alt=3D"Planio" style=3D"vertical-align: middle;"/></a></td>=
</tr>
</table>
</blockquote>
</div>
</div>
</body>
</html>

------------14de34bc0b1bab28111f26cbb1--
    (1-1/1)