Issue #5449
closed
Multiple source repos copy of errata produces different results
Description
Repos used in this issue:
Repo 1 - Contains an errata, and it is missing a RPM version required by the errata.
Repo 2 - Contains the RPM version required by the errata.
Repo 3 - Destination repository.
When copying the errata from Repo 1 to Repo 3, using the "additional_repos" to add Repo 2 , when execute for the first all the dependency resolution is executed properly.
If all the same steps above are execute again, the final result will be different.
Script used to recreate:
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/
pulp-admin rpm repo sync run --repo-id=zoo
pulp-admin rpm repo create --repo-id=bar --relative-url=bar --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/
pulp-admin rpm repo sync run --repo-id=bar
pulp-admin rpm repo create --repo-id=test1
curl -k -u admin:admin --cert ~/.pulp/user-cert.pem -d '{"source_repo_id":"zoo","criteria":{"type_ids":["erratum"],"filters":{"unit":{"id":"RHEA-2012:0059"}}},"override_config":{"recursive_conservative":true,"additional_repos":{"bar": "test1"}}}' -H "Content-Type: application/json" -X POST https://localhost/pulp/api/v2/repositories/test1/actions/associate/
pulp-admin rpm repo create --repo-id=zoo2 --relative-url=zoo2 --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/
pulp-admin rpm repo sync run --repo-id=zoo2
pulp-admin rpm repo create --repo-id=bar2 --relative-url=bar2 --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/
pulp-admin rpm repo sync run --repo-id=bar2
pulp-admin rpm repo create --repo-id=test3
curl -k -u admin:admin --cert ~/.pulp/user-cert.pem -d '{"source_repo_id":"zoo2","criteria":{"type_ids":["erratum"],"filters":{"unit":{"id":"RHEA-2012:0059"}}},"override_config":{"recursive_conservative":true,"additional_repos":{"bar2": "test3"}}}' -H "Content-Type: application/json" -X POST https://localhost/pulp/api/v2/repositories/test3/actions/associate/
pulp-admin repo list
test1 and test3 should have the same content, and number of packages.
Id: test1
Display Name: None
Description: None
Content Unit Counts:
Erratum: 1
Modulemd: 2
Modulemd Defaults: 2
Rpm: 2
Id: zoo2
Display Name: None
Description: None
Content Unit Counts:
Erratum: 6
Modulemd: 10
Modulemd Defaults: 3
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 32
Id: bar2
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 34
Id: test3
Display Name: None
Description: None
Content Unit Counts:
Erratum: 1
Modulemd: 2
Modulemd Defaults: 2
Rpm: 1
RPM packages present in the test1 repository:
['duck-0.7-1.noarch.rpm', 'kangaroo-0.3-1.noarch.rpm']
RPM packages present in the test3 repository:
['duck-0.7-1.noarch.rpm']
Pulp Version:
[root@pulpfipsserver ~]# rpm -qa | grep pulp
pulp-ostree-plugins-1.4.0-0.1.alpha.201909110504gitcc1c559.el7.noarch
python-pulp-bindings-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
pulp-docker-admin-extensions-3.4.0-0.1.alpha.201909110502git5f120a3.el7.noarch
pulp-deb-admin-extensions-1.11.0-0.1.alpha.201909110524git86a8a50.el7.noarch
python-pulp-streamer-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
python-pulp-puppet-common-2.21.0-0.1.alpha.201909110507gite3a1f28.el7.noarch
python-pulp-ostree-common-1.4.0-0.1.alpha.201909110504gitcc1c559.el7.noarch
python-pulp-python-common-2.1.0-0.1.alpha.201909110501git5e2aa35.el7.noarch
pulp-server-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
pulp-deb-plugins-1.11.0-0.1.alpha.201909110524git86a8a50.el7.noarch
pulp-docker-plugins-3.4.0-0.1.alpha.201909110502git5f120a3.el7.noarch
pulp-rpm-plugins-2.21.0-0.1.alpha.201909110511gitb9f593d.el7.noarch
pulp-puppet-tools-2.21.0-0.1.alpha.201909110507gite3a1f28.el7.noarch
pulp-admin-client-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201909110504gitcc1c559.el7.noarch
pulp-puppet-admin-extensions-2.21.0-0.1.alpha.201909110507gite3a1f28.el7.noarch
pulp-rpm-admin-extensions-2.21.0-0.1.alpha.201909110511gitb9f593d.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-docker-common-3.4.0-0.1.alpha.201909110502git5f120a3.el7.noarch
python-pulp-deb-common-1.11.0-0.1.alpha.201909110524git86a8a50.el7.noarch
pulp-selinux-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
python-pulp-repoauth-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
pulp-python-plugins-2.1.0-0.1.alpha.201909110501git5e2aa35.el7.noarch
pulp-puppet-plugins-2.21.0-0.1.alpha.201909110507gite3a1f28.el7.noarch
python-pulp-client-lib-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.201909110501git5e2aa35.el7.noarch
python-pulp-common-2.21.0-0.1.alpha.201909110516git8884300.el7.noarch
python-pulp-rpm-common-2.21.0-0.1.alpha.201909110511gitb9f593d.el7.noarch
python-pulp-oid_validation-2.21.0-0.1.alpha.201909110516git8884300.el7.no
OS Version:
[root@pulpfipsserver ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
Related issues
Updated by kersom over 4 years ago
- Related to Test #5242: Test copy using "additional_repos" to provide multiple source/destination repos via override added
Updated by kersom over 4 years ago
- Related to Story #5067: As a user, multiple source/target repositories can be used for recursive copy added
Added by kersom over 4 years ago
Updated by dalley over 4 years ago
- Priority changed from Normal to Low
- Triaged changed from No to Yes
Updated by rchan over 4 years ago
We will target this for completion in ~December 2019. Please bring up at open floor (triage on pulp-dev) should this start becoming impacting & needing to be addressed earlier.
Updated by dalley over 4 years ago
- Priority changed from Low to Normal
- Sprint Candidate changed from No to Yes
Updated by dalley over 4 years ago
- Assignee set to dalley
Based on us wanting to have this complete before the December, I'm going to add this to the sprint and start working on it.
Updated by dalley over 4 years ago
- Assignee deleted (
dalley) - Sprint deleted (
Sprint 63)
We should try to resolve this, but in the short term there is no capacity to do so, removing it from the sprint.
Updated by ggainey over 4 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ggainey
- Sprint set to Sprint 64
Reproduced using the steps in the description. As an experiment, followed up with these steps:
pulp-admin rpm repo create --repo-id=zoo4 --relative-url=zoo4 --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/
pulp-admin rpm repo sync run --repo-id=zoo4
pulp-admin rpm repo create --repo-id=bar4 --relative-url=bar4 --feed=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned-modified/
pulp-admin rpm repo sync run --repo-id=bar4
pulp-admin rpm repo create --repo-id=test5
curl -k -u admin:admin --cert ~/.pulp/user-cert.pem -d '{"source_repo_id":"zoo3","criteria":{"type_ids":["erratum"],"filters":{"unit":{"id":"RHEA-2012:0059"}}},"override_config":{"recursive_conservative":true,"additional_repos":{"bar4": "test5"}}}' -H "Content-Type: application/json" -X POST https://localhost/pulp/api/v2/repositories/test5/actions/associate/
pulp-admin repo list
and get the same results. Investigation continues.
[edited to remove erroneous test]
Updated by ggainey about 4 years ago
Interesting data :
This issue can be summed up as "create SRC1 with FEED1, SRC2 with FEED2, copy SRC1 and SRC2 into DEST1 with filter RHEA-2012:0059. Sometimes it works, sometimes DEST1 is missing an RPM."
The erratum we filter by in the reproducer lives in FEED1. It is associated with these NEVRAs:
- duck-None:0.7-1.noarch
- kangaroo-None:0.3-1.noarch
The RPM that sometimes doesn't show up in DEST1 is kangaroo.
FEED1 has exactly one version of kangaroo, kangaroo- 0.2-1 .noarch.rpm - which is NOT the one the erratum knows about.
kangaroo-None: 0.3-1 .noarch exists ONLY in FEED2.
This looks to me like FEED1 is a never-happen case - it contains an update that is referring to an RPM that DOES NOT EXIST in the repo.
This may not be the source of the problem - but it certainly is confusing the issue. Investigation continues.
Updated by ggainey about 4 years ago
Discussion w/tteresch points out that 'errata referring to NEVRAs that are not in the repository' is definitely A Thing - consider Fedora, that keeps first/latest NEVRA in a repo, and all the errata in between. So the behavior needs at the very least to be made consistent across runs.
Updated by ggainey about 4 years ago
Writing out more complete explanation of the data involved in this situation, because it's not only complex, it's back to "can that ever happen in a valid repository?" (Specifically -
The erratum in question (RHEA-2012:0059) in the source repo (rpm-with-modules-modified) is a 'modular' erratum - ie, it refers to modular-rpms (see https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules-modified/repodata/bcbf655559b9d850ccd381da095e13f4647b47182ae0696f63da3e5877ad80d8-updateinfo.xml.gz)
The two RPMs the erratum refers to are
<module name="kangaroo" stream="0" version="20180730223407" context="deadbeef" arch="noarch"/
<package arch="noarch" name="kangaroo" release="1" src="http://www.fedoraproject.org" version="0.3">
<filename>kangaroo-0.3-1.noarch.rpm</filename>
</package>
<module name="duck" stream="0" version="20180730233102" context="deadbeef" arch="noarch"/>@
<package arch="noarch" name="duck" release="1" src="http://www.fedoraproject.org" version="0.7">
<filename>duck-0.7-1.noarch.rpm</filename>
</package>
There are two 'kangaroo' modules:
document: modulemd
version: 2
data:
name: kangaroo
stream: 0
version: 20180730223407
context: deadbeef
arch: noarch
summary: Kangaroo 0.3 module
...
profiles:
default:
rpms:
- kangaroo
artifacts:
rpms:
- kangaroo-0:0.3-1.noarch
and
document: modulemd
version: 2
data:
name: kangaroo
stream: 0
version: 20180704111719
context: deadbeef
arch: noarch
summary: Kangaroo 0.2 module
...
profiles:
default:
rpms:
- kangaroo
artifacts:
rpms:
- kangaroo-0:0.2-1.noarch
kangaroo-stream0-version'3407, whic is the module referred to in RHEA-2012:0059, refers to kangaroo-0:0.3-1.noarch - which is not in the repository.
module duck-stream0-version'3102 has a module-level dependency on 'walrus' (yet another module in the primary repo)
The inconsistent-RPM results here are, we always copy the correct kangaroo and duck modules, we sometimes find the correct kangaroo-0:0.3, and sometimes we do not.
The module dependencies suggest we should also be finding the 'walrus' module (and its artifacts) as a dependency, since the 'duck' module is dependent on it.
There is...a lot going on here.
Final note for a Friday: I have not once seen the result fail while in the debugger (other than the 'we never see walrus' issue, of course)
Updated by ggainey about 4 years ago
- Description updated (diff)
- Severity changed from 2. Medium to 1. Low
- Groomed changed from No to Yes
Using the test-scripts here :
https://github.com/ggainey/pulp_startup/tree/master/5449
I can't recreate the bad-behavior in fedora or RHEL8. Only our test-fixtures show the problem, and the modules.yaml in there are...suboptimal (as in, is a never-happen case for current rules regarding modularity).
I am going to lower the priority of this issue and put it on the back burner.
Updated by ggainey about 4 years ago
- Status changed from ASSIGNED to CLOSED - WONTFIX
OK, since we can't recreate this in any 'real' data, and being in the debugger seems to make it Go Away, we're going to close this and its associated BZ as WONTFIX - reopen if we ever hit this in 'real' data
.
Add errata test for additional repos copy
Add errata test for additional repos copy.
https://pulp.plan.io/issues/5449 #5449
https://pulp.plan.io/issues/5242 #5242