Actions
Issue #7298
closed
Sync into the imported repository fails for comps content
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 80
Quarter:
Description
I used Centos8 BaseOS kicstart repo, hopefully any smaller repo with comps.xml will work.
To reproduce:
- Create a repo, sync it, and export it.
- Import a repo (I imported it to a different pulp instance, probably plcean and importing into the same one will work the same way).
- Sync from the same remote URL.
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: pulp: rq.worker:ERROR: ValueError: Cannot create repository version. More than one rpm.packagecategory content with the duplicate values for id., More than one rpm.packagegroup content with the duplicate values for id., More than one rpm.packageenvironment content with the duplicate values for id.
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: Traceback (most recent call last):
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py", line 934, in perform_job
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: rv = job.perform()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py", line 686, in perform
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: self._result = self._execute()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py", line 692, in _execute
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: return self.func(*self.args, **self.kwargs)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 208, in synchronize
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: dv.create()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: loop.run_until_complete(pipeline)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 790, in __exit__
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: repository.finalize_new_version(self)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py", line 173, in finalize_new_version
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: validate_repo_version(new_version)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py", line 140, in validate_repo_version
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: validate_duplicate_content(version)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py", line 110, in validate_duplicate_content
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: _("Cannot create repository version. {msg}").format(msg=", ".join(error_messages))
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: ValueError: Cannot create repository version. More than one rpm.packagecategory content with the duplicate values for id., More than one rpm.packagegroup content with the duplicate values for id., More than one rpm.packageenvironment content with the duplicate values for id.
Updated by ipanova@redhat.com over 2 years ago
check if repo version creation handles correctly duplicates for comps.xml
Updated by ggainey over 2 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ggainey
I haven't been able to recreate this against pulpcore/pulp_rpm master; have tried with fixtures and with CENTOS8-BASE kickstart repo.
The process I'm going thru is essentially the following:
CENTOS8_BASE_URL="http://centos.mirror.rafal.ca/8.2.2004/BaseOS/x86_64/kickstart/"
CENTOS8_NAME="centos8"
CENTOS8_BASE_NAME="centos8-base"
EXPORTER_NAME="centos8"
EXPORTER_URL="/pulp/api/v3/exporters/core/pulp/"
CENTOS_NAME='new-centos8'
IMPORT_NAME="test-centos8"
IMPORTER_URL="/pulp/api/v3/importers/core/pulp/"
MAPPING='{"centos8-base": "new-centos8-base"}'
# create repo/remote/sync
CENTOS8_BASE_HREF=$(http POST :/pulp/api/v3/repositories/rpm/rpm/ name=${CENTOS8_BASE_NAME} | jq -r '.pulp_href')
http POST :/pulp/api/v3/remotes/rpm/rpm/ name=$CENTOS8_BASE_NAME url=$CENTOS8_BASE_URL policy='immediate'
BASE_REMOTE_HREF=$(http :/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${CENTOS8_BASE_NAME}\") | .pulp_href")
http POST :$CENTOS8_BASE_HREF'sync/' remote=$BASE_REMOTE_HREF
# create exporter/export
EXPORTER_HREF=$(http POST :$EXPORTER_URL name="${EXPORTER_NAME}"-exporter repositories:=[\"${CENTOS8_BASE_HREF}\"] path=/tmp/exports/) "
http POST :${EXPORTER_HREF}exports/
# reset pulp
pclean
prestart
# create destination repo/importer
CENTOS8_BASE_HREF=$(http POST :/pulp/api/v3/repositories/rpm/rpm/ name="${CENTOS_NAME}-base" | jq -r '.pulp_href')
IMPORT_HREF=$(http POST :$IMPORTER_URL name="${IMPORT_NAME}"-importer repo_mapping:="${MAPPING}")
# http POST :${IMPORT_HREF}imports/ toc=/tmp/exports/<EXPORT-TOC-HERE>
# sync once import has finished
http POST :/pulp/api/v3/remotes/rpm/rpm/ name=$CENTOS8_BASE_NAME url=$CENTOS8_BASE_URL policy='immediate'
BASE_REMOTE_HREF=$(http :/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${CENTOS8_BASE_NAME}\") | .pulp_href")
http POST :$CENTOS8_BASE_HREF'sync/' remote=$BASE_REMOTE_HREF
ttereshc : do you have the exact repo-URL you used when this failed?
The UpdateCollection changes happened on 12-AUG, but don't appear to have any way to have affected PackageCategoryResource. The kickstart/distr-tree work daviddavis did for import-export happened on 27-JUL, and should therefore be in place already when your test failed - unless the imported-into pulp-instance was older than that?
Updated by ggainey over 2 years ago
Notes from IRC conversation:
<ttereshc> ggainey, interesting, here is what I did differently, but I don't think it should affect anything :/ I used the link form our functional tests, I didn't use mapping (I think), I used a separate instance of pulp, I probably synced with on_demand policy (after import).
<ttereshc> you can close as works for me and we'll see. I'm not sure if I'll have time to test it this week
<ggainey> ok, cool - I'll add that to the issue, and retry - maybe this is an on-demand thing. If it still works, maybe I'll close; will mull it over and see if maybe I'm missing a key step somewhere
<ggainey> I'll make sure I'm using the same repo first, maybe it's a Data Thing
Updated by ggainey over 2 years ago
- Status changed from ASSIGNED to CLOSED - WORKSFORME
I switched to using http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/kickstart/ for the remote, and also switched the resync to on_demand, and still can't make it fail. I'm going to close this as "works for me" - if you get an opportunity to retest and can make it fail again, by all means add detail here and reopen!
Actions
.