Project

Profile

Help

Issue #8116

closed

pulp-import relying on 'next_version' in ways it can't, causing import to fail.

Added by paji@redhat.com over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

  • Create a repo
  • upload a package -> this should generate version 1
  • remove a package -> this should generate version 2
  • optionally upload another package -> this should generate version 3
  • Now export version 1
  • copy and extract the generated tar gz
$ cat pulpcore.app.modelresource.RepositoryResource.json
[{"pulp_type": "rpm.rpm", "name": "life-531063", "description": "", "next_version": 4, "remote": ""}]
  • do an ls
$ ls|grep repo
repository-life-531063_1

so notice that next version is pointing to 4 while the directory name ends with _1

  • Now wipe the db
  • Create a new yum repository called foo
  • Create an importer with the mapping -> {'foo':"life-531063"}
  • Try to import the exported tgz
  • The operation should fail
  • You should see an error along the lines of
an 18 19:57:17  pulpcore-worker-2[1770]: pulp: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmp47swboli/repository-life-531063_3/pulp_rpm.app.modelresource.PackageResource.json.
Jan 18 19:57:17  pulpcore-worker-2[1770]: pulp: rq.worker:ERROR: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp47swboli/repository-life-531063_3/pulp_rpm.app.modelresource.PackageResource.json'
Jan 18 19:57:17  pulpcore-worker-2[1770]: Traceback (most recent call last):
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
Jan 18 19:57:17  pulpcore-worker-2[1770]: rv = job.perform()
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 645, in perform
Jan 18 19:57:17  pulpcore-worker-2[1770]: self._result = self._execute()
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 651, in _execute
Jan 18 19:57:17  pulpcore-worker-2[1770]: return self.func(*self.args, **self.kwargs)
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 150, in import_repository_version
Jan 18 19:57:17  pulpcore-worker-2[1770]: a_result = _import_file(os.path.join(rv_path, filename), res_class)
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 57, in _import_file
Jan 18 19:57:17  pulpcore-worker-2[1770]: with open(fpath, "r") as json_file:
Jan 18 19:57:17  pulpcore-worker-2[1770]: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp47swboli/repository-life-531063_3/pulp_rpm.app.modelresource.PackageResource.json'
Jan 18 19:57:17  pulpcore-worker-2[1770]: Traceback (most recent call last):
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
Jan 18 19:57:17  pulpcore-worker-2[1770]: rv = job.perform()
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 645, in perform
Jan 18 19:57:17  pulpcore-worker-2[1770]: self._result = self._execute()
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 651, in _execute
Jan 18 19:57:17  pulpcore-worker-2[1770]: return self.func(*self.args, **self.kwargs)
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 150, in import_repository_version
Jan 18 19:57:17  pulpcore-worker-2[1770]: a_result = _import_file(os.path.join(rv_path, filename), res_class)
Jan 18 19:57:17  pulpcore-worker-2[1770]: File "/usr/local/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 57, in _import_file
Jan 18 19:57:17  pulpcore-worker-2[1770]: with open(fpath, "r") as json_file:
Jan 18 19:57:17  pulpcore-worker-2[1770]: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp47swboli/repository-life-531063_3/pulp_rpm.app.modelresource.PackageResource.json'

Not sure if its the importer's fault or exporter's. But we definitely need to be able to handle this.


Files

export-dump.tgz (81.8 KB) export-dump.tgz paji@redhat.com, 01/18/2021 10:44 PM
Actions #1

Updated by paji@redhat.com over 3 years ago

Also uploading my tgz

Actions #2

Updated by fao89 over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 88
Actions #3

Updated by fao89 over 3 years ago

  • Triaged changed from Yes to No
  • Sprint deleted (Sprint 88)
Actions #4

Updated by fao89 about 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 89
Actions #5

Updated by daviddavis about 3 years ago

  • Tags Katello added
Actions #6

Updated by ggainey about 3 years ago

export is just recording what next_version is on the upstream machine. The problem is that import can't be relying on that for...anything, really, given how selective one can be on-export. Need to rethink the import-algorithm at this point.

Actions #7

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 89 to Sprint 90
Actions #8

Updated by daviddavis about 3 years ago

  • Sprint/Milestone set to 3.12.0
  • Sprint deleted (Sprint 90)
Actions #9

Updated by ggainey about 3 years ago

  • Subject changed from Export is generating the incorrect 'next_version' causing import to fail. to pulp-import relying on 'next_version' in ways it can't, causing import to fail.

Updated subject to better reflect the problem. Also, to reproduce, no need to wipe the db, you can just attempt to import into a clean/new repository.

Actions #10

Updated by ggainey about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
Actions #11

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST

Added by ggainey about 3 years ago

Revision b10ac5e8 | View on GitHub

Taught PulpImport to not rely on next_version from upstream.

Also taught import and export tests to share more utility code.

fixes #8116

Actions #12

Updated by ggainey about 3 years ago

  • Status changed from POST to MODIFIED
Actions #13

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF