Project

Profile

Help

Story #7538

As a user, I can provide an option to log and skip missing or corrupt files during artifact creation

Added by jsherril@redhat.com about 1 month ago. Updated 13 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 84
Quarter:

Description

A user that is using pulp2 may have 100s of thousands of rpms synced or uploaded. Right now if a single rpm is missing or corrupt the entire migration process will fail. They may not care about this one particular rpm (it could be the 3rd out of 50 kernels released) for example, and otherwise are oblivious to this rpm being corrupt or missing.

We should provide an option to skip and warn the user in this situation. Here's the traceback:

    Sep 21 15:30:02 content-migration pulpcore-worker-2: Traceback (most recent call last):
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
    Sep 21 15:30:02 content-migration pulpcore-worker-2: rv = job.perform()
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 657, in perform
    Sep 21 15:30:02 content-migration pulpcore-worker-2: self._result = self._execute()
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
    Sep 21 15:30:02 content-migration pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 141, in migrate_from_pulp2
    Sep 21 15:30:02 content-migration pulpcore-worker-2: migrate_content(plan)
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
    Sep 21 15:30:02 content-migration pulpcore-worker-2: plugin.migrator.migrate_content_to_pulp3()
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 142, in migrate_content_to_pulp3
    Sep 21 15:30:02 content-migration pulpcore-worker-2: loop.run_until_complete(dm.create())
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    Sep 21 15:30:02 content-migration pulpcore-worker-2: return future.result()
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 86, in create
    Sep 21 15:30:02 content-migration pulpcore-worker-2: await pipeline
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    Sep 21 15:30:02 content-migration pulpcore-worker-2: await asyncio.gather(*futures)
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    Sep 21 15:30:02 content-migration pulpcore-worker-2: await self.run()
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 191, in run
    Sep 21 15:30:02 content-migration pulpcore-worker-2: self.migrate_to_pulp3(cmodel, ctype)
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 376, in migrate_to_pulp3
    Sep 21 15:30:02 content-migration pulpcore-worker-2: downloaded=pulp2content.downloaded
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/local/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 147, in create_artifact
    Sep 21 15:30:02 content-migration pulpcore-worker-2: expected_size=expected_size)
    Sep 21 15:30:02 content-migration pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 231, in init_and_validate
    Sep 21 15:30:02 content-migration pulpcore-worker-2: with open(file, "rb") as f:
    Sep 21 15:30:02 content-migration pulpcore-worker-2: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/content/units/rpm/b1/4d69e4e6065886d8ed2c9976782e02113ce7146cc54c7f7ceece50eb5f31e5/libsss_certmap-devel-1.16.4-37.el7.s390.rpm'

History

#1 Updated by ttereshc 29 days ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 82

#2 Updated by ttereshc 29 days ago

  • Tracker changed from Issue to Story
  • Subject changed from during migration, provide option to log and skip missing or corrupt files during artifcate creation to As a user, I can provide an option to log and skip missing or corrupt files during artifact creation
  • % Done set to 0
  • Severity deleted (2. Medium)
  • Triaged deleted (Yes)

#3 Updated by rchan 26 days ago

  • Sprint changed from Sprint 82 to Sprint 83

#4 Updated by ttereshc 13 days ago

This should cover every single content we rely on for migration to be present on a disk.

  • RPM/SRPM (if downloaded=True)
  • .treeinfo file for DistributionTrees
  • module and module-default snippets (I think we use files with snippets to migrate it)

Feel free to update this list if anyone recalls more.

#5 Updated by rchan 13 days ago

  • Sprint changed from Sprint 83 to Sprint 84

Please register to edit this issue

Also available in: Atom PDF