Issue #7744: pulp2 data is deleted during migration - Migration Plugin - Pulp

Actions

Send by e-mail Copy link

Issue #7744

closed

pulp2 data is deleted during migration

Added by adam.winberg@smhi.se over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - DUPLICATE

Priority:

Normal

Assignee:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Platform Release:

OS:

Triaged:

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

While running migration I run into error message like:

Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     rv = job.perform()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     self._result = self._execute()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     return self.func(*self.args, **self.kwargs)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 139, in migrate_from_pulp2
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     migrate_content(plan)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     plugin.migrator.migrate_content_to_pulp3()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     loop.run_until_complete(dm.create())
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     return future.result()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 88, in create
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     await pipeline
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     await asyncio.gather(*futures)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     await self.run()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 169, in run
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     self.migrate_to_pulp3(cmodel, ctype)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 365, in migrate_to_pulp3
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     downloaded=pulp2content.downloaded
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 125, in create_artifact
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     expected_size=expected_size)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:   File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 266, in init_and_validate
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]:     with open(file, "rb") as f:
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/content/units/rpm/a9/8836abda4102abc7cf2f9f179a71f77cd3132599117a695380ec2568382548/libusnic_verbs-utils-1.1.0.237-2.el7.x86_64.rpm'

The file is indeed missing, but it is present in my pulp2 db and marked 'downloaded'. So to fix it I run a repo sync in pulp2 with '--force-full' and then rerun the migration from scratch. Only to once again run into the same error for some other file. And after doing this a couple of times the same filenames start to re-appear, files I have made certain are indeed present before starting the migration.

So it seems that the migration is actually deleting files from my pulp2 content dir, and deleting it before it tries to import it. My understanding was that the migration can be run in parallell while still using pulp2 in production, which makes this behaviour very odd. Have I misunderstood something?

This is on an rpm based installation on RHEL8:

python3-pulpcore-3.7.1-3.el8.noarch
python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulp-2to3-migration-0.5.0-1.el8.noarch

Related issues

Actions

Copy link

Updated by adam.winberg@smhi.se over 3 years ago

It doesn't seem like it's just a couple of files being deleted either, when i discover the problem and run 'sync' on pulp2 with 'force-full' there are quite a few rpm's being downloaded, some examples from different repos:

RPMs:       2848/2848 items
...
RPMs:       2/2 items
...
RPMs:       2609/2609 items

Actions

Copy link

Updated by dalley over 3 years ago

Status changed from NEW to CLOSED - DUPLICATE

Nope, you understand perfectly, this is a serious issue. Luckily I just found the cause a few hours ago.

Actions

Copy link

Updated by dalley over 3 years ago

Is duplicate of Issue #7676: django-cleanup can silently delete files being used by existing artifacts in some circumstances added

Actions

Copy link

Updated by dalley over 3 years ago

Status changed from CLOSED - DUPLICATE to NEW

While it's pretty likely to be the same issue, marking it duplicate was probably premature.

I'm attempting to reproduce it now. Which repositories do you have synced and/or are trying to migrate? It might even be helpful if we could duplicate your setup and test that specific configuration in-depth, both to help you avoid future issues, and to make discovering them easier for us. Thanks again for helping us with finding and troubleshooting these issues.

If is the same issue as the one I linked earlier, this should resolve it. If you'd rather wait for me to try to reproduce it first that's totally fine:

edit /usr/lib/python $$version$$/site-packages/pulpcore/app/settings.py (swap out $$version$$ for whatever Python is on your system, probably 3.6
remove or comment out the line "django_cleanup.apps.CleanupConfig" in the "INSTALLED_APPS" section
restart, repair pulp 2, remigrate

Actions

Copy link

Updated by dalley over 3 years ago

I was able to reproduce. Will now test the w/ the patch applied.

Actions

Copy link

Updated by dalley over 3 years ago

Status changed from NEW to CLOSED - DUPLICATE

It's the same issue, the patch resolves this as well, yay.

Will be included in pulpcore 3.7.3, and the filelists fix should go into migration plugin 0.5.1. Not sure when those will pop into the repository, probably by the end of the week.

Actions

Copy link