Issue #7744
closedpulp2 data is deleted during migration
Description
While running migration I run into error message like:
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: rv = job.perform()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: self._result = self._execute()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: return self.func(*self.args, **self.kwargs)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 139, in migrate_from_pulp2
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: migrate_content(plan)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 36, in migrate_content
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: plugin.migrator.migrate_content_to_pulp3()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 140, in migrate_content_to_pulp3
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: loop.run_until_complete(dm.create())
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: return future.result()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 88, in create
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: await pipeline
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: await asyncio.gather(*futures)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: await self.run()
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 169, in run
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: self.migrate_to_pulp3(cmodel, ctype)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 365, in migrate_to_pulp3
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: downloaded=pulp2content.downloaded
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 125, in create_artifact
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: expected_size=expected_size)
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 266, in init_and_validate
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: with open(file, "rb") as f:
Oct 26 06:40:46 lxserv2285.smhi.se rq[3185274]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/content/units/rpm/a9/8836abda4102abc7cf2f9f179a71f77cd3132599117a695380ec2568382548/libusnic_verbs-utils-1.1.0.237-2.el7.x86_64.rpm'
The file is indeed missing, but it is present in my pulp2 db and marked 'downloaded'. So to fix it I run a repo sync in pulp2 with '--force-full' and then rerun the migration from scratch. Only to once again run into the same error for some other file. And after doing this a couple of times the same filenames start to re-appear, files I have made certain are indeed present before starting the migration.
So it seems that the migration is actually deleting files from my pulp2 content dir, and deleting it before it tries to import it. My understanding was that the migration can be run in parallell while still using pulp2 in production, which makes this behaviour very odd. Have I misunderstood something?
This is on an rpm based installation on RHEL8:
python3-pulpcore-3.7.1-3.el8.noarch
python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulp-2to3-migration-0.5.0-1.el8.noarch
Related issues
Updated by adam.winberg@smhi.se almost 4 years ago
It doesn't seem like it's just a couple of files being deleted either, when i discover the problem and run 'sync' on pulp2 with 'force-full' there are quite a few rpm's being downloaded, some examples from different repos:
RPMs: 2848/2848 items
...
RPMs: 2/2 items
...
RPMs: 2609/2609 items
Updated by dalley almost 4 years ago
- Status changed from NEW to CLOSED - DUPLICATE
Nope, you understand perfectly, this is a serious issue. Luckily I just found the cause a few hours ago.
Updated by dalley almost 4 years ago
- Is duplicate of Issue #7676: django-cleanup can silently delete files being used by existing artifacts in some circumstances added
Updated by dalley almost 4 years ago
- Status changed from CLOSED - DUPLICATE to NEW
While it's pretty likely to be the same issue, marking it duplicate was probably premature.
I'm attempting to reproduce it now. Which repositories do you have synced and/or are trying to migrate? It might even be helpful if we could duplicate your setup and test that specific configuration in-depth, both to help you avoid future issues, and to make discovering them easier for us. Thanks again for helping us with finding and troubleshooting these issues.
If is the same issue as the one I linked earlier, this should resolve it. If you'd rather wait for me to try to reproduce it first that's totally fine:
- edit /usr/lib/python $$version$$/site-packages/pulpcore/app/settings.py (swap out $$version$$ for whatever Python is on your system, probably 3.6
- remove or comment out the line "django_cleanup.apps.CleanupConfig" in the "INSTALLED_APPS" section
- restart, repair pulp 2, remigrate
Updated by dalley almost 4 years ago
I was able to reproduce. Will now test the w/ the patch applied.
Updated by dalley almost 4 years ago
- Status changed from NEW to CLOSED - DUPLICATE
It's the same issue, the patch resolves this as well, yay.
Will be included in pulpcore 3.7.3, and the filelists fix should go into migration plugin 0.5.1. Not sure when those will pop into the repository, probably by the end of the week.
Updated by adam.winberg@smhi.se almost 4 years ago
awesome work!
Will update and test as soon as packages are available.
Updated by adam.winberg@smhi.se almost 4 years ago
Status update, still waiting on the 3.7.3 package in the foreman repo.
Updated by adam.winberg@smhi.se almost 4 years ago
I have now been able to test with 3.7.3 and this is indeed solved.