Issue #4741
closed
Migration of modular content fails for very old Fedora repos (upgrade from <=2.16 to 2.17+)
Description
The error is seen for repos with f26 content, those repos don't exist any more:
http://modularity.fedorainfracloud.org/modularity/compat/fedora-os-stream/
http://modularity.fedorainfracloud.org/modularity/compat/fedora-modular/
ValidationError (Modulemd:938e503c-9913-48e0-8ebd-c68791db80d0) (Field is required: ['context'])
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 239, in main
return _auto_manage_db(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 306, in _auto_manage_db
migrate_database(options)
File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 135, in migrate_database
update_current_version=not options.test)
File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 189, in apply_migration
migration.migrate()
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0500_upstream_0043_add_modulemd_modulemd-defaults.py", line 343, in migrate
add_modulemds(repository, modulemds, working_dir)
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0500_upstream_0043_add_modulemd_modulemd-defaults.py", line 218, in add_modulemds
add_modulemd(repository, modulemd, model, working_dir)
File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0500_upstream_0043_add_modulemd_modulemd-defaults.py", line 164, in add_modulemd
model.save_and_import_content(path)
File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 921, in save_and_import_content
self.save()
File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 304, in save
self.validate(clean=clean)
File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 413, in validate
raise ValidationError(message, errors=errors)
ValidationError: ValidationError (Modulemd:938e503c-9913-48e0-8ebd-c68791db80d0) (Field is required: ['context'])
Modules in those repos have no context which is a part of unit_key for a modulemd content so it cant' be imported into Pulp.
Possible solutions:
- provide list of content for user to manually remove before migration
- skip this broken content and proceed with migration
- skipping it will mean that this content will be removed from a repo and become orphaned.
- anything else? I'm hesitant to keep bad modules.yaml files as unparsed yum_repo_metadata_file content for consistency reasons.
No data how many users might be affected. Hopefully not many since this issue didn't blow up earlier.
Updated by ttereshc over 5 years ago
This patch is for skipping broken content.
diff --git a/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py b/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
index 20612c57..17bd9fd4 100644
--- a/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
+++ b/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
@@ -6,7 +6,7 @@ from uuid import uuid4
from hashlib import sha256
import bson
-from mongoengine import Q, NotUniqueError
+from mongoengine import Q, NotUniqueError, ValidationError
from pulp.plugins.loader import api as plugin_api
from pulp.server.controllers import repository as repository_controller
@@ -164,6 +164,11 @@ def add_modulemd(repository, modulemd, model, working_dir):
model.save_and_import_content(path)
except NotUniqueError:
model = Modulemd.objects.get(**model.unit_key)
+ except ValidationError as e:
+ _logger.warn("Modulemd content import failure. Repo: %s, content: %s, "
+ "error: %s" % (repository.repo_id, model.unit_key, e))
+ return
+
repository_controller.associate_single_unit(repository, model)
NOTE: it prints the message for every bad modulemd and user sees it in the output of pulp-manage-db.
Updated by dkliban@redhat.com over 5 years ago
We should just skip the content and let it get orphaned. We should provide release notes that clearly state that content from F26 will stop working after the migration runs.
Added by ttereshc over 5 years ago
Updated by ttereshc over 5 years ago
- Status changed from NEW to POST
- Triaged changed from No to Yes
- Sprint set to Sprint 52
Updated by ttereshc over 5 years ago
- Status changed from POST to MODIFIED
Applied in changeset e43c39ee16d1787df587024cd8dfb077b11a8f8d.
Added by ttereshc over 5 years ago
Revision 78a47006 | View on GitHub
Skip malformed modular content during migration
Some users might have old experimental modular content in Pulp.
Such content doesn't contain all the necessary info to be imported
in Pulp. E.g. no context which is a part of unit_key.
closes #4741 https://pulp.plan.io/issues/4741
(cherry picked from commit e43c39ee16d1787df587024cd8dfb077b11a8f8d)
Updated by ttereshc over 5 years ago
Applied in changeset 78a470060a84c6be5c19fc90800f1f82dcfcedad.
Updated by dkliban@redhat.com over 5 years ago
- Status changed from MODIFIED to 5
Updated by dkliban@redhat.com over 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Skip malformed modular content during migration
Some users might have old experimental modular content in Pulp. Such content doesn't contain all the necessary info to be imported in Pulp. E.g. no
contextwhich is a part of unit_key.closes #4741 https://pulp.plan.io/issues/4741