Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #4617

closed

Migration of modular content can fail (upgrade from <=2.16 to 2.17+)

Added by kfiresmith over 5 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
dkliban@redhat.com
Sprint/Milestone:
Pulp - 2.19.1
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.19.1
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 51
Quarter:

Description

Hello folks,
Our upgrade of a (albeit smaller but otherwise identical) beta Pulp server went just fine, so this morning I upgraded production which went fine until the database migration.

The migration failed very early at 'Applying migration pulp_rpm.plugins.migrations.0043_add_modulemd_modulemd-defaults failed.' The underlying traceback shows 'IOError: Not a gzipped file'.

Full paste lives here: https://paste.fedoraproject.org/paste/wEIu5a3Tf8OejEzMNsXN9w

Packages installed are:
$rpm -qa | egrep 'pulp|qpid|gofer|celery|mongo' | sort
mongodb-2.6.12-6.el7.x86_64
mongodb-server-2.6.12-6.el7.x86_64
pulp-admin-client-2.18.1-2.el7.noarch
pulp-docker-admin-extensions-3.2.2-1.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-puppet-admin-extensions-2.18.1-1.el7.noarch
pulp-puppet-plugins-2.18.1-1.el7.noarch
pulp-rpm-admin-extensions-2.18.1-1.el7.noarch
pulp-rpm-plugins-2.18.1-1.el7.noarch
pulp-selinux-2.18.1-2.el7.noarch
pulp-server-2.18.1-2.el7.noarch
python2-celery-4.0.2-7.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-qmf-1.39.0-1.el7.x86_64
python-gofer-2.12.5-1.el7.noarch
python-gofer-qpid-2.12.5-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-mongoengine-0.10.5-1.el7.noarch
python-pulp-bindings-2.18.1-2.el7.noarch
python-pulp-client-lib-2.18.1-2.el7.noarch
python-pulp-common-2.18.1-2.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
python-pulp-oid_validation-2.18.1-2.el7.noarch
python-pulp-puppet-common-2.18.1-1.el7.noarch
python-pulp-repoauth-2.18.1-2.el7.noarch
python-pulp-rpm-common-2.18.1-1.el7.noarch
python-pymongo-3.2-2.el7.x86_64
python-pymongo-gridfs-3.2-2.el7.x86_64
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-proton-c-0.26.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch

I'm a bit stumped because I'm not a developer, or a database wiz, so any help in getting our prod instance back up would be very much appreciated! Thanks!

Summary

Some repos unexpectedly contain uncompressed modules.yaml metadata.
Pulp should not fail but try to work with uncompressed data instead.

Applying pulp_rpm.plugins.migrations version 43
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest_list = pulp_docker.plugins.models:ManifestList
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: modulemd_defaults = pulp_rpm.plugins.db.models:ModulemdDefaults
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: modulemd = pulp_rpm.plugins.db.models:Modulemd
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
Loaded plugin puppet_distributor for types: puppet_module
Loaded plugin puppet_file_distributor for types: puppet_module
Loaded plugin puppet_install_distributor for types: puppet_module
Loaded plugin docker_distributor_web for types: docker_image,docker_blob,docker_manifest,docker_tag
Loaded plugin docker_rsync_distributor for types: docker_image,docker_blob,docker_manifest,docker_tag
Loaded plugin docker_distributor_export for types: docker_image
Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin rpm_rsync_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin iso_distributor for types: iso
Loaded plugin iso_rsync_distributor for types: iso
Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin puppet_importer for types: puppet_module
Loaded plugin docker_importer for types: docker_blob,docker_image,docker_manifest,docker_manifest_list,docker_tag
Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment,package_langpacks,modulemd,modulemd_defaults
Loaded plugin iso_importer for types: iso
Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Loaded plugin yum_profiler for types: rpm,erratum,modulemd
Loaded plugin yum for types: rpm
Loaded plugin rhui for types: rpm
Applying migration pulp_rpm.plugins.migrations.0043_add_modulemd_modulemd-defaults failed.

Halting migrations due to a migration failure.
Not a gzipped file
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 239, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 306, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 135, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 189, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py", line 341, in migrate
    modulemds, defaults = load(fp, working_dir)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py", line 317, in load
    bfr = metadata.read(1024000)
  File "/usr/lib64/python2.7/gzip.py", line 261, in read
    self._read(readsize)
  File "/usr/lib64/python2.7/gzip.py", line 296, in _read
    self._read_gzip_header()
  File "/usr/lib64/python2.7/gzip.py", line 190, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file

Related issues

Has duplicate RPM Support - Issue #4619: Migration of modular content can fail (upgrade from <=2.16 to 2.17+)CLOSED - DUPLICATEActions
Copied to RPM Support - Test #4729: Migration of modular content can fail (upgrade from <=2.16 to 2.17+)CLOSED - WONTFIXActions
Actions
Copy link
 #1

Updated by dkliban@redhat.com over 5 years ago

Looks like the problem was with some of the content on the CDN. Here is a patch that fixes the problem. We'll include this fix in 2.19.1.

diff --git a/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py b/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
index 7736a5a8..bb2bcb55 100644
--- a/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
+++ b/plugins/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py
@@ -336,14 +336,26 @@ def migrate(*args, **kwargs):
     repos_to_republish = set()
     for file in metadatafiles:
         repository = Repository.objects.get(repo_id=file.repo_id)
-        with gzip.open(file._storage_path, 'r') as fp:
-            working_dir = "/var/cache/pulp"
-            modulemds, defaults = load(fp, working_dir)
-            add_modulemds(repository, modulemds, working_dir)
-            add_defaults(repository, defaults, working_dir)
-            repository_controller.disassociate_units(repository, [file])
-            repository_controller.rebuild_content_unit_counts(repository)
-            repos_to_republish.add(file.repo_id)
+        try:
+            with gzip.open(file._storage_path, 'r') as fp:
+                working_dir = "/var/cache/pulp"
+                modulemds, defaults = load(fp, working_dir)
+                add_modulemds(repository, modulemds, working_dir)
+                add_defaults(repository, defaults, working_dir)
+                repository_controller.disassociate_units(repository, [file])
+                repository_controller.rebuild_content_unit_counts(repository)
+                repos_to_republish.add(file.repo_id)
+        except IOError:
+            _logger.info(file.unit_key)
+            with open(file._storage_path, 'r') as fp:
+                working_dir = "/var/cache/pulp"
+                modulemds, defaults = load(fp, working_dir)
+                add_modulemds(repository, modulemds, working_dir)
+                add_defaults(repository, defaults, working_dir)
+                repository_controller.disassociate_units(repository, [file])
+                repository_controller.rebuild_content_unit_counts(repository)
+                repos_to_republish.add(file.repo_id)
+
     if repos_to_republish:
         with open('/var/lib/pulp/0043_add_modulemd_modulemd-defaults.txt', 'w') as f:
             f.write(str(list(repos_to_republish)))
Actions
Copy link
 #2

Updated by ttereshc over 5 years ago

  • Project changed from 31 to RPM Support
  • Subject changed from Pulp failed upgrade on production 2.16.3 --> 2.18.1 (IO Error: not a gzipped file) to Migration of modular content can fail (upgrade from <=2.16 to 2.17+)
  • Description updated (diff)
Actions
Copy link
 #3

Updated by ttereshc over 5 years ago

  • Has duplicate Issue #4619: Migration of modular content can fail (upgrade from <=2.16 to 2.17+) added
Actions
Copy link
 #4

Updated by kfiresmith over 5 years ago

Thanks very much DKliban. I'm back in business. For what it's worth for the rest of Pulp 2.x's lifespan, trapping exceptions as done in the patch with some logging goodness to stdout at bail-time would be very much welcome to all those who may encounter such a traceback.

Actions
Copy link
 #6

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

Added by dkliban@redhat.com over 5 years ago

Revision 4e4a6b4a | View on GitHub

Problem: migration fails for uncompressed modules files

Solution: handle IOError and read the uncompressed file

fixes: #4617 https://pulp.plan.io/issues/4617

Actions
Copy link
 #7

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions
Copy link
 #8

Updated by ttereshc over 5 years ago

  • Sprint set to Sprint 51
Actions
Copy link
 #9

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Copy link
 #10

Updated by bherring over 5 years ago

  • Copied to Test #4729: Migration of modular content can fail (upgrade from <=2.16 to 2.17+) added
Actions
Copy link
 #11

Updated by dkliban@redhat.com over 5 years ago

  • Platform Release set to 2.19.1
Actions
Copy link
 #12

Updated by dkliban@redhat.com over 5 years ago

  • Sprint/Milestone set to 2.19.1

Added by dkliban@redhat.com over 5 years ago

Revision dc9cb3be | View on GitHub

Problem: migration fails for uncompressed modules files

Solution: handle IOError and read the uncompressed file

fixes: #4617 https://pulp.plan.io/issues/4617 (cherry picked from commit 4e4a6b4ad4514df16f07ce43fd2e197695f4cbf5)

Actions
Copy link
 #14

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #15

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF