RPM Support

Hello folks,
Our upgrade of a (albeit smaller but otherwise identical) beta Pulp server went just fine, so this morning I upgraded production which went fine until the database migration.

The migration failed very early at 'Applying migration pulp_rpm.plugins.migrations.0043_add_modulemd_modulemd-defaults failed.' The underlying traceback shows 'IOError: Not a gzipped file'.

Full paste lives here: https://paste.fedoraproject.org/paste/wEIu5a3Tf8OejEzMNsXN9w

Packages installed are:
$rpm -qa | egrep 'pulp|qpid|gofer|celery|mongo' | sort
mongodb-2.6.12-6.el7.x86_64
mongodb-server-2.6.12-6.el7.x86_64
pulp-admin-client-2.18.1-2.el7.noarch
pulp-docker-admin-extensions-3.2.2-1.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-puppet-admin-extensions-2.18.1-1.el7.noarch
pulp-puppet-plugins-2.18.1-1.el7.noarch
pulp-rpm-admin-extensions-2.18.1-1.el7.noarch
pulp-rpm-plugins-2.18.1-1.el7.noarch
pulp-selinux-2.18.1-2.el7.noarch
pulp-server-2.18.1-2.el7.noarch
python2-celery-4.0.2-7.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-qmf-1.39.0-1.el7.x86_64
python-gofer-2.12.5-1.el7.noarch
python-gofer-qpid-2.12.5-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-mongoengine-0.10.5-1.el7.noarch
python-pulp-bindings-2.18.1-2.el7.noarch
python-pulp-client-lib-2.18.1-2.el7.noarch
python-pulp-common-2.18.1-2.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
python-pulp-oid_validation-2.18.1-2.el7.noarch
python-pulp-puppet-common-2.18.1-1.el7.noarch
python-pulp-repoauth-2.18.1-2.el7.noarch
python-pulp-rpm-common-2.18.1-1.el7.noarch
python-pymongo-3.2-2.el7.x86_64
python-pymongo-gridfs-3.2-2.el7.x86_64
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-proton-c-0.26.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch

I'm a bit stumped because I'm not a developer, or a database wiz, so any help in getting our prod instance back up would be very much appreciated! Thanks!

Summary¶

Some repos unexpectedly contain uncompressed modules.yaml metadata.
Pulp should not fail but try to work with uncompressed data instead.

Applying pulp_rpm.plugins.migrations version 43
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest_list = pulp_docker.plugins.models:ManifestList
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: modulemd_defaults = pulp_rpm.plugins.db.models:ModulemdDefaults
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: modulemd = pulp_rpm.plugins.db.models:Modulemd
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
Loaded plugin puppet_distributor for types: puppet_module
Loaded plugin puppet_file_distributor for types: puppet_module
Loaded plugin puppet_install_distributor for types: puppet_module
Loaded plugin docker_distributor_web for types: docker_image,docker_blob,docker_manifest,docker_tag
Loaded plugin docker_rsync_distributor for types: docker_image,docker_blob,docker_manifest,docker_tag
Loaded plugin docker_distributor_export for types: docker_image
Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin rpm_rsync_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin iso_distributor for types: iso
Loaded plugin iso_rsync_distributor for types: iso
Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin puppet_importer for types: puppet_module
Loaded plugin docker_importer for types: docker_blob,docker_image,docker_manifest,docker_manifest_list,docker_tag
Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment,package_langpacks,modulemd,modulemd_defaults
Loaded plugin iso_importer for types: iso
Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Loaded plugin yum_profiler for types: rpm,erratum,modulemd
Loaded plugin yum for types: rpm
Loaded plugin rhui for types: rpm
Applying migration pulp_rpm.plugins.migrations.0043_add_modulemd_modulemd-defaults failed.

Halting migrations due to a migration failure.
Not a gzipped file
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 239, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 306, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 135, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 189, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py", line 341, in migrate
    modulemds, defaults = load(fp, working_dir)
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0043_add_modulemd_modulemd-defaults.py", line 317, in load
    bfr = metadata.read(1024000)
  File "/usr/lib64/python2.7/gzip.py", line 261, in read
    self._read(readsize)
  File "/usr/lib64/python2.7/gzip.py", line 296, in _read
    self._read_gzip_header()
  File "/usr/lib64/python2.7/gzip.py", line 190, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file