Project

Profile

Help

Issue #1945

pulp 2.8.3 migration fails on unit 0028

Added by kfiresmith over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.3
Platform Release:
2.8.4
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Halting migrations due to a migration failure.

[Errno 20] Not a directory: u'/var/lib/pulp/content/units/iso/dd/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 124, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 186, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0028_standard_storage_path.py", line 17, in migrate
    migration()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 266, in __call__
    batch()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 112, in __call__
    self._migrate()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 101, in _migrate
    Unit.migrate(item.plan, item.unit_id, item.storage_path, item.new_path)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 364, in migrate
    shutil.move(path, new_path)
  File "/usr/lib64/python2.7/shutil.py", line 301, in move
    copy2(src, real_dst)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 20] Not a directory: u'/var/lib/pulp/content/units/iso/dd/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6'

Associated revisions

Revision bb0b21e2 View on GitHub
Added by jortel@redhat.com over 4 years ago

Fix storage path migration issue with ISO and distributions. closes #1945

Revision 9099c5a0 View on GitHub
Added by jortel@redhat.com over 4 years ago

Fix storage path migration issue with ISO and distributions. closes #1945

History

#1 Updated by kfiresmith over 4 years ago

@jortel is aware of the issue and is looking into the migration unit code.

#2 Updated by bmbouter over 4 years ago

  • Description updated (diff)

#3 Updated by rbarlow over 4 years ago

  • Priority changed from Normal to Urgent

I suggest urgent priority since this issue blocks users from upgrading to 2.8.3, which is a security release.

#4 Updated by bmbouter over 4 years ago

+1 to urgent, and if this is being worked on it should be in ASSIGNED state.

#5 Updated by jortel@redhat.com over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#6 Updated by ehelms@redhat.com over 4 years ago

Related to this migration another upgrade traceback, if you'd like a separate issue for it please let me know.


Migration to pulp_docker.plugins.migrations version 2 complete.
Applying pulp_puppet.plugins.migrations version 4
Migration to pulp_puppet.plugins.migrations version 4 complete.
Applying pulp_puppet.plugins.migrations version 5
***Note. You may want to re-publish the list of repos found in /tmp/repos_to_republish.txt.
   This migration fixed an issue with modules installation related to wrong puppet_module name.
Migration to pulp_puppet.plugins.migrations version 5 complete.
Applying pulp_rpm.plugins.migrations version 28
Applying migration pulp_rpm.plugins.migrations.0028_standard_storage_path failed.

Halting migrations due to a migration failure.
'variant'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 124, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 186, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0028_standard_storage_path.py", line 17, in migrate
    migration()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 262, in __call__
    for unit in chain(*self.plans):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 188, in __iter__
    unit.new_path = self._new_path(unit)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 165, in _new_path
    digest = unit.key_digest()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 338, in key_digest
    for key, value in sorted(self.key.items()):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 319, in key
    return dict([(k, self.document[k]) for k in self.plan.key_fields])
KeyError: 'variant'

#7 Updated by kfiresmith over 4 years ago

Hi Eric,
That actually looks unrelated; we think what I'm seeing is an issue manifesting from outside the DB, and your traceback looks very much like an issue from within the database. Just my $0.02 but I think your issue will turn out to be something independent of what I reported. :)

#8 Updated by jortel@redhat.com over 4 years ago

Reproduced with tons of help from @kfiresmith. Big thanks.

Some of the plugins released with early 2.8 had been updated (incorrectly) to use the new platform storage model (introduced in 2.8) for unit files. This resulted in a period of time when units were created with storage paths that should have ended with the file names but did not. Apparently the ISO importer was one of these plugins but was fixed in later 2.8 releases. The ISO storage-path migration just needs to be updated to detect these units and correct the storage path.

#9 Updated by jortel@redhat.com over 4 years ago

  • Status changed from ASSIGNED to POST

#10 Updated by bmbouter over 4 years ago

  • Triaged changed from No to Yes

#11 Updated by kfiresmith over 4 years ago

Thanks very much Jeff - I can confirm this fixed the failed migration we saw.

#12 Updated by jortel@redhat.com over 4 years ago

PR updated to deal with missing variant in distributions.

#13 Updated by jortel@redhat.com over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#14 Updated by semyers over 4 years ago

  • Platform Release set to 2.8.4

#15 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to 5

#16 Updated by pthomas@redhat.com over 4 years ago

  • Status changed from 5 to 6

verified
Ran migration on an upgraded pulp with data in rpm.puppet, iso and docker repos

[root@mgmt8 ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, docker_manifest, docker_blob, erratum, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 23
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 4
Applying pulp_rpm.plugins.migrations version 29
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:23: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:28: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

/usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  upsert=upsert, **write_concern)

Migration to pulp_rpm.plugins.migrations version 29 complete.
Applying pulp_rpm.plugins.migrations version 30
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0030_remove_errata_unused_fields.py:23: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

Migration to pulp_rpm.plugins.migrations version 30 complete.
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.
[root@mgmt8 ~]# 

#17 Updated by semyers over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#19 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF