Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1945

closed

pulp 2.8.3 migration fails on unit 0028

Added by kfiresmith almost 8 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Assignee:
jortel@redhat.com
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.3
Platform Release:
2.8.4
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Halting migrations due to a migration failure.

[Errno 20] Not a directory: u'/var/lib/pulp/content/units/iso/dd/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 124, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 186, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0028_standard_storage_path.py", line 17, in migrate
    migration()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 266, in __call__
    batch()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 112, in __call__
    self._migrate()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 101, in _migrate
    Unit.migrate(item.plan, item.unit_id, item.storage_path, item.new_path)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 364, in migrate
    shutil.move(path, new_path)
  File "/usr/lib64/python2.7/shutil.py", line 301, in move
    copy2(src, real_dst)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 20] Not a directory: u'/var/lib/pulp/content/units/iso/dd/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6/189ab6381d724c0219065338e058dc65e72221c3f7b8da6957f43d78f0c9f6'
Actions
Copy link
 #1

Updated by kfiresmith almost 8 years ago

@jortel is aware of the issue and is looking into the migration unit code.

Actions
Copy link
 #2

Updated by bmbouter almost 8 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by rbarlow almost 8 years ago

  • Priority changed from Normal to Urgent

I suggest urgent priority since this issue blocks users from upgrading to 2.8.3, which is a security release.

Actions
Copy link
 #4

Updated by bmbouter almost 8 years ago

+1 to urgent, and if this is being worked on it should be in ASSIGNED state.

Actions
Copy link
 #5

Updated by jortel@redhat.com almost 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions
Copy link
 #6

Updated by ehelms@redhat.com almost 8 years ago

Related to this migration another upgrade traceback, if you'd like a separate issue for it please let me know.


Migration to pulp_docker.plugins.migrations version 2 complete.
Applying pulp_puppet.plugins.migrations version 4
Migration to pulp_puppet.plugins.migrations version 4 complete.
Applying pulp_puppet.plugins.migrations version 5
***Note. You may want to re-publish the list of repos found in /tmp/repos_to_republish.txt.
   This migration fixed an issue with modules installation related to wrong puppet_module name.
Migration to pulp_puppet.plugins.migrations version 5 complete.
Applying pulp_rpm.plugins.migrations version 28
Applying migration pulp_rpm.plugins.migrations.0028_standard_storage_path failed.

Halting migrations due to a migration failure.
'variant'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 124, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 186, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0028_standard_storage_path.py", line 17, in migrate
    migration()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 262, in __call__
    for unit in chain(*self.plans):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 188, in __iter__
    unit.new_path = self._new_path(unit)
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 165, in _new_path
    digest = unit.key_digest()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 338, in key_digest
    for key, value in sorted(self.key.items()):
  File "/usr/lib/python2.7/site-packages/pulp/plugins/migration/standard_storage_path.py", line 319, in key
    return dict([(k, self.document[k]) for k in self.plan.key_fields])
KeyError: 'variant'
Actions
Copy link
 #7

Updated by kfiresmith almost 8 years ago

Hi Eric,
That actually looks unrelated; we think what I'm seeing is an issue manifesting from outside the DB, and your traceback looks very much like an issue from within the database. Just my $0.02 but I think your issue will turn out to be something independent of what I reported. :)

Actions
Copy link
 #8

Updated by jortel@redhat.com almost 8 years ago

Reproduced with tons of help from kfiresmith. Big thanks.

Some of the plugins released with early 2.8 had been updated (incorrectly) to use the new platform storage model (introduced in 2.8) for unit files. This resulted in a period of time when units were created with storage paths that should have ended with the file names but did not. Apparently the ISO importer was one of these plugins but was fixed in later 2.8 releases. The ISO storage-path migration just needs to be updated to detect these units and correct the storage path.

Actions
Copy link
 #9

Updated by jortel@redhat.com almost 8 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #10

Updated by bmbouter almost 8 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #11

Updated by kfiresmith almost 8 years ago

Thanks very much Jeff - I can confirm this fixed the failed migration we saw.

Added by jortel@redhat.com almost 8 years ago

Revision bb0b21e2 | View on GitHub

Fix storage path migration issue with ISO and distributions. closes #1945

Actions
Copy link
 #12

Updated by jortel@redhat.com almost 8 years ago

PR updated to deal with missing variant in distributions.

Actions
Copy link
 #13

Updated by jortel@redhat.com almost 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #14

Updated by semyers almost 8 years ago

  • Platform Release set to 2.8.4

Added by jortel@redhat.com almost 8 years ago

Revision 9099c5a0 | View on GitHub

Fix storage path migration issue with ISO and distributions. closes #1945

Actions
Copy link
 #15

Updated by semyers almost 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #16

Updated by pthomas@redhat.com almost 8 years ago

  • Status changed from 5 to 6

verified
Ran migration on an upgraded pulp with data in rpm.puppet, iso and docker repos

[root@mgmt8 ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, docker_manifest, docker_blob, erratum, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 23
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 4
Applying pulp_rpm.plugins.migrations version 29
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:23: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0029_remove_rpm_unused_fields.py:28: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

/usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  upsert=upsert, **write_concern)

Migration to pulp_rpm.plugins.migrations version 29 complete.
Applying pulp_rpm.plugins.migrations version 30
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/migrations/0030_remove_errata_unused_fields.py:23: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
  multi=True

Migration to pulp_rpm.plugins.migrations version 30 complete.
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.
[root@mgmt8 ~]#
Actions
Copy link
 #17

Updated by semyers almost 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #19

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF