Project

Profile

Help

Issue #8277

closed

Pulp upgrade fails - core.0049_add_file_field_to_uploadchunk

Added by wibbit about 3 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation
Sprint:
Sprint 105
Quarter:
Q3-2021

Description

The following error was found when recently updating to PulpCore 3.10.0

Pastebin Service

Create
API
About

Untitled From WebUI, 14 Hours ago, written in Plain Text. This paste will go to its last resting place in 9 Hours. URL https://paste.centos.org/view/7305a03a Embed Download Paste or View Raw

    bash-4.2$ LD_LIBRARY_PATH=/opt/acme/lib64:/opt/acme/lib PULP_SETTINGS=/opt/acme/etc/pulp/settings.py DJANGO_SETTINGS_MODULE="pulpcore.app.settings" /opt/acme/bin/pulpcore-manager migrate
    Operations to perform:
      Apply all migrations: admin, auth, container, contenttypes, core, file, guardian, rpm, sessions
    Running migrations:
      Applying core.0049_add_file_field_to_uploadchunk...Traceback (most recent call last):
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
    psycopg2.errors.NotNullViolation: column "file" contains null values
     
     
    The above exception was the direct cause of the following exception:
     
    Traceback (most recent call last):
      File "/opt/acme/bin/pulpcore-manager", line 33, in <module>
        sys.exit(load_entry_point('pulpcore==3.10.0', 'console_scripts', 'pulpcore-manager')())
      File "/opt/acme/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
        execute_from_command_line(sys.argv)
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
        utility.execute()
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
        output = self.handle(*args, **options)
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
        res = handle_func(*args, **kwargs)
      File "/opt/acme/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
        post_migrate_state = executor.migrate(
      File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
        state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
      File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
        state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
      File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
        operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
      File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 110, in database_forwards
        schema_editor.add_field(
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 447, in add_field
        self.execute(sql, params)
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
        cursor.execute(sql, params)
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
        return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
        return executor(sql, params, many, context)
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
      File "/opt/acme/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
      File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
    django.db.utils.IntegrityError: column "file" contains null values

Reply to "Untitled" Author Title Language Your paste - Paste your paste here bash-4.2$ LD_LIBRARY_PATH=/opt/acme/lib64:/opt/acme/lib PULP_SETTINGS=/opt/acme/etc/pulp/settings.py DJANGO_SETTINGS_MODULE="pulpcore.app.settings" /opt/acme/bin/pulpcore-manager migrate Operations to perform: Apply all migrations: admin, auth, container, contenttypes, core, file, guardian, rpm, sessions Running migrations: Applying core.0049_add_file_field_to_uploadchunk...Traceback (most recent call last): File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.NotNullViolation: column "file" contains null values

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/acme/bin/pulpcore-manager", line 33, in sys.exit(load_entry_point('pulpcore==3.10.0', 'console_scripts', 'pulpcore-manager')()) File "/opt/acme/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage execute_from_command_line(sys.argv) File "/opt/acme/lib/python3.8/site-packages/django/core/management/init.py", line 381, in execute_from_command_line utility.execute() File "/opt/acme/lib/python3.8/site-packages/django/core/management/init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/opt/acme/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped res = handle_func(*args, **kwargs) File "/opt/acme/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle post_migrate_state = executor.migrate( File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration state = migration.apply(state, schema_editor) File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply operation.database_forwards(self.app_label, schema_editor, old_state, project_state) File "/opt/acme/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 110, in database_forwards schema_editor.add_field( File "/opt/acme/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 447, in add_field self.execute(sql, params) File "/opt/acme/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute cursor.execute(sql, params) File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/opt/acme/lib/python3.8/site-packages/django/db/utils.py", line 89, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/acme/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: column "file" contains null values Create Shorturl - Create a shorter url that redirects to your paste? Private - Private paste aren't shown in recent listings. Delete After - When should we delete your paste? Powered by Stikked

With the following outstanding migrations.

Pastebin Service

Create
API
About

Untitled From WebUI, 13 Hours ago, written in Plain Text. This paste will join the choir invisible in 10 Hours. URL https://paste.centos.org/view/72199584 Embed Download Paste or View Raw

    Planned operations:
    core.0049_add_file_field_to_uploadchunk
        Remove field file from upload
        Add field file to uploadchunk
    container.0009_container_namespace
        Create model ContainerNamespace
        Add field namespace to containerdistribution
        Raw Python operation ->     Look for all ContainerDistribut…
    container.0010_remove_uploadchunk
        Remove field file from upload
        Remove field md5 from upload
        Remove field offset from upload
        Remove field pulp_created from upload
        Remove field pulp_id from upload
        Remove field pulp_last_updated from upload
        Remove field sha1 from upload
        Remove field sha224 from upload
        Remove field sha256 from upload
        Remove field sha384 from upload
        Remove field sha512 from upload
        Remove field size from upload
        Add field upload_ptr to upload
    container.0011_add_container_repository_permissions
        Change Meta options on containerrepository
    container.0012_add_container_namespace_permissions
        Change Meta options on containernamespace
    container.0013_add_pull_push_permissions
        Change Meta options on containerdistribution
    container.0014_containerdistribution_private
        Add field private to containerdistribution
    container.0015_manage_tags_push_repo
        Change Meta options on containerpushrepository
    container.0016_add_delete_versions_permission
        Change Meta options on containerrepository
    container.0017_add_granular_perms
        Change Meta options on containernamespace
    container.0018_containerdistribution_description
        Add field description to containerdistribution
    core.0050_namespace_access_policies
        Raw Python operation
    core.0051_timeoutfields
        Add field connect_timeout to remote
        Add field sock_connect_timeout to remote
        Add field sock_read_timeout to remote
        Add field total_timeout to remote
    core.0052_tasking_logging_cid
        Add field logging_cid to task
    core.0053_remote_headers
        Add field headers to remote
    core.0054_add_public_key
        Add field public_key to signingservice
        Add field pubkey_fingerprint to signingservice
        Raw Python operation
        Alter field public_key on signingservice
        Alter field pubkey_fingerprint on signingservice
    core.0055_label
        Create model Label
    core.0056_remote_rate_limit
        Add field rate_limit to remote
    core.0057_add_label_indexes
        Alter field key on label
        Alter field value on label
    core.0058_accesspolicy_customized
        Add field customized to accesspolicy
    rpm.0026_add_gpgcheck_options
        Add field gpgcheck to rpmpublication
        Add field repo_gpgcheck to rpmpublication
    rpm.0027_checksum_null
        Raw Python operation
        Alter field checksum on checksum
    rpm.0028_rpmrepository_last_sync_repomd_cheksum
        Add field last_sync_repomd_checksum to rpmrepository
    rpm.0029_rpmpublication_sqlite_metadata
        Add field sqlite_metadata to rpmpublication
        Raw Python operation
    rpm.0030_DATA_fix_updaterecord
        Alter field updated_date on updaterecord
        Raw Python operation

Reply to "Untitled" Author Title Language Your paste - Paste your paste here Planned operations: core.0049_add_file_field_to_uploadchunk Remove field file from upload Add field file to uploadchunk container.0009_container_namespace Create model ContainerNamespace Add field namespace to containerdistribution Raw Python operation -> Look for all ContainerDistribut… container.0010_remove_uploadchunk Remove field file from upload Remove field md5 from upload Remove field offset from upload Remove field pulp_created from upload Remove field pulp_id from upload Remove field pulp_last_updated from upload Remove field sha1 from upload Remove field sha224 from upload Remove field sha256 from upload Remove field sha384 from upload Remove field sha512 from upload Remove field size from upload Add field upload_ptr to upload container.0011_add_container_repository_permissions Change Meta options on containerrepository container.0012_add_container_namespace_permissions Change Meta options on containernamespace container.0013_add_pull_push_permissions Change Meta options on containerdistribution container.0014_containerdistribution_private Add field private to containerdistribution container.0015_manage_tags_push_repo Change Meta options on containerpushrepository container.0016_add_delete_versions_permission Change Meta options on containerrepository container.0017_add_granular_perms Change Meta options on containernamespace container.0018_containerdistribution_description Add field description to containerdistribution core.0050_namespace_access_policies Raw Python operation core.0051_timeoutfields Add field connect_timeout to remote Add field sock_connect_timeout to remote Add field sock_read_timeout to remote Add field total_timeout to remote core.0052_tasking_logging_cid Add field logging_cid to task core.0053_remote_headers Add field headers to remote core.0054_add_public_key Add field public_key to signingservice Add field pubkey_fingerprint to signingservice Raw Python operation Alter field public_key on signingservice Alter field pubkey_fingerprint on signingservice core.0055_label Create model Label core.0056_remote_rate_limit Add field rate_limit to remote core.0057_add_label_indexes Alter field key on label Alter field value on label core.0058_accesspolicy_customized Add field customized to accesspolicy rpm.0026_add_gpgcheck_options Add field gpgcheck to rpmpublication Add field repo_gpgcheck to rpmpublication rpm.0027_checksum_null Raw Python operation Alter field checksum on checksum rpm.0028_rpmrepository_last_sync_repomd_cheksum Add field last_sync_repomd_checksum to rpmrepository rpm.0029_rpmpublication_sqlite_metadata Add field sqlite_metadata to rpmpublication Raw Python operation rpm.0030_DATA_fix_updaterecord Alter field updated_date on updaterecord Raw Python operation Create Shorturl - Create a shorter url that redirects to your paste? Private - Private paste aren't shown in recent listings. Delete After - When should we delete your paste? Powered by Stikked

Please note core_uploadchunk is not empty on my system, quite possibly due to failed/early termination of uploads.

Actions #1

Updated by wibbit about 3 years ago

IRC Conversation with Mr Gainey

ggainey
mornin wibbit - appears you were having fun last night
hm
the uploadchunks error - I assume that the core_uploadchunk table has rows in it in your db, yeah?
so yeah, the first pastebin will be resolved when 0052_tasking_logging_cid.py is applied, but the presence of rows in uploadchunk is failing the migration
ggainey
wibbit: I think Red Hat QE should start paying you a stipend
ttereshc: ping - are you around? can you read ^^ and tell me if my reasoning is correct regarding wibbit's migration issue?
ggainey
oh argh, I forgot DevConf starts today
mcorr
:) 
wibbit
Would it be useful to send the previous versions I had, not just PulpCore
ggainey
wibbit: can you look into the db you're trying to upgrade, and tell me if core_uploadchunks has rows in it?
wibbit
ggainey: regarding rows in core_uploadchunk, yes, I've got 1272 rows in that table
ggainey
yeah, that's the problem - the migration doesn't work, because it wants to add the file column, which has no default, which works fine *if the table is empty*
this is a bug that we need to fix
ugh, the 'obvious' workaround is "truncate the table" and then apply the migration
wibbit
I can probably leave my system as is and await the patch, though I'm guessing this is easy enough to test, if this is infact the issue
I take it, this table not being empty isn't unusual, just not some thing that was caught with the test
s
ggainey
well, I haven't been in this workflow, so I'm guessing - but uploadchunk rows get deleted once the whole file has been uploaded. I think.
so this table usually is empty. I think.
wibbit
Interesting, so it could be related to failed uploads, as I don't wipe this DB between my integration testing.
ggainey
yeah, that's what it feels like to me
wibbit
I use the pulp api, to delete things like publications etc, so that I can re-create them.
In fact, looking at these dates they are back in october
ggainey
so my suggestion is thus - can you, please, open an issue with this discussion in it, because we do need to handle this case. Then, for your own sanity, you can truncate that table in your DB, and go on your merry upgrade-way
Actions #2

Updated by fao89 about 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 91
Actions #3

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 91 to Sprint 92
Actions #4

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 92 to Sprint 93
Actions #5

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 93 to Sprint 94
Actions #6

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 94 to Sprint 95
Actions #7

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #8

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #9

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #10

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #11

Updated by lmjachky over 2 years ago

Based on the attached discussion, the issue could be really caused by uncommitted uploads (https://docs.pulpproject.org/pulpcore/changes.html#id230). Since we encouraged users to purge all uncommitted uploaded files, I believe the problem is not on our side. Still, further investigation is required.

Actions #12

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #13

Updated by dalley over 2 years ago

If uncommitted uploads can actually cause upgrade failures then that warning is definitely insufficient, unless it was called out explicitly in the announcement email as well. Manually adding some text to the changelog to make it more prominent wouldn't hurt either.

Just a note for the future, probably not worth worrying too much about now.

Actions #14

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #15

Updated by dalley over 2 years ago

  • Tags Documentation added
Actions #16

Updated by ipanova@redhat.com over 2 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions #17

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions #18

Updated by ttereshc over 2 years ago

  • Quarter set to Q3-2021
Actions #19

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions #20

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #21

Updated by lmjachky over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

I am closing this because the issue is relevant to uncommitted uploads and we cannot fix it.

Lessons learned:

  • send announcements and warnings about potential errors that may arise during the migrations and do not leave the encouragements in changelogs only...
Removals

* The local file system directory used for uploaded chunks is specified by the setting CHUNKED_UPLOAD_DIR. Users are encouraged to remove all uncommitted uploaded files before applying this change. #4498
Actions #22

Updated by lmjachky over 2 years ago

The commit which initially introduced the problem is right here: https://github.com/pulp/pulpcore/commit/1b6c7360c30cc7d9e2d3b9fc5062ac1f7d69c2de. Before this change, we were appending chunks to a single file. Right now, we have one file per uploaded chunk.

Note that by deleting uncommitted uploads, I mean deleting DB records. Deleting the DB records should eventually result in deleting the referenced files (uploads) on the system storage.

This step (deleting uncommitted uploads) was necessary to accomplish due to the fact that the new chunked uploading system could not properly determine how should a single file be split into multiple files representing each uploaded chunk during the migration.

Also available in: Atom PDF