Issue #7836: Import fails when ArtifactResource.json has blank checksums - Pulp

Actions

Send by e-mail Copy link

Issue #7836

closed

Task #7960: FIPS and support for ALLOWED_CONTENT_CHECKSUMS

Import fails when ArtifactResource.json has blank checksums

Added by ggainey about 4 years ago. Updated almost 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

daviddavis

Category:

Sprint/Milestone:

3.10.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Sprint 87

Quarter:

Q4-2020

Description

The export delivers empty checksums as empty strings:

    {
        "file": "artifact/9c/3f67ddd806d47c09d563ef94b622cd5a1736552b2f5bfb4c71f98cebb14729",
        "size": 1859,
        "md5": "",
        "sha1": "",
        "sha224": "",
        "sha256": "9c3f67ddd806d47c09d563ef94b622cd5a1736552b2f5bfb4c71f98cebb14729",
        "sha384": "",
        "sha512": "8b53cbae3f6b504981bc44d9653bc63dd21ded38ed48a168b50d515d745df13444ec0e5bddc53ed662f767d0905571eb754e4e4daa591c33ec6850bda1fb769e"
    },

At import, this fails with an error like this one :

Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: pulp: pulpcore.app.tasks.importer:INFO: ...Importing resource ArtifactResource.
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self.cursor.execute(sql, params)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_artifact_sha384_key"
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: DETAIL:  Key (sha384)=() already exists.
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: The above exception was the direct cause of the following exception:
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]: Traceback (most recent call last):
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/worker.py", line 936, in perform_job
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     rv = job.perform()
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/job.py", line 684, in perform
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     self._result = self._execute()
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/job.py", line 690, in _execute
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self.func(*self.args, **self.kwargs)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 339, in pulp_import
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     ar_result = _import_file(os.path.join(temp_dir, ARTIFACT_FILE), ArtifactResource)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 61, in _import_file
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return resource.import_data(data, raise_errors=True)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/import_export/resources.py", line 745, in import_data
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self.import_data_inner(dataset, dry_run, raise_errors, using_transactions, collect_failed_rows, **kwargs)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/import_export/resources.py", line 792, in import_data_inner
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     raise row_result.errors[-1].error
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/import_export/resources.py", line 672, in import_row
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     self.save_instance(instance, using_transactions, dry_run)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/import_export/resources.py", line 450, in save_instance
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     instance.save()
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/models/content.py", line 140, in save
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     super().save(*args, **kwargs)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django_lifecycle/mixins.py", line 129, in save
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     save(*args, **kwargs)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 743, in save
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     self.save_base(using=using, force_insert=force_insert,
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     updated = self._save_table(
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return manager._insert([self], fields=fields, return_id=update_pk,
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return query.get_compiler(using=using).execute_sql(return_id)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     cursor.execute(sql, params)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return executor(sql, params, many, context)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self.cursor.execute(sql, params)
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:   File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 14 20:38:12 pulp3-source-fedora32.padre-fedora.example.com rq[309851]:     return self.cursor.execute(sql, params)

Related issues

Actions

Copy link

Updated by ggainey about 4 years ago

Related to Task #7537: Add support for ALLOWED_CONTENT_CHECKSUMS added

Actions

Copy link

Updated by ggainey about 4 years ago

This is only a problem when a) you have a CharField that b) is null=True unique=True that you c) export when it's null. Django-import-export exports null CharFields as blank strings.

The only fields I can find, in pulpcore, pulp_rpm, pulp_file, and pulp_container, that are exposed to this bug, are the sha224, sha384, and sha512 fields. I think if we set them to unique=False like all the checksum fields other than sha256 already are, and always have been, we can get past this.

Actions

Copy link

Updated by ggainey about 4 years ago

Another option for fixing this would be to follow the pattern from here:

https://medium.com/@bramblexu/django-allow-null-for-certain-model-filed-with-blank-and-null-b0f402eeca7e

and make the checksum-fields be 'CharNullField' instead of CharField.

Actions

Copy link

Updated by fao89 about 4 years ago

Triaged changed from No to Yes
Sprint set to Sprint 86
Quarter set to Q4-2020

Actions

Copy link

Updated by rchan almost 4 years ago

Sprint changed from Sprint 86 to Sprint 87

Actions

Copy link

Updated by daviddavis almost 4 years ago

Status changed from NEW to ASSIGNED
Assignee set to daviddavis

Either of those options sound fine to me.

Thinking out loud: would a third option be to add a before_import_row to ArtifactResource and convert these blank strings to null for sha224, sha384, and sha512?

Actions

Copy link

Updated by daviddavis almost 4 years ago

Parent issue set to #7960

Actions

Copy link

Updated by ggainey almost 4 years ago

daviddavis wrote:

Either of those options sound fine to me.

Thinking out loud: would a third option be to add a before_import_row to ArtifactResource and convert these blank strings to null for sha224, sha384, and sha512?

This one would let us address the issue without a model-fix. It will have some impact on import-speed, since we'd be inspecting each row before importing - but the cost is relatively small (a function-call and three is-empty checks). Sounds reasonable to me.

Actions

Copy link