Pulp

Can only reproduce this when also migrating the centos8-baseos repo simultaneously with centos8-kickstart. If attempting to migrate either of them individually, it works, even when migrating the full set of content from both.

I notice that when I migrate centos8-baseos and then sync, all the package groups and the distribution tree are removed and the same number of new units added, which possibly signals something wrong with the migration?

So this is interesting.

The file is created, properly, but gets deleted at some point during the migration. If I pause it in the right place I can see the file there on the filesystem and read it and it's got all the right contents. And then when I continue the migration it fails because the file can't be found. Weird.

I figured out how to make the file undeletable [0] and got this error when continuing the migration

Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: pulp: django_cleanup.handlers:ERROR: There was an exception deleting the file `/var/lib/pulp/artifact/48/66eceaf78ffabb1d25774932085c6653f9f654411d7669db7b
91e2ef0fd2a3` on field `core.artifact.file`
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: Traceback (most recent call last):
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return self.cursor.execute(sql, params)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_artifact_pkey"
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: DETAIL:  Key (pulp_id)=(5ee472a8-aae9-447c-b7a1-74aa0efee3a5) already exists.
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: The above exception was the direct cause of the following exception:
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: Traceback (most recent call last):
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/content.py", line 87, in bulk_get_or_create
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return super().bulk_create(objs, batch_size=batch_size)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 468, in bulk_create
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 1204, in _batched_insert
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: ignore_conflicts=ignore_conflicts,
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return query.get_compiler(using=using).execute_sql(return_id)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: cursor.execute(sql, params)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return executor(sql, params, many, context)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return self.cursor.execute(sql, params)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: raise dj_exc_value.with_traceback(traceback) from exc_value
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: return self.cursor.execute(sql, params)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_artifact_pkey"
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: DETAIL:  Key (pulp_id)=(5ee472a8-aae9-447c-b7a1-74aa0efee3a5) already exists.
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: During handling of the above exception, another exception occurred:
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: Traceback (most recent call last):
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django_cleanup/handlers.py", line 96, in run_on_commit
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: file_.delete(save=False)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/fields/files.py", line 105, in delete
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: self.storage.delete(self.name)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/files/storage.py", line 303, in delete
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: os.remove(name)
Oct 27 04:46:58 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[7711]: PermissionError: [Errno 1] Operation not permitted: '/var/lib/pulp/artifact/48/66eceaf78ffabb1d25774932085c6653f9f654411d7669db7b91e2ef0fd2a3'

[0] https://www.tecmint.com/make-file-directory-undeletable-immutable-in-linux/

Alright, I think I know what's going on.

Here's the code of our mixin that provides bulk_get_or_create()

        objs = list(objs)
        try:
            with transaction.atomic():
                return super().bulk_create(objs, batch_size=batch_size)
        except IntegrityError:
            for i in range(len(objs)):
                try:
                    with transaction.atomic():
                        objs[i].save()
                except IntegrityError:
                    objs[i] = objs[i].__class__.objects.get(objs[i].q())
        return objs

When it falls back to saving the objects manually, and attempts to save the Artifact in question, it fails on a uniqueness constraint. Which is fine and expected and all of that, except that about 2 months ago we added django-cleanup.

https://github.com/pulp/pulpcore/commit/f508d6b2ba4d624d64bcdbfa6c851afe159c6fac#diff-4d7c51b1efe9043e44439a949dfd92e5827321b34082903477fd04876edb7552

And django-cleanup sees that our model failed to save and decides to helpfully clean up the file that was "left behind" except that in this case it's actually shared with the existing Artifact which is already saved.

I think this would be a problem anywhere that multiple content units share the same artifact, and the artifact has already been downloaded prior to the second content unit being processed?

We notice this only because the RPM plugin migration because it's a place where we have two of the same DeclarativeContent running through the pipeline, and we end up reading the distribution tree file at publish time - most plugins don't actually try to open the file, so you wouldn't notice until requesting the file via the content app. Or maybe not? Do we handle this case where the file disappeared in the content app and try to correct the corruption?

Confirmed that removing the django-cleanup middleware resolves the original issue.

dalley can you remove the middleware as a PR and also reopen whatever issue was the one that introduced it in the first place?

bmbouter, yes, but we should probably think about how we try to find and fix existing corruption.

• In a migration?
• In a global integrity check feature?
• Lazily, in the content app?

Is it good enough to just delete the existing Artifact and then let the lazy sync machinery re-download?

dalley wrote:

bmbouter, yes, but we should probably think about how we try to find and fix existing corruption.

• In a migration?
• In a global integrity check feature? We had wondered if we were going to make this before. We currently have a "per-repository" repair, so this would be a "global" version of that. I'm in favor of this.

• Lazily, in the content app? I think it already does this actually, but we need to check... IIRC, it came up with Katello. I have not checked. If not I'm in favor of this.

Is it good enough to just delete the existing Artifact and then let the lazy sync machinery re-download? I think it is.