Debian Support

The full trace for the error would be useful.

The only thing that strikes the eye is that with all of Ubuntu bionic and two architectures this will be a large sync. (I don't think I have a test system with a large enough disk to reproduce it.) Of course large sync should nevertheless work!

The error sounds like there is some artifact that pulp_deb is trying to save more than once, perhaps the back trace can give us some clues what that artifact is. (We have the checksum, so searching through packages indices might also work...)

The error for the sync with "ignore_missing_package_indices=true" looks odd, since it suggests it is trying to create some artifact with a relative path of dists/bionic-backports. But that path is a folder and not a file, and should never be used for any artifact... I will need to investigate.

Traceback for the duplicate key error with ignore_missing_package_indices=false:

  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job
    rv = job.perform()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform
    self._result = self._execute()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute
    return self.func(*self.args, **self.kwargs)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize
    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create
    loop.run_until_complete(pipeline)
  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete
    return future.result()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__
    await self.run()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 304, in run 
    da.artifact.save()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/app/models/content.py\", line 126, in save
    super().save(*args, **kwargs)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django_lifecycle/mixins.py\", line 129, in save
    save(*args, **kwargs)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/base.py\", line 744, in save
    force_update=force_update, update_fields=update_fields)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/base.py\", line 782, in save_base
    force_update, using, update_fields,
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/base.py\", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/base.py\", line 911, in _do_insert
    using=using, raw=raw)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1377, in execute_sql
    cursor.execute(sql, params)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute
    return self.cursor.execute(sql, params)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute
    return self.cursor.execute(sql, params)

Traceback for the "ContentArtifact matching query does not exist" error with ignore_missing_package_indices=true:

  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job
    rv = job.perform()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform
    self._result = self._execute()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute
    return self.func(*self.args, **self.kwargs) 
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize
    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create
    loop.run_until_complete(pipeline)
  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete
    return future.result()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__
    await self.run()
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 113, in run
    ContentArtifact.objects.bulk_get_or_create(content_artifact_bulk)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/pulpcore/app/models/content.py\", line 89, in bulk_get_or_create
    objs[i] = objs[i].__class__.objects.get(objs[i].q())
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File \"/site/apps/pulp3/pulpenv/lib64/python3.6/site-packages/django/db/models/query.py\", line 408, in get
    self.model._meta.object_name

In case it's helpful, if I patch the remote to only include bionic bionic-security bionic-updates, the sync succeeds.

The minimal remote that I was able to reproduce this with goes as follows:

ENTITIES_NAME='bionic_backports'
UPSTREAM_URL='http://mirror.us.leaseweb.net/ubuntu'
ARCHITECTURES='amd64'
COMPONENTS='multiverse restricted'
DISTRIBUTIONS='bionic-backports'

REPO_HREF=$(http post "${BASE_ADDR}"/pulp/api/v3/repositories/deb/apt/ name="${ENTITIES_NAME}" | jq -r '.pulp_href')
REMOTE_HREF=$(http post "${BASE_ADDR}"/pulp/api/v3/remotes/deb/apt/ name="${ENTITIES_NAME}" distributions="${DISTRIBUTIONS}" architectures="${ARCHITECTURES}" components="${COMPONENTS}" url="${UPSTREAM_URL}" | jq -r '.pulp_href')
SYNC_TASK_HREF=$(http post "${BASE_ADDR}${REPO_HREF}"sync/ remote="${REMOTE_HREF}" | jq -r '.task')
http get "${BASE_ADDR}${SYNC_TASK_HREF}" | jq '.state'

The two components multiverse and restricted can be individually synced, but not both within the same pulp instance. It looks like something from those two components clashes within pulp_deb.

Update: It looks like the problematic components are both empty (contain no packages), and Pulp is tripping over the fact that both Packages file artifacts being created are empty and hence identical. However, it is very strange that we are tripping over this, since we have been able to sync empty repositories (which should suffer from the same problem) and indeed we have test coverage for empty repositories.

See also: https://pulp.plan.io/issues/7344

I hit the same problem trying to sync "xenial xenial-updates xenial-backports" from http://no.archive.ubuntu.com/ubuntu/

How can I tell if the components are empty on this one too?