Debian Support

The remote looks fine to me.

It is also important to note, that Debian appears to have given up on their policy of publishing their security repos using an "unusual" structure starting with "bullseye".

This does NOT appear to be a codename != distribution issue!

I will need to investigate.

May I ask what version of pulp_deb you are using?

Hi, I am using 2.3.0b1 and manually applied the fixes from following issues : 6907 6873 6876 6787

I have no clue if all these fixes have been published in a version installed with pip or not yet. If you'd like me to upgrade pulp_deb or reinstal pulp server, let me know

As far as I can tell the main trouble with the bullseye-security repo is that it does not yet contain any packages.

(There is a possible second issue, in that this repository includes a weird circular symlink for backwards compatibility with the legacy dist folder structure).

The plugin is currently completely unable to handle valid repositories that do not contain any packages.

Syncing such repositories is a valid use case that needs to be fixed.

I have opend a draft PR for adding relevant test fixtures here: https://github.com/pulp/pulp-fixtures/pull/187

Update: Syncing empty (no package) repositories works and has test coverage, but bullseye-security is still broken.

My only remaining theory is that the circular symlink is the root cause. However, it is still far from clear to me, why it should matter.

So testing this with latest main-branch of pulp_deb I got the following error on the first attempt. It seems to try to save the empty file artifact multiple times.

Interestingly, the second attempt succeeds.

pulp [cc8b664323d54993bb1e018edc73f432]: rq.worker:ERROR: Traceback (most recent call last):
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_artifact_sha256_key"
DETAIL:  Key (sha256)=(e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/worker.py", line 1013, in perform_job
    rv = job.perform()
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/job.py", line 709, in perform
    self._result = self._execute()
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/rq/job.py", line 732, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/home/vagrant/devel/pulp_deb/pulp_deb/app/tasks/synchronizing.py", line 122, in synchronize
    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 149, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/home/vagrant/devel/pulp_deb/pulp_deb/app/tasks/synchronizing.py", line 320, in run
    da.artifact.save()
  File "/home/vagrant/devel/pulpcore/pulpcore/app/models/content.py", line 126, in save
    super().save(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django_lifecycle/mixins.py", line 134, in save
    save(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_artifact_sha256_key"
DETAIL:  Key (sha256)=(e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) already exists.

I was able to reproduce the same effect using the following remote:

REMOTE_HREF=$(http post "${BASE_ADDR}"/pulp/api/v3/remotes/deb/apt/ name="${ENTITIES_NAME}" distributions="buster/updates" url="http://ftp.de.debian.org/debian-security/" | jq -r '.pulp_href')