Project

Profile

Help

Issue #5822

closed

Debian Repos - Not possible sync multiple releases

Added by flamarion over 4 years ago. Updated over 3 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Debian:
Platform Release:
Target Release - Debian:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

HI,

I've been trying to figure out the best way to work with the Pulp and Debian repositories and when I found the best way for me I did hit a roadblock.

If I create a repository with only one release the sync works fine, but when I create a repo with multiple releases I'm getting an error regarding the checksum mismatch.

How to reproduce:

1 - Create the repo

 pulp-admin deb repo create --repo-id xenial-amd64 --feed http://us.archive.ubuntu.com/ubuntu --releases xenial,xenial-updates,xenial-backports,xenial-security --components main --architectures amd64 --remove-missing true --relative-url ubuntu --serve-http true --serve-https true --validate false --require-signature false --publish-default-release true

2 - Sync the repo

pulp-admin -v deb repo sync run --repo-id xenial-amd64

Then I'm getting this error:

pulp-admin -v deb repo sync run --repo-id xenial-amd64
+----------------------------------------------------------------------+
                Synchronizing Repository [xenial-amd64]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

Checksum did not match

And an exception is raised on the logs:

Nov 29 08:59:32 n5lpulp00001.nike.com pulp[91990]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f24b083b-d9fb-4861-8b0b-5a3474aa8924]
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[91756]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[2adfd3be-c02a-47eb-8aee-92217f0441bc]
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92612]: celery.app.trace:INFO: [f24b083b] Task pulp.server.async.tasks._queue_reserved_task[f24b083b-d9fb-4861-8b0b-5a3474aa8924] succeeded in 0.0138911469985s: None
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[91756]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[598a1745-6b45-42d1-8d19-e0e0d74007e0]
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: py.warnings:WARNING: [2adfd3be] (92804-71200) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: py.warnings:WARNING: [2adfd3be] (92804-71200)   "MongoClient opened before fork. Create MongoClient "
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: py.warnings:WARNING: [2adfd3be] (92804-71200)
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: pulp_deb.plugins.importers.importer:INFO: [2adfd3be] Repo sync started.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: pulp_deb.plugins.importers.sync:INFO: [2adfd3be] Downloading http://us.archive.ubuntu.com/ubuntu/dists/xenial/Release
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: pulp_deb.plugins.importers.sync:INFO: [2adfd3be] Downloading http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/Release
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: pulp_deb.plugins.importers.sync:INFO: [2adfd3be] Downloading http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/Release
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: pulp_deb.plugins.importers.sync:INFO: [2adfd3be] Downloading http://us.archive.ubuntu.com/ubuntu/dists/xenial-security/Release
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): us.archive.ubuntu.com
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): us.archive.ubuntu.com
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): us.archive.ubuntu.com
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): us.archive.ubuntu.com
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): us.archive.ubuntu.com
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial/Release.gpg.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/Release.gpg.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/Release.gpg.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/Release.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-security/Release.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/Release.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-security/Release.gpg.
Nov 29 08:59:33 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial/Release.
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): us.archive.ubuntu.com
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): us.archive.ubuntu.com
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): us.archive.ubuntu.com
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): us.archive.ubuntu.com
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/main/binary-amd64/Packages.xz.
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64/Packages.xz.
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-amd64/Packages.xz.
Nov 29 08:59:34 n5lpulp00001.nike.com pulp[92804]: nectar.downloaders.threaded:INFO: Download succeeded: http://us.archive.ubuntu.com/ubuntu/dists/xenial/main/binary-amd64/Packages.xz.
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: pulp.server.async.tasks:INFO: [2adfd3be] Task failed : [2adfd3be-c02a-47eb-8aee-92217f0441bc]
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200) Task pulp.server.managers.repo.sync.sync[2adfd3be-c02a-47eb-8aee-92217f0441bc] raised unexpected: Exception(u'Checksum did not match',)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200) Traceback (most recent call last):
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     R = retval = fun(*args, **kwargs)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     return super(Task, self).__call__(*args, **kwargs)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     return super(PulpTask, self).__call__(*args, **kwargs)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     return self.run(*args, **kwargs)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 770, in sync
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     return f(*args, **kwargs)
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py", line 135, in sync_repo
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     report = self._current_sync.process_lifecycle()
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     super(PluginStep, self).process_lifecycle()
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     step.process()
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     self._process_block()
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     self.process_main()
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py", line 272, in process_main
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     if dlr.url in self.parent.packages_urls[release]])
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 302, in validate_component_arch_packages_downloads
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200)     raise Exception("Checksum did not match")
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:ERROR: [2adfd3be] (92804-71200) Exception: Checksum did not match
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[92804]: celery.app.trace:INFO: [598a1745] Task pulp.server.async.tasks._release_resource[598a1745-6b45-42d1-8d19-e0e0d74007e0] succeeded in 0.00169369099604s: None
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[93432]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Nov 29 08:59:35 n5lpulp00001.nike.com pulp[93432]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Nov 29 08:59:36 n5lpulp00001.nike.com pulp[93432]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}

My installation has this components and statuses:

python-pulp-oid_validation-2.18.1-2.el7.noarch
python-pulp-rpm-common-2.18.1-1.el7.noarch
pulp-server-2.18.1-2.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-ostree-admin-extensions-1.4.0-1.el7.noarch
python-pulp-repoauth-2.18.1-2.el7.noarch
pulp-puppet-plugins-2.18.1-1.el7.noarch
python-pulp-client-lib-2.18.1-2.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
python-pulp-puppet-common-2.18.1-1.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
pulp-deb-plugins-1.8.0-1.el7.noarch
python-pulp-bindings-2.18.1-2.el7.noarch
pulp-rpm-admin-extensions-2.18.1-1.el7.noarch
python-pulp-ostree-common-1.4.0-1.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-rpm-plugins-2.18.1-1.el7.noarch
pulp-puppet-tools-2.18.1-1.el7.noarch
pulp-admin-client-2.18.1-2.el7.noarch
pulp-docker-admin-extensions-3.2.2-1.el7.noarch
pulp-selinux-2.18.1-2.el7.noarch
pulp-ostree-plugins-1.4.0-1.el7.noarch
pulp-puppet-admin-extensions-2.18.1-1.el7.noarch
python-pulp-common-2.18.1-2.el7.noarch
python-pulp-deb-common-1.8.0-1.el7.noarch
pulp-deb-admin-extensions-1.8.0-1.el7.noarch

Status:

pulp-admin status
+----------------------------------------------------------------------+
                          Status of the server
+----------------------------------------------------------------------+

Api Version:           2
Database Connection:
  Connected: True
Known Workers:
  _id:            scheduler@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:16Z
  _id:            reserved_resource_worker-5@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-3@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-6@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-9@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-11@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-0@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-1@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-7@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-8@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            resource_manager@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-10@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-2@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-12@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:15Z
  _id:            reserved_resource_worker-13@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:16Z
  _id:            reserved_resource_worker-14@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:16Z
  _id:            reserved_resource_worker-15@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:16Z
  _id:            reserved_resource_worker-4@n5lpulp00001.nike.com
  _ns:            workers
  Last Heartbeat: 2019-11-29T09:06:12Z
Messaging Connection:
  Connected: True
Versions:
  Platform Version: 2.18.1

Could you, please, let me know if this is a bug indeed or if I'm doing something wrong?

So far I was creating separated relative URL's for each release and syncing separately, but the repository was a mess, then I decided to try to reproduce the same structure used by the upstream repositories and create a structure more consolidated.

Regards,

Flamarion

Also available in: Atom PDF