Issue #9237
closedVery slowly publication and progress report
Description
Ticket moved to GitHub: "pulp/pulp_deb/418":https://github.com/pulp/pulp_deb/issues/418
Today i started mirroring for http://archive.ubuntu.com
# pulp task list --state running | jq
[
{
"pulp_href": "/pulp/api/v3/tasks/d41cd931-9d7b-4ead-a381-d6aae8c709e0/",
"pulp_created": "2021-08-12T07:50:28.476170Z",
"state": "running",
"name": "pulp_deb.app.tasks.publishing.publish",
"logging_cid": "60c1f76bef074040943b9ed4558a8732",
"started_at": "2021-08-12T07:50:28.621880Z",
"finished_at": null,
"error": null,
"worker": "/pulp/api/v3/workers/b64aee98-0684-4daf-9b67-6f8a4aa36370/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [],
"created_resources": [
null
],
"reserved_resources_record": [
"/pulp/api/v3/repositories/deb/apt/a1a61370-650c-4d9f-9c28-79e444110dda/"
]
}
]
And they still running, i can't understand how long me need waiting to complete.
"versions": [
{
"component": "core",
"version": "3.14.3"
},
{
"component": "rpm",
"version": "3.14.0"
},
{
"component": "python",
"version": "3.4.0"
},
{
"component": "file",
"version": "1.8.1"
},
{
"component": "deb",
"version": "2.14.1"
},
{
"component": "container",
"version": "2.7.0"
}
],
I think we have data for specify progress (count of contents and count packages in repository), please, if it can, add more information about publication process to output
Updated by wilful over 2 years ago
{ [159/1881]
"architectures": "amd64",
"ca_cert": null,
"client_cert": null,
"components": null,
"connect_timeout": null,
"distributions": "trusty trusty-backports trusty-proposed trusty-security trusty-updates focal focal-backports focal-proposed focal-security focal-updates",
"download_concurrency": null,
"gpgkey": null,
"headers": null,
"ignore_missing_package_indices": false,
"max_retries": null,
"name": "archive.ubuntu.com",
"policy": "on_demand",
"proxy_url": null,
"pulp_created": "2021-08-12T06:57:21.826903Z",
"pulp_href": "/pulp/api/v3/remotes/deb/apt/248db97b-b81a-4be2-9048-faf441b3dfaa/",
"pulp_labels": {},
"pulp_last_updated": "2021-08-12T06:57:21.826926Z",
"rate_limit": null,
"sock_connect_timeout": null,
"sock_read_timeout": null,
"sync_installer": false,
"sync_sources": false,
"sync_udebs": false,
"tls_validation": true,
"total_timeout": null,
"url": "http://archive.ubuntu.com/ubuntu/"
},
Updated by quba42 over 2 years ago
What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.
Updated by wilful over 2 years ago
Postgres starting only SELECT
postgres 124116 94.5 7.5 3829508 1234272 ? Rs 07:50 407:09 postgres: pulp pulp 127.0.0.1(4810) SELECT
Updated by wilful over 2 years ago
quba42 wrote:
What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.
Sorry, what is mean "What publisher"? I am using one what have in API instruction for example: /pulp/api/v3/docs/#operation/publications_deb_apt_create
My goal make local mirror repository as http://archive.ubuntu.com/ubuntu/ This repository now have all 10 parts in one (before it was different by domain)
Updated by quba42 over 2 years ago
If you are specifically trying to mirror, consider using the Verbatim publisher, rather than the APT publisher. It should be more efficient in the publication since it does not need to generate any metadata.
See the high level documentation for both types of publication:
- https://docs.pulpproject.org/pulp_deb/feature_overview.html#simple-and-structured-publishing
- https://docs.pulpproject.org/pulp_deb/feature_overview.html#verbatim-publishing
Or see the relevant API docs:
Updated by wilful over 2 years ago
thanks, i going to try Verbatim publisher (or split repositories by dist), may be it help me create identical mirror as Ubuntu.
Updated by pulpbot over 2 years ago
- Description updated (diff)
- Status changed from NEW to CLOSED - DUPLICATE