Issue #9237: Very slowly publication and progress report - Debian Support - Pulp

Actions

Send by e-mail Copy link

Issue #9237

closed

Very slowly publication and progress report

Added by wilful over 2 years ago. Updated over 2 years ago.

Status:

CLOSED - DUPLICATE

Priority:

Normal

Assignee:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version - Debian:

Platform Release:

Target Release - Debian:

OS:

Triaged:

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

Ticket moved to GitHub: "pulp/pulp_deb/418":https://github.com/pulp/pulp_deb/issues/418

Today i started mirroring for http://archive.ubuntu.com

# pulp task list --state running | jq
[
  {
    "pulp_href": "/pulp/api/v3/tasks/d41cd931-9d7b-4ead-a381-d6aae8c709e0/",
    "pulp_created": "2021-08-12T07:50:28.476170Z",
    "state": "running",
    "name": "pulp_deb.app.tasks.publishing.publish",
    "logging_cid": "60c1f76bef074040943b9ed4558a8732",
    "started_at": "2021-08-12T07:50:28.621880Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/b64aee98-0684-4daf-9b67-6f8a4aa36370/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      null
    ],
    "reserved_resources_record": [
      "/pulp/api/v3/repositories/deb/apt/a1a61370-650c-4d9f-9c28-79e444110dda/"
    ]
  }
]

And they still running, i can't understand how long me need waiting to complete.

  "versions": [
    {
      "component": "core",
      "version": "3.14.3"
    },
    {
      "component": "rpm",
      "version": "3.14.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.1"
    },
    {
      "component": "deb",
      "version": "2.14.1"
    },
    {
      "component": "container",
      "version": "2.7.0"
    }
  ],

I think we have data for specify progress (count of contents and count packages in repository), please, if it can, add more information about publication process to output

Actions

Copy link

Updated by wilful over 2 years ago

        {                                                                                                                                                               [159/1881]
            "architectures": "amd64",
            "ca_cert": null,
            "client_cert": null,
            "components": null,
            "connect_timeout": null,
            "distributions": "trusty trusty-backports trusty-proposed trusty-security trusty-updates focal focal-backports focal-proposed focal-security focal-updates",
            "download_concurrency": null,
            "gpgkey": null,
            "headers": null,
            "ignore_missing_package_indices": false,
            "max_retries": null,
            "name": "archive.ubuntu.com",
            "policy": "on_demand",
            "proxy_url": null,
            "pulp_created": "2021-08-12T06:57:21.826903Z",
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/248db97b-b81a-4be2-9048-faf441b3dfaa/",
            "pulp_labels": {},
            "pulp_last_updated": "2021-08-12T06:57:21.826926Z",
            "rate_limit": null,
            "sock_connect_timeout": null,
            "sock_read_timeout": null,
            "sync_installer": false,
            "sync_sources": false,
            "sync_udebs": false,
            "tls_validation": true,
            "total_timeout": null,
            "url": "http://archive.ubuntu.com/ubuntu/"
        },

Actions

Copy link

Updated by quba42 over 2 years ago

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

Actions

Copy link

Updated by wilful over 2 years ago

Postgres starting only SELECT

postgres 124116 94.5  7.5 3829508 1234272 ?     Rs   07:50 407:09 postgres: pulp pulp 127.0.0.1(4810) SELECT

Actions

Copy link

Updated by wilful over 2 years ago

quba42 wrote:

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

Sorry, what is mean "What publisher"? I am using one what have in API instruction for example: /pulp/api/v3/docs/#operation/publications_deb_apt_create

My goal make local mirror repository as http://archive.ubuntu.com/ubuntu/ This repository now have all 10 parts in one (before it was different by domain)

Actions

Copy link

Updated by quba42 over 2 years ago

If you are specifically trying to mirror, consider using the Verbatim publisher, rather than the APT publisher. It should be more efficient in the publication since it does not need to generate any metadata.

See the high level documentation for both types of publication:

Or see the relevant API docs: