Project

Profile

Help

Issue #9237

closed

Very slowly publication and progress report

Added by wilful over 1 year ago. Updated 11 months ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Debian:
Platform Release:
Target Release - Debian:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Ticket moved to GitHub: "pulp/pulp_deb/418":https://github.com/pulp/pulp_deb/issues/418


Today i started mirroring for http://archive.ubuntu.com

# pulp task list --state running | jq
[
  {
    "pulp_href": "/pulp/api/v3/tasks/d41cd931-9d7b-4ead-a381-d6aae8c709e0/",
    "pulp_created": "2021-08-12T07:50:28.476170Z",
    "state": "running",
    "name": "pulp_deb.app.tasks.publishing.publish",
    "logging_cid": "60c1f76bef074040943b9ed4558a8732",
    "started_at": "2021-08-12T07:50:28.621880Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/b64aee98-0684-4daf-9b67-6f8a4aa36370/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      null
    ],
    "reserved_resources_record": [
      "/pulp/api/v3/repositories/deb/apt/a1a61370-650c-4d9f-9c28-79e444110dda/"
    ]
  }
]

And they still running, i can't understand how long me need waiting to complete.

  "versions": [
    {
      "component": "core",
      "version": "3.14.3"
    },
    {
      "component": "rpm",
      "version": "3.14.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.1"
    },
    {
      "component": "deb",
      "version": "2.14.1"
    },
    {
      "component": "container",
      "version": "2.7.0"
    }
  ],

I think we have data for specify progress (count of contents and count packages in repository), please, if it can, add more information about publication process to output

Actions #1

Updated by wilful over 1 year ago

        {                                                                                                                                                               [159/1881]
            "architectures": "amd64",
            "ca_cert": null,
            "client_cert": null,
            "components": null,
            "connect_timeout": null,
            "distributions": "trusty trusty-backports trusty-proposed trusty-security trusty-updates focal focal-backports focal-proposed focal-security focal-updates",
            "download_concurrency": null,
            "gpgkey": null,
            "headers": null,
            "ignore_missing_package_indices": false,
            "max_retries": null,
            "name": "archive.ubuntu.com",
            "policy": "on_demand",
            "proxy_url": null,
            "pulp_created": "2021-08-12T06:57:21.826903Z",
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/248db97b-b81a-4be2-9048-faf441b3dfaa/",
            "pulp_labels": {},
            "pulp_last_updated": "2021-08-12T06:57:21.826926Z",
            "rate_limit": null,
            "sock_connect_timeout": null,
            "sock_read_timeout": null,
            "sync_installer": false,
            "sync_sources": false,
            "sync_udebs": false,
            "tls_validation": true,
            "total_timeout": null,
            "url": "http://archive.ubuntu.com/ubuntu/"
        },
Actions #2

Updated by quba42 over 1 year ago

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

Actions #3

Updated by wilful over 1 year ago

Postgres starting only SELECT

postgres 124116 94.5  7.5 3829508 1234272 ?     Rs   07:50 407:09 postgres: pulp pulp 127.0.0.1(4810) SELECT
Actions #4

Updated by wilful over 1 year ago

quba42 wrote:

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

Sorry, what is mean "What publisher"? I am using one what have in API instruction for example: /pulp/api/v3/docs/#operation/publications_deb_apt_create

My goal make local mirror repository as http://archive.ubuntu.com/ubuntu/ This repository now have all 10 parts in one (before it was different by domain)

Actions #5

Updated by quba42 over 1 year ago

If you are specifically trying to mirror, consider using the Verbatim publisher, rather than the APT publisher. It should be more efficient in the publication since it does not need to generate any metadata.

See the high level documentation for both types of publication:

Or see the relevant API docs:

Actions #6

Updated by wilful over 1 year ago

thanks, i going to try Verbatim publisher (or split repositories by dist), may be it help me create identical mirror as Ubuntu.

Actions #7

Updated by pulpbot 11 months ago

  • Description updated (diff)
  • Status changed from NEW to CLOSED - DUPLICATE

Also available in: Atom PDF