Project

Profile

Help

Issue #9237

Very slowly publication and progress report

Added by wilful 2 months ago. Updated 2 months ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Debian:
Platform Release:
Target Release - Debian:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Today i started mirroring for http://archive.ubuntu.com

# pulp task list --state running | jq
[
  {
    "pulp_href": "/pulp/api/v3/tasks/d41cd931-9d7b-4ead-a381-d6aae8c709e0/",
    "pulp_created": "2021-08-12T07:50:28.476170Z",
    "state": "running",
    "name": "pulp_deb.app.tasks.publishing.publish",
    "logging_cid": "60c1f76bef074040943b9ed4558a8732",
    "started_at": "2021-08-12T07:50:28.621880Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/b64aee98-0684-4daf-9b67-6f8a4aa36370/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      null
    ],
    "reserved_resources_record": [
      "/pulp/api/v3/repositories/deb/apt/a1a61370-650c-4d9f-9c28-79e444110dda/"
    ]
  }
]

And they still running, i can't understand how long me need waiting to complete.

  "versions": [
    {
      "component": "core",
      "version": "3.14.3"
    },
    {
      "component": "rpm",
      "version": "3.14.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.1"
    },
    {
      "component": "deb",
      "version": "2.14.1"
    },
    {
      "component": "container",
      "version": "2.7.0"
    }
  ],

I think we have data for specify progress (count of contents and count packages in repository), please, if it can, add more information about publication process to output

History

#1 Updated by wilful 2 months ago

        {                                                                                                                                                               [159/1881]
            "architectures": "amd64",
            "ca_cert": null,
            "client_cert": null,
            "components": null,
            "connect_timeout": null,
            "distributions": "trusty trusty-backports trusty-proposed trusty-security trusty-updates focal focal-backports focal-proposed focal-security focal-updates",
            "download_concurrency": null,
            "gpgkey": null,
            "headers": null,
            "ignore_missing_package_indices": false,
            "max_retries": null,
            "name": "archive.ubuntu.com",
            "policy": "on_demand",
            "proxy_url": null,
            "pulp_created": "2021-08-12T06:57:21.826903Z",
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/248db97b-b81a-4be2-9048-faf441b3dfaa/",
            "pulp_labels": {},
            "pulp_last_updated": "2021-08-12T06:57:21.826926Z",
            "rate_limit": null,
            "sock_connect_timeout": null,
            "sock_read_timeout": null,
            "sync_installer": false,
            "sync_sources": false,
            "sync_udebs": false,
            "tls_validation": true,
            "total_timeout": null,
            "url": "http://archive.ubuntu.com/ubuntu/"
        },

#2 Updated by quba42 2 months ago

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

#3 Updated by wilful 2 months ago

Postgres starting only SELECT

postgres 124116 94.5  7.5 3829508 1234272 ?     Rs   07:50 407:09 postgres: pulp pulp 127.0.0.1(4810) SELECT

#4 Updated by wilful 2 months ago

quba42 wrote:

What publisher are you using with what options? And what is your goal. Synchronizing 10 large Ubuntu distributions into a single repository is about as resource intensive as it is possible to go and most likely untested.

Sorry, what is mean "What publisher"? I am using one what have in API instruction for example: /pulp/api/v3/docs/#operation/publications_deb_apt_create

My goal make local mirror repository as http://archive.ubuntu.com/ubuntu/ This repository now have all 10 parts in one (before it was different by domain)

#5 Updated by quba42 2 months ago

If you are specifically trying to mirror, consider using the Verbatim publisher, rather than the APT publisher. It should be more efficient in the publication since it does not need to generate any metadata.

See the high level documentation for both types of publication:

Or see the relevant API docs:

#6 Updated by wilful 2 months ago

thanks, i going to try Verbatim publisher (or split repositories by dist), may be it help me create identical mirror as Ubuntu.

Please register to edit this issue

Also available in: Atom PDF