Project

Profile

Help

Issue #8275

ComplexRepoMigration fails with "file doesn't exists or not a regular file"

Added by laugmanuel@gmail.com 13 days ago. Updated 12 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Hello,

currently I'm trying to migrate RPM content from Pulp2 to Pulp3 using the migration plugin. The migration plan is generated by Katello (everything is part of a Katello installation).

The content migration seems to break during publishing/distributor creation with File icons.tar.gz doesn't exists or not a regular file.

There are three tasks having the same problem; all seem to be related to a CentOS mirror (although I'm not 100% certain on that).

This is the task group:

 root@pulp:/var/lib/pulp# http --cert /etc/pki/katello/certs/pulp-client.crt --cert-key /etc/pki/katello/private/pulp-client.key "https://$(hostname -f)/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 275
Content-Type: application/json
Date: Thu, 18 Feb 2021 10:17:35 GMT
Keep-Alive: timeout=15, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 pulp.example.com
X-Frame-Options: SAMEORIGIN

{
    "all_tasks_dispatched": true,
    "canceled": 0,
    "completed": 74,
    "description": "Migration Sub-tasks",
    "failed": 3,
    "group_progress_reports": [
        {
            "code": "create.repo_version",
            "done": 0,
            "message": "Repo version creation",
            "suffix": null,
            "total": 0
        },
        {
            "code": "create.distribution",
            "done": 0,
            "message": "Distribution creation",
            "suffix": null,
            "total": 3
        }
    ],
    "pulp_href": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
    "running": 0,
    "skipped": 0,
    "waiting": 0
}

And here is the output of the failed tasks inside the task group:

 root@pulp:/var/lib/pulp# http --cert /etc/pki/katello/certs/pulp-client.crt --cert-key /etc/pki/katello/private/pulp-client.key "https://$(hostname -f)/pulp/api/v3/tasks/?state=failed&task_group=/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 1071
Content-Type: application/json
Date: Thu, 18 Feb 2021 10:15:50 GMT
Keep-Alive: timeout=15, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 pulp.example.com
X-Frame-Options: SAMEORIGIN

{
    "count": 3,
    "next": null,
    "previous": null,
    "results": [
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:20.595544Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:06:01.215851Z",
            "pulp_href": "/pulp/api/v3/tasks/e5259ac1-8e31-4c69-908e-d7c4ee93db9c/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/1fa2b31f-be32-4b5a-94eb-026539d42f98/"
            ],
            "started_at": "2021-02-17T16:06:08.142256Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/559e24b9-912e-425a-85b0-f70775a17138/"
        },
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:25.453339Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:05:59.983297Z",
            "pulp_href": "/pulp/api/v3/tasks/c2400638-fc38-421f-860e-e60cc861bcb8/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/3b8f140d-2781-471c-aa28-d38537e3c545/"
            ],
            "started_at": "2021-02-17T16:06:02.210939Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/2873b7db-1d77-4c5f-99dd-9fa408aacb3a/"
        },
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:23.617334Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:05:59.487062Z",
            "pulp_href": "/pulp/api/v3/tasks/bd51d20a-fdaa-463d-9232-fc82f91ddd5b/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/a9fe0b38-2440-498c-b4ed-e3523129c0b9/"
            ],
            "started_at": "2021-02-17T16:06:00.325158Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/14cfcce7-4444-4a13-aa11-aacf404ac93e/"
        }
    ]
}

Versions used:

 root@pulp:/var/lib/pulp# rpm -qi pulp-server | head -n3
Name        : pulp-server
Version     : 2.21.5
Release     : 1.el7

 root@pulp:/var/lib/pulp# http "https://$(hostname -f)/pulp/api/v3/status/" | jq '.versions'
[
  {
    "component": "pulpcore",
    "version": "3.7.3"
  },
  {
    "component": "pulp_2to3_migration",
    "version": "0.6.0"
  },
  {
    "component": "pulp_rpm",
    "version": "3.7.0"
  },
  {
    "component": "pulp_file",
    "version": "1.3.0"
  },
  {
    "component": "pulp_deb",
    "version": "2.7.0"
  },
  {
    "component": "pulp_container",
    "version": "2.1.0"
  },
  {
    "component": "pulp_certguard",
    "version": "1.0.3"
  }
]

History

#1 Updated by laugmanuel@gmail.com 12 days ago

I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side. The problematic repos had metadata files which seem to have caused this problem:

However, finding that information was not trivial at all. I had to add some debug output to pulp_rpm/app/tasks/publishing.py:483 and log the name and publication.repository.name variables to find the problematic repos and metadata file/type.

 root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          appdata-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          app-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
                    d318
Checksum Type:      sha256
Data Type:          appdata
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

After deleting the problematic metadata file from Pulp2, the migration continued successfully:

 root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0 --in "data_type=app-icons"

#2 Updated by laugmanuel@gmail.com 12 days ago

Just to give the full information:

The problematic repos were:

  • 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Debuginfo-Updates for sle-12-x86_64
  • 1878 SUSE Linux Enterprise Server 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64
  • 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64

I guess this problem is relatively specific to these repos.

#3 Updated by ttereshc 12 days ago

Thank you very much for the detailed report and debugging. I'm glad you were able to resolve it.

I'm noting 2 things for Pulp to fix:

  • it looks like a bug to me that during a Pulp 2to3 migration, one can't publish those files, they are just custom files in repodata and Pulp should handle those. It's a bug either in a migration plugin or in RPM plugin (in case the same error appears during sync/publish workflow in Pulp3).
  • the error message needs to be expanded to have more details to locate the problematic file.

#4 Updated by ttereshc 12 days ago

  • Triaged changed from No to Yes

Please register to edit this issue

Also available in: Atom PDF