Project

Profile

Help

Issue #8275

closed

ComplexRepoMigration fails with "file doesn't exists or not a regular file"

Added by laugmanuel@gmail.com about 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 101
Quarter:

Description

Hello,

currently I'm trying to migrate RPM content from Pulp2 to Pulp3 using the migration plugin. The migration plan is generated by Katello (everything is part of a Katello installation).

The content migration seems to break during publishing/distributor creation with File icons.tar.gz doesn't exists or not a regular file.

There are three tasks having the same problem; all seem to be related to a CentOS mirror (although I'm not 100% certain on that).

This is the task group:

 root@pulp:/var/lib/pulp# http --cert /etc/pki/katello/certs/pulp-client.crt --cert-key /etc/pki/katello/private/pulp-client.key "https://$(hostname -f)/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 275
Content-Type: application/json
Date: Thu, 18 Feb 2021 10:17:35 GMT
Keep-Alive: timeout=15, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 pulp.example.com
X-Frame-Options: SAMEORIGIN

{
    "all_tasks_dispatched": true,
    "canceled": 0,
    "completed": 74,
    "description": "Migration Sub-tasks",
    "failed": 3,
    "group_progress_reports": [
        {
            "code": "create.repo_version",
            "done": 0,
            "message": "Repo version creation",
            "suffix": null,
            "total": 0
        },
        {
            "code": "create.distribution",
            "done": 0,
            "message": "Distribution creation",
            "suffix": null,
            "total": 3
        }
    ],
    "pulp_href": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
    "running": 0,
    "skipped": 0,
    "waiting": 0
}

And here is the output of the failed tasks inside the task group:

 root@pulp:/var/lib/pulp# http --cert /etc/pki/katello/certs/pulp-client.crt --cert-key /etc/pki/katello/private/pulp-client.key "https://$(hostname -f)/pulp/api/v3/tasks/?state=failed&task_group=/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 1071
Content-Type: application/json
Date: Thu, 18 Feb 2021 10:15:50 GMT
Keep-Alive: timeout=15, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie,Accept-Encoding
Via: 1.1 pulp.example.com
X-Frame-Options: SAMEORIGIN

{
    "count": 3,
    "next": null,
    "previous": null,
    "results": [
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:20.595544Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:06:01.215851Z",
            "pulp_href": "/pulp/api/v3/tasks/e5259ac1-8e31-4c69-908e-d7c4ee93db9c/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/1fa2b31f-be32-4b5a-94eb-026539d42f98/"
            ],
            "started_at": "2021-02-17T16:06:08.142256Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/559e24b9-912e-425a-85b0-f70775a17138/"
        },
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:25.453339Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:05:59.983297Z",
            "pulp_href": "/pulp/api/v3/tasks/c2400638-fc38-421f-860e-e60cc861bcb8/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/3b8f140d-2781-471c-aa28-d38537e3c545/"
            ],
            "started_at": "2021-02-17T16:06:02.210939Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/2873b7db-1d77-4c5f-99dd-9fa408aacb3a/"
        },
        {
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "File icons.tar.gz doesn't exists or not a regular file",
                "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 277, in complex_repo_migration\n    migrated_repo.pulp3_repository_version\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 401, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 76, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 290, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 486, in create_repomd_xml\n    record.fill(checksum_type)\n"
            },
            "finished_at": "2021-02-17T16:06:23.617334Z",
            "name": "pulp_2to3_migration.app.migration.complex_repo_migration",
            "parent_task": "/pulp/api/v3/tasks/23e52c2e-1b99-4ee8-b491-c0b2aead656d/",
            "progress_reports": [],
            "pulp_created": "2021-02-17T16:05:59.487062Z",
            "pulp_href": "/pulp/api/v3/tasks/bd51d20a-fdaa-463d-9232-fc82f91ddd5b/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/rpm/rpm/a9fe0b38-2440-498c-b4ed-e3523129c0b9/"
            ],
            "started_at": "2021-02-17T16:06:00.325158Z",
            "state": "failed",
            "task_group": "/pulp/api/v3/task-groups/5d1dc966-ec76-4cd4-bb9b-320796808fa6/",
            "worker": "/pulp/api/v3/workers/14cfcce7-4444-4a13-aa11-aacf404ac93e/"
        }
    ]
}

Versions used:

 root@pulp:/var/lib/pulp# rpm -qi pulp-server | head -n3
Name        : pulp-server
Version     : 2.21.5
Release     : 1.el7

 root@pulp:/var/lib/pulp# http "https://$(hostname -f)/pulp/api/v3/status/" | jq '.versions'
[
  {
    "component": "pulpcore",
    "version": "3.7.3"
  },
  {
    "component": "pulp_2to3_migration",
    "version": "0.6.0"
  },
  {
    "component": "pulp_rpm",
    "version": "3.7.0"
  },
  {
    "component": "pulp_file",
    "version": "1.3.0"
  },
  {
    "component": "pulp_deb",
    "version": "2.7.0"
  },
  {
    "component": "pulp_container",
    "version": "2.1.0"
  },
  {
    "component": "pulp_certguard",
    "version": "1.0.3"
  }
]

Related issues

Related to RPM Support - Issue #9096: Sync of SUSE, post-2to3-migration, exposes a bug in resolve_advisory_conflict()CLOSED - CURRENTRELEASEggaineyActions
Related to RPM Support - Backport #9113: Backport 8275 to pulp_rpm 3.11CLOSED - CURRENTRELEASEttereshc

Actions
Actions #1

Updated by laugmanuel@gmail.com about 3 years ago

I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side. The problematic repos had metadata files which seem to have caused this problem:

However, finding that information was not trivial at all. I had to add some debug output to pulp_rpm/app/tasks/publishing.py:483 and log the name and publication.repository.name variables to find the problematic repos and metadata file/type.

 root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          appdata-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          app-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
                    d318
Checksum Type:      sha256
Data Type:          appdata
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

After deleting the problematic metadata file from Pulp2, the migration continued successfully:

 root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0 --in "data_type=app-icons"
Actions #2

Updated by laugmanuel@gmail.com about 3 years ago

Just to give the full information:

The problematic repos were:

  • 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Debuginfo-Updates for sle-12-x86_64
  • 1878 SUSE Linux Enterprise Server 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64
  • 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64

I guess this problem is relatively specific to these repos.

Actions #3

Updated by ttereshc about 3 years ago

Thank you very much for the detailed report and debugging. I'm glad you were able to resolve it.

I'm noting 2 things for Pulp to fix:

  • it looks like a bug to me that during a Pulp 2to3 migration, one can't publish those files, they are just custom files in repodata and Pulp should handle those. It's a bug either in a migration plugin or in RPM plugin (in case the same error appears during sync/publish workflow in Pulp3).
  • the error message needs to be expanded to have more details to locate the problematic file.
Actions #4

Updated by ttereshc about 3 years ago

  • Triaged changed from No to Yes
Actions #5

Updated by pabloalcantara almost 3 years ago

Hi, It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem? Thanks a lot wrote:

I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side. The problematic repos had metadata files which seem to have caused this problem:

However, finding that information was not trivial at all. I had to add some debug output to pulp_rpm/app/tasks/publishing.py:483 and log the name and publication.repository.name variables to find the problematic repos and metadata file/type.

 root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          appdata-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          app-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
                    d318
Checksum Type:      sha256
Data Type:          appdata
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

After deleting the problematic metadata file from Pulp2, the migration continued successfully:

 root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0 --in "data_type=app-icons"
Actions #6

Updated by laugmanuel@gmail.com almost 3 years ago

Hi,

I've basically replaced https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/publishing.py#L552 with the following:

try:
    record.fill(checksum_type)
except:
    with open('/tmp/pulp3_publish_task.log', 'a') as the_file:
        the_file.write("File: %s, Path: %s, DB to update: %s, Repodata path: %s, Record: %s, Location href: %s, OS Join Path: %s, Repomd Path: %s, Repo: %s\n" %
          (name, path, db_to_update, repodata_path, record, record.location_href, os.path.join(repodata_path), repomd_path, publication.repository.name))

The output was something like:

File: app-icons, Path: icons.tar.gz, DB to update: None, Repodata path: repodata, Record: <createrepo_c.RepomdRecord app-icons object>, Location href: repodata/icons.tar.gz, OS Join Path: repodata, Repomd Path: /var/lib/pulp/tmp/120233@pulp.example.com/fa077a59-d492-4df6-9f71-7778d7957200/repomd.xml, Repo: 9995b83f-0162-45de-9725-dae79b1eb8b0

The given repo id was the interesting part to find the problematic Pulp2 repo (as seen above). I hope this helps.

Manuel

pabloalcantara wrote:

Hi, It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem? Thanks a lot

Actions #7

Updated by ttereshc almost 3 years ago

  • Tags Katello added
Actions #8

Updated by dalley almost 3 years ago

  • Priority changed from Normal to High
  • Sprint set to Sprint 97
Actions #9

Updated by ttereshc almost 3 years ago

Priority went up because there are more reports for this issue, outside this tracker as well. See repos to test with in the comment#2.

Actions #11

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #12

Updated by ttereshc almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions #13

Updated by ttereshc almost 3 years ago

Reproduced it locally by adding a tar.gz file to a pulp 2 repo and running a migration.
Smells like a pulp_rpm problem, potentially a createrepo_c issue.
I do not think it is migration related, I expect the same issue to see with sync/publish in Pulp 3.

If someone with access to SLES repos can sync into Pulp 3 one of the repos from comment 2, it will be highly appreciated.

Actions #14

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #15

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #16

Updated by dalley over 2 years ago

ttereshc, ggainey has access to SUSE repos

Actions #17

Updated by dalley over 2 years ago

  • Assignee changed from ttereshc to ggainey
Actions #18

Updated by ggainey over 2 years ago

I haven't recreated the icons.tar.gz problem yet, but I had had issues post-2to3 w/ one of the repos from #c2.

Sync'd into Pulp2, migrated, resync'd, FAIL:

CONFIG="{\"query_auth_token\": \"_INSERT_REPO_TOKEN_HERE\" }"
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id sles12-update-sp5
  --feed https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/ \
  --download-policy on_demand
http --verify no --auth admin:admin \
  PUT https://localhost/pulp/api/v2/repositories/sles12-update-sp5  importer_config:="${CONFIG}"
pulp-admin rpm repo sync run --repo-id sles12-update-sp5

PLAN_HREF=$(pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}' | jq -r .pulp_href)
echo "Plan: $PLAN_HREF"
pulp -T 0 migration plan run --href ${PLAN_HREF}

REMOTE_HREF=$(pulp rpm repository show --name sles12-update-sp5 | jq -r .remote)  
echo "Remote-href: ${REMOTE_HREF}"
pulp rpm repository sync --name sles12-update-sp5
...
sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token : 
Remote-href: /pulp/api/v3/remotes/rpm/rpm/339d06e9-59dd-447f-b1f2-7bf065b9bfe9/
Started background task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/
......................................................................................................Error: Task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/ failed: ''<' not supported between instances of 'datetime.datetime' and 'NoneType''
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm remote list

Use same remote to sync directly to an empty Pulp3 repo, SUCCEED:

pulp rpm repository create --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm repository sync --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
Started background task /pulp/api/v3/tasks/18a5773e-f979-4aa3-91a5-bf45f0aba044/
.......................................................................................................Done.

Other SUSE repos migrate-and-sync successfully. I'll continue the experiments with the remaining repos from #c2 and report back here with results.

Actions #19

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #20

Updated by hellokatello123 over 2 years ago

Hi,

I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:

2021-07-15 21:49:07 +0000: Distribution creation 5704/5711Migration failed, You will want to investigate: https://katello.a.space.corp/foreman_tasks/tasks/8228cbae-de57-4866-bbfb-cc28fce8b7bb
rake aborted!
ForemanTasks::TaskError: Task 8228cbae-de57-4866-bbfb-cc28fce8b7bb: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

The reason why I started to migrate was that I was having problems with a SLES 15 content view. I was using an rpm include filter and the option to resolve deps on the CV to limit the number of packages I have in this CV. Basically I have 10-15 dummy rpm packges that only have deps. All of these deps should be resolved and only these packages should be available in the CV. As I kept having varying results over and over again I had the feeling that this was a bug and it might be releated especially to the SLES 15 repos. I use the SCC plugin in Katello to sync the repos. I wrote to foreman commnunity:

https://community.theforeman.org/t/enable-solve-dependencies-for-content-views/20721/13

Where they suggest to me to upgrade to pulp3. Then I stumbled over this bug report and I have the feeling that my initial problem is related to this one. I'm no expert in pulp, but If someone could give me a hint how to debug this further, I',m more then willing to do so.

Best Regards, Oliver

Actions #21

Updated by ggainey over 2 years ago

The problem in #c18 looks more like a pulp_rpm issue, opened https://pulp.plan.io/issues/9096 for it.

Actions #22

Updated by ggainey over 2 years ago

hellokatello123 wrote:

Hi,

I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:

Hey Oliver - what does "journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00' " tell you?

There are def some SUSE repos that break some of Pulp3's assumptions about "sane repositories"; I'm working on trying to address them e'en as I type, more info would be great!

G

Actions #23

Updated by hellokatello123 over 2 years ago

Hi,

here is the output:

[root@katello ~]# journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00'
-- Logs begin at Thu 2021-07-15 15:28:40 UTC, end at Fri 2021-07-16 15:38:00 UTC. --
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (33440ee4-f5b0-4046-9635-5259a884fc36)
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: Cleaning registries for queue: 1127@katello.a.space.corp
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: 4270f7a1-8a22-4f03-9c5f-0061179aa8c5
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (4270f7a1-8a22-4f03-9c5f-0061179aa8c5)
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:49:38 katello.a.space.corp pulpcore-worker-8[1141]: pulp: rq.worker:INFO: Cleaning registries for queue: 1141@katello.a.space.corp
Jul 15 21:49:40 katello.a.space.corp pulpcore-worker-3[1136]: pulp: rq.worker:INFO: Cleaning registries for queue: 1136@katello.a.space.corp
Actions #24

Updated by ggainey over 2 years ago

Syncing into a clean Pulp3 database (ie, not post-migration) recreates the icons.tar.gz problem:

sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/edbbba67-d0d1-44e1-95df-d4196577cd4e/"
"/pulp/api/v3/repositories/rpm/rpm/df2e38e2-e512-4cea-baca-620508df5871/"
Started background task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/
.....................................................................Error: Task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/ failed: 'File icons.tar.gz doesn't exists or not a regular file'
sles12-debug-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/34e45336-b432-44a5-ac6d-86183817b321/"
"/pulp/api/v3/repositories/rpm/rpm/941b278d-31a6-4180-8fea-9f8ab0a091be/"
Started background task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/
...................................Error: Task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/ failed: 'File icons.tar.gz doesn't exists or not a regular file'

I added some debug code to capture the actual stacktrace - createrepo_c gets Upset when asked to compute the checksum of a file that doesn't exist:

Jul 16 16:52:11 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2977]: cr_repomd_record_fill: File icons.tar.gz doesn't exists
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO: Task c9e06f22-4b96-4614-84c5-7874429f5d3f failed (File icons.tar.gz doesn't exists or not a regular file)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 268, in _perform_task
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: result = func(*args, **kwargs)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 422, in synchronize
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: version = dv.create()
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: loop.run_until_complete(pipeline)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 987, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: repository.on_new_version(self)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py", line 288, in on_new_version
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: sqlite_metadata=self.sqlite_metadata,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 366, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: return publication
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/publication.py", line 183, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: raise exc_val.with_traceback(exc_tb)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 345, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: metadata_signing_service=metadata_signing_service,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 584, in generate_repo_metadata
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: record.fill(checksum_type)

The problem is that extra-repomd files don't know they live in the worker's CWD, and can't be found. Investigating for a fix.

Actions #25

Updated by ggainey over 2 years ago

  • Project changed from Migration Plugin to RPM Support

SUSE repos are Special, and pulp_rpm has some questionable heuristics in code. PR shortly.

Actions #26

Updated by pulpbot over 2 years ago

  • Status changed from ASSIGNED to POST
Actions #27

Updated by dalley over 2 years ago

  • Sprint/Milestone set to 3.14.0
Actions #28

Updated by dalley over 2 years ago

  • Related to Issue #9096: Sync of SUSE, post-2to3-migration, exposes a bug in resolve_advisory_conflict() added
Actions #29

Updated by ggainey over 2 years ago

Added by ggainey over 2 years ago

Revision 47584d3c | View on GitHub

Adapted to some (more) SUSE repo edge-cases.

  • Some metadata-files can have '-' in the filename
  • Some advisory datetimes are just timestamps.

fixes #8275 fixes #9096

Actions #30

Updated by ggainey over 2 years ago

  • Status changed from POST to MODIFIED
Actions #31

Updated by ggainey over 2 years ago

The repo-urls that failed on icons.tar.gz, that I have access to for SUSE, included:

https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug

(My dev-acct doesn't appear to have access to SAP repos so I couldn't test those)

The failure, both when migrating and with a sync-into-Pulp3, was the one noted in the description of "File icons.tar.gz doesn't exists or not a regular file"

Sync-and-migrate test is here: https://github.com/ggainey/pulp_startup/blob/main/rel_path/sync_and_migrate.sh

Actions #32

Updated by hellokatello123 over 2 years ago

Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?

Best Regards

Actions #33

Updated by ggainey over 2 years ago

hellokatello123 wrote:

Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?

Best Regards

It's in Pulp3. The error lies in how Pulp3/rpm treats extra-repomd-files, that have dashes in their filename, in sync code that is shared between migration and 'regular' sync. I can recreate the problem just syncing SUSE into a "clean" pulp3 instance.

Actions #34

Updated by hellokatello123 over 2 years ago

Hi,

thanks a lot for the clarification. Then I will have to wait for a fix. :)

Cheers, Oliver

Actions #35

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF