RPM Support

I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side. The problematic repos had metadata files which seem to have caused this problem:

However, finding that information was not trivial at all. I had to add some debug output to pulp_rpm/app/tasks/publishing.py:483 and log the name and publication.repository.name variables to find the problematic repos and metadata file/type.

 root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          appdata-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          app-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
                    d318
Checksum Type:      sha256
Data Type:          appdata
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

After deleting the problematic metadata file from Pulp2, the migration continued successfully:

 root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0 --in "data_type=app-icons"

Just to give the full information:

The problematic repos were:

• 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Debuginfo-Updates for sle-12-x86_64
• 1878 SUSE Linux Enterprise Server 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64
• 1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64

I guess this problem is relatively specific to these repos.

Thank you very much for the detailed report and debugging. I'm glad you were able to resolve it.

I'm noting 2 things for Pulp to fix:

• it looks like a bug to me that during a Pulp 2to3 migration, one can't publish those files, they are just custom files in repodata and Pulp should handle those. It's a bug either in a migration plugin or in RPM plugin (in case the same error appears during sync/publish workflow in Pulp3).
• the error message needs to be expanded to have more details to locate the problematic file.

Hi, It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem? Thanks a lot laugmanuel@gmail.com wrote:

I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side. The problematic repos had metadata files which seem to have caused this problem:

However, finding that information was not trivial at all. I had to add some debug output to pulp_rpm/app/tasks/publishing.py:483 and log the name and publication.repository.name variables to find the problematic repos and metadata file/type.

 root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          appdata-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
                    f400
Checksum Type:      sha256
Data Type:          app-icons
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

Checksum:           c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
                    d318
Checksum Type:      sha256
Data Type:          appdata
Downloaded:         True
Pulp User Metadata:
Repo Id:            9995b83f-0162-45de-9725-dae79b1eb8b0

After deleting the problematic metadata file from Pulp2, the migration continued successfully:

 root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile --repo-id 9995b83f-0162-45de-9725-dae79b1eb8b0 --in "data_type=app-icons"

Hi,

I've basically replaced https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/publishing.py#L552 with the following:

try:
    record.fill(checksum_type)
except:
    with open('/tmp/pulp3_publish_task.log', 'a') as the_file:
        the_file.write("File: %s, Path: %s, DB to update: %s, Repodata path: %s, Record: %s, Location href: %s, OS Join Path: %s, Repomd Path: %s, Repo: %s\n" %
          (name, path, db_to_update, repodata_path, record, record.location_href, os.path.join(repodata_path), repomd_path, publication.repository.name))

The output was something like:

File: app-icons, Path: icons.tar.gz, DB to update: None, Repodata path: repodata, Record: <createrepo_c.RepomdRecord app-icons object>, Location href: repodata/icons.tar.gz, OS Join Path: repodata, Repomd Path: /var/lib/pulp/tmp/120233@pulp.example.com/fa077a59-d492-4df6-9f71-7778d7957200/repomd.xml, Repo: 9995b83f-0162-45de-9725-dae79b1eb8b0

The given repo id was the interesting part to find the problematic Pulp2 repo (as seen above). I hope this helps.

Manuel

pabloalcantara wrote:

Hi, It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem? Thanks a lot

Priority went up because there are more reports for this issue, outside this tracker as well. See repos to test with in the comment#2.

relevant katello posts:

https://community.theforeman.org/t/pulp-2-to-pulp-3-migration-failed/23702/10

https://community.theforeman.org/t/pulp3-migration-failed/22473/33

https://community.theforeman.org/t/katello-3-18-2-pulp-2to3-migration-error/23659

Reproduced it locally by adding a tar.gz file to a pulp 2 repo and running a migration.
Smells like a pulp_rpm problem, potentially a createrepo_c issue.
I do not think it is migration related, I expect the same issue to see with sync/publish in Pulp 3.

If someone with access to SLES repos can sync into Pulp 3 one of the repos from comment 2, it will be highly appreciated.

ttereshc, ggainey has access to SUSE repos

I haven't recreated the icons.tar.gz problem yet, but I had had issues post-2to3 w/ one of the repos from #c2.

Sync'd into Pulp2, migrated, resync'd, FAIL:

CONFIG="{\"query_auth_token\": \"_INSERT_REPO_TOKEN_HERE\" }"
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id sles12-update-sp5
  --feed https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/ \
  --download-policy on_demand
http --verify no --auth admin:admin \
  PUT https://localhost/pulp/api/v2/repositories/sles12-update-sp5  importer_config:="${CONFIG}"
pulp-admin rpm repo sync run --repo-id sles12-update-sp5

PLAN_HREF=$(pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}' | jq -r .pulp_href)
echo "Plan: $PLAN_HREF"
pulp -T 0 migration plan run --href ${PLAN_HREF}

REMOTE_HREF=$(pulp rpm repository show --name sles12-update-sp5 | jq -r .remote)  
echo "Remote-href: ${REMOTE_HREF}"
pulp rpm repository sync --name sles12-update-sp5
...
sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token : 
Remote-href: /pulp/api/v3/remotes/rpm/rpm/339d06e9-59dd-447f-b1f2-7bf065b9bfe9/
Started background task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/
......................................................................................................Error: Task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/ failed: ''<' not supported between instances of 'datetime.datetime' and 'NoneType''
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm remote list

Use same remote to sync directly to an empty Pulp3 repo, SUCCEED:

pulp rpm repository create --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm repository sync --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
Started background task /pulp/api/v3/tasks/18a5773e-f979-4aa3-91a5-bf45f0aba044/
.......................................................................................................Done.

Other SUSE repos migrate-and-sync successfully. I'll continue the experiments with the remaining repos from #c2 and report back here with results.

Hi,

I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:

2021-07-15 21:49:07 +0000: Distribution creation 5704/5711Migration failed, You will want to investigate: https://katello.a.space.corp/foreman_tasks/tasks/8228cbae-de57-4866-bbfb-cc28fce8b7bb
rake aborted!
ForemanTasks::TaskError: Task 8228cbae-de57-4866-bbfb-cc28fce8b7bb: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

The reason why I started to migrate was that I was having problems with a SLES 15 content view. I was using an rpm include filter and the option to resolve deps on the CV to limit the number of packages I have in this CV. Basically I have 10-15 dummy rpm packges that only have deps. All of these deps should be resolved and only these packages should be available in the CV. As I kept having varying results over and over again I had the feeling that this was a bug and it might be releated especially to the SLES 15 repos. I use the SCC plugin in Katello to sync the repos. I wrote to foreman commnunity:

https://community.theforeman.org/t/enable-solve-dependencies-for-content-views/20721/13

Where they suggest to me to upgrade to pulp3. Then I stumbled over this bug report and I have the feeling that my initial problem is related to this one. I'm no expert in pulp, but If someone could give me a hint how to debug this further, I',m more then willing to do so.

Best Regards, Oliver

The problem in #c18 looks more like a pulp_rpm issue, opened https://pulp.plan.io/issues/9096 for it.

hellokatello123 wrote:

Hi,

I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:

Hey Oliver - what does "journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00' " tell you?

There are def some SUSE repos that break some of Pulp3's assumptions about "sane repositories"; I'm working on trying to address them e'en as I type, more info would be great!

G

Hi,

here is the output:

[root@katello ~]# journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00'
-- Logs begin at Thu 2021-07-15 15:28:40 UTC, end at Fri 2021-07-16 15:38:00 UTC. --
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (33440ee4-f5b0-4046-9635-5259a884fc36)
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: Cleaning registries for queue: 1127@katello.a.space.corp
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: 4270f7a1-8a22-4f03-9c5f-0061179aa8c5
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (4270f7a1-8a22-4f03-9c5f-0061179aa8c5)
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:49:38 katello.a.space.corp pulpcore-worker-8[1141]: pulp: rq.worker:INFO: Cleaning registries for queue: 1141@katello.a.space.corp
Jul 15 21:49:40 katello.a.space.corp pulpcore-worker-3[1136]: pulp: rq.worker:INFO: Cleaning registries for queue: 1136@katello.a.space.corp

Syncing into a clean Pulp3 database (ie, not post-migration) recreates the icons.tar.gz problem:

sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/edbbba67-d0d1-44e1-95df-d4196577cd4e/"
"/pulp/api/v3/repositories/rpm/rpm/df2e38e2-e512-4cea-baca-620508df5871/"
Started background task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/
.....................................................................Error: Task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/ failed: 'File icons.tar.gz doesn't exists or not a regular file'
sles12-debug-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/34e45336-b432-44a5-ac6d-86183817b321/"
"/pulp/api/v3/repositories/rpm/rpm/941b278d-31a6-4180-8fea-9f8ab0a091be/"
Started background task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/
...................................Error: Task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/ failed: 'File icons.tar.gz doesn't exists or not a regular file'

I added some debug code to capture the actual stacktrace - createrepo_c gets Upset when asked to compute the checksum of a file that doesn't exist:

Jul 16 16:52:11 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2977]: cr_repomd_record_fill: File icons.tar.gz doesn't exists
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO: Task c9e06f22-4b96-4614-84c5-7874429f5d3f failed (File icons.tar.gz doesn't exists or not a regular file)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 268, in _perform_task
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: result = func(*args, **kwargs)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 422, in synchronize
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: version = dv.create()
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: loop.run_until_complete(pipeline)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 987, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: repository.on_new_version(self)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py", line 288, in on_new_version
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: sqlite_metadata=self.sqlite_metadata,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 366, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: return publication
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/publication.py", line 183, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: raise exc_val.with_traceback(exc_tb)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 345, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: metadata_signing_service=metadata_signing_service,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 584, in generate_repo_metadata
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: record.fill(checksum_type)

The problem is that extra-repomd files don't know they live in the worker's CWD, and can't be found. Investigating for a fix.

The repo-urls that failed on icons.tar.gz, that I have access to for SUSE, included:

https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug

(My dev-acct doesn't appear to have access to SAP repos so I couldn't test those)

The failure, both when migrating and with a sync-into-Pulp3, was the one noted in the description of "File icons.tar.gz doesn't exists or not a regular file"

Sync-and-migrate test is here: https://github.com/ggainey/pulp_startup/blob/main/rel_path/sync_and_migrate.sh

Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?

Best Regards

hellokatello123 wrote:

Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?

Best Regards

It's in Pulp3. The error lies in how Pulp3/rpm treats extra-repomd-files, that have dashes in their filename, in sync code that is shared between migration and 'regular' sync. I can recreate the problem just syncing SUSE into a "clean" pulp3 instance.

Hi,

thanks a lot for the clarification. Then I will have to wait for a fix. :)

Cheers, Oliver