https://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-02-19T08:49:38ZPulpRPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=678552021-02-19T08:49:38Zlaugmanuel@gmail.com
<ul></ul><p>I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side.
The problematic repos had metadata files which seem to have caused this problem:</p>
<p>However, finding that information was not trivial at all. I had to add some debug output to <code>pulp_rpm/app/tasks/publishing.py:483</code> and log the <code>name</code> and <code>publication.repository.name</code> variables to find the problematic repos and metadata file/type.</p>
<pre><code class="sh syntaxhl" data-language="sh"> root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile <span class="nt">--repo-id</span> 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: 746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
f400
Checksum Type: sha256
Data Type: appdata-icons
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: 746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
f400
Checksum Type: sha256
Data Type: app-icons
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
d318
Checksum Type: sha256
Data Type: appdata
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
</code></pre>
<p>After deleting the problematic metadata file from Pulp2, the migration continued successfully:</p>
<pre><code class="sh syntaxhl" data-language="sh"> root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile <span class="nt">--repo-id</span> 9995b83f-0162-45de-9725-dae79b1eb8b0 <span class="nt">--in</span> <span class="s2">"data_type=app-icons"</span>
</code></pre> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=678572021-02-19T09:30:49Zlaugmanuel@gmail.com
<ul></ul><p>Just to give the full information:</p>
<p>The problematic repos were:</p>
<ul>
<li>1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Debuginfo-Updates for sle-12-x86_64</li>
<li>1878 SUSE Linux Enterprise Server 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64</li>
<li>1880 SUSE Linux Enterprise Server for SAP Applications 12 SP5 x86_64 SLES12-SP5-Updates for sle-12-x86_64</li>
</ul>
<p>I guess this problem is relatively specific to these repos.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=678602021-02-19T10:23:51Zttereshcttereshc@redhat.com
<ul></ul><p>Thank you very much for the detailed report and debugging. I'm glad you were able to resolve it.</p>
<p>I'm noting 2 things for Pulp to fix:</p>
<ul>
<li>it looks like a bug to me that during a Pulp 2to3 migration, one can't publish those files, they are just custom files in repodata and Pulp should handle those. It's a bug either in a migration plugin or in RPM plugin (in case the same error appears during sync/publish workflow in Pulp3).</li>
<li>the error message needs to be expanded to have more details to locate the problematic file.</li>
</ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=678612021-02-19T10:24:39Zttereshcttereshc@redhat.com
<ul><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=706582021-05-11T19:56:12Zpabloalcantara
<ul></ul><p>Hi,
It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem?
Thanks a lot
<a href="mailto:laugmanuel@gmail.com" class="email">laugmanuel@gmail.com</a> wrote:</p>
<blockquote>
<p>I found a cause of this problem - there were three (somwhat) broken RPM repos on Pulp2 side.
The problematic repos had metadata files which seem to have caused this problem:</p>
<p>However, finding that information was not trivial at all. I had to add some debug output to <code>pulp_rpm/app/tasks/publishing.py:483</code> and log the <code>name</code> and <code>publication.repository.name</code> variables to find the problematic repos and metadata file/type.</p>
<pre><code class="sh syntaxhl" data-language="sh"> root@pulp:/var/lib/pulp# pulp-admin rpm repo content metafile <span class="nt">--repo-id</span> 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: 746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
f400
Checksum Type: sha256
Data Type: appdata-icons
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: 746807760418ca81c23d90a2aa91b0fc98139cc83b38ce5ab9119b148ae6
f400
Checksum Type: sha256
Data Type: app-icons
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
Checksum: c33ef84117bf852f7286a63bbe84d238868c839bb979c83dc7fd80af33ef
d318
Checksum Type: sha256
Data Type: appdata
Downloaded: True
Pulp User Metadata:
Repo Id: 9995b83f-0162-45de-9725-dae79b1eb8b0
</code></pre>
<p>After deleting the problematic metadata file from Pulp2, the migration continued successfully:</p>
<pre><code class="sh syntaxhl" data-language="sh"> root@pulp:/var/lib/pulp# pulp-admin rpm repo remove metafile <span class="nt">--repo-id</span> 9995b83f-0162-45de-9725-dae79b1eb8b0 <span class="nt">--in</span> <span class="s2">"data_type=app-icons"</span>
</code></pre>
</blockquote> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=706652021-05-12T06:44:38Zlaugmanuel@gmail.com
<ul></ul><p>Hi,</p>
<p>I've basically replaced <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/publishing.py#L552" class="external">https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/publishing.py#L552</a> with the following:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="k">try</span><span class="p">:</span>
<span class="n">record</span><span class="p">.</span><span class="n">fill</span><span class="p">(</span><span class="n">checksum_type</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'/tmp/pulp3_publish_task.log'</span><span class="p">,</span> <span class="s">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">the_file</span><span class="p">:</span>
<span class="n">the_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">"File: %s, Path: %s, DB to update: %s, Repodata path: %s, Record: %s, Location href: %s, OS Join Path: %s, Repomd Path: %s, Repo: %s</span><span class="se">\n</span><span class="s">"</span> <span class="o">%</span>
<span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">db_to_update</span><span class="p">,</span> <span class="n">repodata_path</span><span class="p">,</span> <span class="n">record</span><span class="p">,</span> <span class="n">record</span><span class="p">.</span><span class="n">location_href</span><span class="p">,</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">repodata_path</span><span class="p">),</span> <span class="n">repomd_path</span><span class="p">,</span> <span class="n">publication</span><span class="p">.</span><span class="n">repository</span><span class="p">.</span><span class="n">name</span><span class="p">))</span>
</code></pre>
<p>The output was something like:</p>
<pre><code>File: app-icons, Path: icons.tar.gz, DB to update: None, Repodata path: repodata, Record: <createrepo_c.RepomdRecord app-icons object>, Location href: repodata/icons.tar.gz, OS Join Path: repodata, Repomd Path: /var/lib/pulp/tmp/120233@pulp.example.com/fa077a59-d492-4df6-9f71-7778d7957200/repomd.xml, Repo: 9995b83f-0162-45de-9725-dae79b1eb8b0
</code></pre>
<p>The given repo id was the interesting part to find the problematic Pulp2 repo (as seen above).
I hope this helps.</p>
<p>Manuel</p>
<p>pabloalcantara wrote:</p>
<blockquote>
<p>Hi,
It´s possible to you show me what change you did on pulp_rpm/app/tasks/publishing.py to show the repo problem?
Thanks a lot</p>
</blockquote> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=715332021-06-01T20:26:04Zttereshcttereshc@redhat.com
<ul><li><strong>Tags</strong> <i>Katello</i> added</li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=715342021-06-01T20:27:42Zdalleydalley@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Sprint</strong> set to <i>Sprint 97</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=715352021-06-01T20:30:26Zttereshcttereshc@redhat.com
<ul></ul><p>Priority went up because there are more reports for this issue, outside this tracker as well.
See repos to test with in the comment#2.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=715362021-06-01T20:37:28Zjsherril@redhat.comjsherril@redhat.com
<ul></ul><p>relevant katello posts:</p>
<p><a href="https://community.theforeman.org/t/pulp-2-to-pulp-3-migration-failed/23702/10" class="external">https://community.theforeman.org/t/pulp-2-to-pulp-3-migration-failed/23702/10</a></p>
<p><a href="https://community.theforeman.org/t/pulp3-migration-failed/22473/33" class="external">https://community.theforeman.org/t/pulp3-migration-failed/22473/33</a></p>
<p><a href="https://community.theforeman.org/t/katello-3-18-2-pulp-2to3-migration-error/23659" class="external">https://community.theforeman.org/t/katello-3-18-2-pulp-2to3-migration-error/23659</a></p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=715812021-06-02T16:02:22Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 97</i> to <i>Sprint 98</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=718642021-06-11T17:19:44Zttereshcttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>ttereshc</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=718682021-06-11T19:18:37Zttereshcttereshc@redhat.com
<ul></ul><p>Reproduced it locally by adding a tar.gz file to a pulp 2 repo and running a migration.<br>
Smells like a pulp_rpm problem, potentially a createrepo_c issue.<br>
I do not think it is migration related, I expect the same issue to see with sync/publish in Pulp 3.</p>
<p>If someone with access to SLES repos can sync into Pulp 3 one of the repos from comment 2, it will be highly appreciated.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=721422021-06-18T21:14:13Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 98</i> to <i>Sprint 99</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=727582021-07-02T19:23:54Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 99</i> to <i>Sprint 100</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=732382021-07-14T21:27:53Zdalleydalley@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/281">ttereshc</a>, ggainey has access to SUSE repos</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=732392021-07-14T21:28:27Zdalleydalley@redhat.com
<ul><li><strong>Assignee</strong> changed from <i>ttereshc</i> to <i>ggainey</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=732702021-07-15T20:17:57Zggainey
<ul></ul><p>I haven't recreated the icons.tar.gz problem yet, but I had had issues post-2to3 w/ one of the repos from #c2.</p>
<p>Sync'd into Pulp2, migrated, resync'd, FAIL:</p>
<pre><code>CONFIG="{\"query_auth_token\": \"_INSERT_REPO_TOKEN_HERE\" }"
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id sles12-update-sp5
--feed https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/ \
--download-policy on_demand
http --verify no --auth admin:admin \
PUT https://localhost/pulp/api/v2/repositories/sles12-update-sp5 importer_config:="${CONFIG}"
pulp-admin rpm repo sync run --repo-id sles12-update-sp5
PLAN_HREF=$(pulp migration plan create --plan '{"plugins": [{"type": "rpm"}]}' | jq -r .pulp_href)
echo "Plan: $PLAN_HREF"
pulp -T 0 migration plan run --href ${PLAN_HREF}
REMOTE_HREF=$(pulp rpm repository show --name sles12-update-sp5 | jq -r .remote)
echo "Remote-href: ${REMOTE_HREF}"
pulp rpm repository sync --name sles12-update-sp5
...
sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token :
Remote-href: /pulp/api/v3/remotes/rpm/rpm/339d06e9-59dd-447f-b1f2-7bf065b9bfe9/
Started background task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/
......................................................................................................Error: Task /pulp/api/v3/tasks/b0ce0fdb-a0b2-44cf-83d2-61ec46644897/ failed: ''<' not supported between instances of 'datetime.datetime' and 'NoneType''
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm remote list
</code></pre>
<p>Use same remote to sync directly to an empty Pulp3 repo, SUCCEED:</p>
<pre><code>pulp rpm repository create --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 rel_path]$ pulp rpm repository sync --name sles12-sp5-update-two --remote 60f08f2891199215a200d708-sles12-update-sp5
Started background task /pulp/api/v3/tasks/18a5773e-f979-4aa3-91a5-bf45f0aba044/
.......................................................................................................Done.
</code></pre>
<p>Other SUSE repos migrate-and-sync successfully. I'll continue the experiments with the remaining repos from #c2 and report back here with results.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=732882021-07-15T21:08:14Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 100</i> to <i>Sprint 101</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733442021-07-16T07:09:39Zhellokatello123
<ul></ul><p>Hi,</p>
<p>I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:</p>
<pre><code>2021-07-15 21:49:07 +0000: Distribution creation 5704/5711Migration failed, You will want to investigate: https://katello.a.space.corp/foreman_tasks/tasks/8228cbae-de57-4866-bbfb-cc28fce8b7bb
rake aborted!
ForemanTasks::TaskError: Task 8228cbae-de57-4866-bbfb-cc28fce8b7bb: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
</code></pre>
<p>The reason why I started to migrate was that I was having problems with a SLES 15 content view. I was using an rpm include filter and the option to resolve deps on the CV to limit the number of packages I have in this CV. Basically I have 10-15 dummy rpm packges that only have deps. All of these deps should be resolved and only these packages should be available in the CV. As I kept having varying results over and over again I had the feeling that this was a bug and it might be releated especially to the SLES 15 repos. I use the SCC plugin in Katello to sync the repos. I wrote to foreman commnunity:</p>
<p><a href="https://community.theforeman.org/t/enable-solve-dependencies-for-content-views/20721/13" class="external">https://community.theforeman.org/t/enable-solve-dependencies-for-content-views/20721/13</a></p>
<p>Where they suggest to me to upgrade to pulp3. Then I stumbled over this bug report and I have the feeling that my initial problem is related to this one. I'm no expert in pulp, but If someone could give me a hint how to debug this further, I',m more then willing to do so.</p>
<p>Best Regards,
Oliver</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733472021-07-16T14:05:12Zggainey
<ul></ul><p>The problem in #c18 looks more like a pulp_rpm issue, opened <a href="https://pulp.plan.io/issues/9096" class="external">https://pulp.plan.io/issues/9096</a> for it.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733482021-07-16T14:12:07Zggainey
<ul></ul><p>hellokatello123 wrote:</p>
<blockquote>
<p>Hi,</p>
<p>I stumbled over this bug report while searching for clues on why my pulpl2 -> pulp3 migration in katello fails. I got further and further but now I'm stuck in the prep phase:</p>
</blockquote>
<p>Hey Oliver - what does "journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00' " tell you?</p>
<p>There are def some SUSE repos that break some of Pulp3's assumptions about "sane repositories"; I'm working on trying to address them e'en as I type, more info would be great!</p>
<p>G</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733592021-07-16T15:38:35Zhellokatello123
<ul></ul><p>Hi,</p>
<p>here is the output:</p>
<pre><code>[root@katello ~]# journalctl -u pulpcore* --since '2021-07-15 21:48:07' --until '2021-07-15 21:50:00'
-- Logs begin at Thu 2021-07-15 15:28:40 UTC, end at Fri 2021-07-16 15:38:00 UTC. --
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:19 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:19 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:35 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:35 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:48:51 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:48:51 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (33440ee4-f5b0-4046-9635-5259a884fc36)
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: Cleaning registries for queue: 1127@katello.a.space.corp
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: 4270f7a1-8a22-4f03-9c5f-0061179aa8c5
Jul 15 21:48:59 katello.a.space.corp pulpcore-worker-1[1127]: pulp: rq.worker:INFO: 1127@katello.a.space.corp: Job OK (4270f7a1-8a22-4f03-9c5f-0061179aa8c5)
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/tasks/01b838e6-5a6d-4ad0-8bea-2588070a4b7a/ HTTP/1.1" 200 24453 "-" "OpenAPI-Generator/3.7.1/ruby
Jul 15 21:49:07 katello.a.space.corp pulpcore-api[1110]: - - [15/Jul/2021:21:49:07 +0000] "GET /pulp/api/v3/task-groups/829a5ade-e80d-4c60-a518-8f4cfcd0a0ee/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/3.7.1/
Jul 15 21:49:38 katello.a.space.corp pulpcore-worker-8[1141]: pulp: rq.worker:INFO: Cleaning registries for queue: 1141@katello.a.space.corp
Jul 15 21:49:40 katello.a.space.corp pulpcore-worker-3[1136]: pulp: rq.worker:INFO: Cleaning registries for queue: 1136@katello.a.space.corp
</code></pre> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733622021-07-16T17:26:15Zggainey
<ul></ul><p>Syncing into a clean Pulp3 database (ie, not post-migration) recreates the icons.tar.gz problem:</p>
<pre><code>sles12-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/edbbba67-d0d1-44e1-95df-d4196577cd4e/"
"/pulp/api/v3/repositories/rpm/rpm/df2e38e2-e512-4cea-baca-620508df5871/"
Started background task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/
.....................................................................Error: Task /pulp/api/v3/tasks/82b6e9d3-b666-4277-aa03-5e06638e11ab/ failed: 'File icons.tar.gz doesn't exists or not a regular file'
sles12-debug-update-sp5 : https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug/
Token : <REPO-TOKEN>
"/pulp/api/v3/remotes/rpm/rpm/34e45336-b432-44a5-ac6d-86183817b321/"
"/pulp/api/v3/repositories/rpm/rpm/941b278d-31a6-4180-8fea-9f8ab0a091be/"
Started background task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/
...................................Error: Task /pulp/api/v3/tasks/7c955976-0fc0-4475-8905-cb736f00d6d8/ failed: 'File icons.tar.gz doesn't exists or not a regular file'
</code></pre>
<p>I added some debug code to capture the actual stacktrace - createrepo_c gets Upset when asked to compute the checksum of a file that doesn't exist:</p>
<pre><code>Jul 16 16:52:11 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2977]: cr_repomd_record_fill: File icons.tar.gz doesn't exists
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO: Task c9e06f22-4b96-4614-84c5-7874429f5d3f failed (File icons.tar.gz doesn't exists or not a regular file)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: pulp [db17d82e2a8a4828a92be2f355d6bf66]: pulpcore.tasking.pulpcore_worker:INFO: File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 268, in _perform_task
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: result = func(*args, **kwargs)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 422, in synchronize
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: version = dv.create()
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: loop.run_until_complete(pipeline)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 987, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: repository.on_new_version(self)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py", line 288, in on_new_version
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: sqlite_metadata=self.sqlite_metadata,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 366, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: return publication
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulpcore/pulpcore/app/models/publication.py", line 183, in __exit__
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: raise exc_val.with_traceback(exc_tb)
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 345, in publish
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: metadata_signing_service=metadata_signing_service,
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 584, in generate_repo_metadata
Jul 16 16:52:12 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com pulpcore-worker[2089]: record.fill(checksum_type)
</code></pre>
<p>The problem is that extra-repomd files don't know they live in the worker's CWD, and can't be found. Investigating for a fix.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733742021-07-16T20:30:36Zggainey
<ul><li><strong>Project</strong> changed from <i>Migration Plugin</i> to <i>RPM Support</i></li></ul><p>SUSE repos are Special, and pulp_rpm has some questionable heuristics in code. PR shortly.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733772021-07-16T21:30:22Zpulpbot
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p>PR: <a href="https://github.com/pulp/pulp_rpm/pull/2043" class="external">https://github.com/pulp/pulp_rpm/pull/2043</a></p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733782021-07-16T22:16:26Zdalleydalley@redhat.com
<ul><li><strong>Sprint/Milestone</strong> set to <i>3.14.0</i></li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=733822021-07-17T01:25:32Zdalleydalley@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-6 priority-default closed" href="/issues/9096">Issue #9096</a>: Sync of SUSE, post-2to3-migration, exposes a bug in resolve_advisory_conflict()</i> added</li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=734532021-07-19T18:16:51Zggainey
<ul><li><strong>Related to</strong> <i><a class="issue tracker-7 status-11 priority-6 priority-default closed" href="/issues/9113">Backport #9113</a>: Backport 8275 to pulp_rpm 3.11</i> added</li></ul> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=734552021-07-19T18:17:19Zggainey
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Adapted to some (more) SUSE repo edge-cases. * Some metadata-files can have '-' in the filename ..." href="https://pulp.plan.io/projects/pulp_rpm/repository/9/revisions/47584d3c562e17ab8d2cfc873cae19b4a8855cb1">47584d3c562e17ab8d2cfc873cae19b4a8855cb1</a>.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=734572021-07-19T19:10:38Zggainey
<ul></ul><p>The repo-urls that failed on icons.tar.gz, that I have access to for SUSE, included:</p>
<pre><code>https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/
https://updates.suse.com/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update_debug
</code></pre>
<p>(My dev-acct doesn't appear to have access to SAP repos so I couldn't test those)</p>
<p>The failure, both when migrating and with a sync-into-Pulp3, was the one noted in the description of "File icons.tar.gz doesn't exists or not a regular file"</p>
<p>Sync-and-migrate test is here: <a href="https://github.com/ggainey/pulp_startup/blob/main/rel_path/sync_and_migrate.sh" class="external">https://github.com/ggainey/pulp_startup/blob/main/rel_path/sync_and_migrate.sh</a></p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=734632021-07-20T11:45:57Zhellokatello123
<ul></ul><p>Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?</p>
<p>Best Regards</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=735102021-07-21T11:47:18Zggainey
<ul></ul><p>hellokatello123 wrote:</p>
<blockquote>
<p>Just one question. Would deleting all SLES repos, upgrading to pulp3 and then re-adding the repos solve the issue or is the issue also in pulp3?</p>
<p>Best Regards</p>
</blockquote>
<p>It's in Pulp3. The error lies in how Pulp3/rpm treats extra-repomd-files, that have dashes in their filename, in sync code that is shared between migration and 'regular' sync. I can recreate the problem just syncing SUSE into a "clean" pulp3 instance.</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=735922021-07-22T08:24:30Zhellokatello123
<ul></ul><p>Hi,</p>
<p>thanks a lot for the clarification. Then I will have to wait for a fix. :)</p>
<p>Cheers,
Oliver</p> RPM Support - Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file"https://pulp.plan.io/issues/8275?journal_id=737372021-07-24T04:12:01Zpulpbot
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>