Issue #671: On sync and delete, relative urls conflict with relative urls of deleted repos - Pulp

Actions

Send by e-mail Copy link

Issue #671

closed

On sync and delete, relative urls conflict with relative urls of deleted repos

Added by igulina@redhat.com over 8 years ago. Updated about 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

amacdona@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

1. Low

Version:

Platform Release:

2.6.1

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

pulp 2.6 beta 5 on el6

Description of problem:

1. Create a repo with URL or local feed which doesn't contain /repodata/repomd.xm
2. Sync that repo - get an error
3. Try to delete this repo - get Task Fail and PulpExecutionException, however repo will be deleted, for example try to delete the same repo again - it says the repo doesn't exist.

Version-Release number of selected component (if applicable):

rpm -qa pulp-server

pulp-server-2.6.0-0.5.beta.el6.noarch

How reproducible:
always

pulp-admin rpm repo create --repo-id epel6_2 --feed file:////var/lib/pulp/published/yum/master/yum_distributor/epel6_1/

Successfully created repository [epel6_2]

pulp-admin rpm repo list

~~--------------------------------------------------------------------~~
RPM Repositories
~~--------------------------------------------------------------------~~
Id: epel6_1
Display Name: epel6_1
Description: None
Content Unit Counts:
Erratum: 3635
Package Category: 3
Package Group: 208
Rpm: 11178
Yum Repo Metadata File: 1

Id: epel6_2
Display Name: epel6_2
Description: None
Content Unit Counts:

pulp-admin rpm repo sync run --repo-id epel6_2

~~--------------------------------------------------------------------~~
Synchronizing Repository [epel6_2]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'//var/lib/pulp/published/yum/master/yum_distributor/epel6_1/repodata/repomd.xm
l'

Task Failed

Importer indicated a failed response

pulp-admin rpm repo list

Id: epel6_2
Display Name: epel6_2
Description: None
Content Unit Counts:

pulp-admin rpm repo delete --repo-id epel6_2

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Failed

Pulp exception occurred: PulpExecutionException

pulp-admin rpm repo delete --repo-id epel6_2

Repository [epel6_2] does not exist on the server

pulp-admin rpm repo list

~~--------------------------------------------------------------------~~
RPM Repositories
~~--------------------------------------------------------------------~~

Id: epel6_1
Display Name: epel6_1
Description: None
Content Unit Counts:
Erratum: 3635
Package Category: 3
Package Group: 208
Rpm: 11178
Yum Repo Metadata File: 1

less ~/.pulp/server_calls.log

2015-01-19 14:34:09,129 - INFO - Response body :
{
"exception": null,
"task_type": "pulp.server.tasks.repository.delete",
"_href": "/pulp/api/v2/tasks/adc97668-3cb6-4e1d-b9eb-dacea5b9b8f5/",
"task_id": "adc97668-3cb6-4e1d-b9eb-dacea5b9b8f5",
"tags": [
"pulp:repository:epel_6_2",
"pulp:action:delete"
],
"finish_time": "2015-01-19T19:34:08Z",
"_ns": "task_status",
"start_time": "2015-01-19T19:34:08Z",
"traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py\", line 328, in call\n return super(Task, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.6/site-packages/celery/app/trace.py\", line 437, in protected_call\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py\", line 43, in delete\n manager.delete_repo(repo_id)\n File \"/usr/lib/python2.6/site-packages/pulp/server/managers/repo/cud.py\", line 286, in delete_repo\n raise pe\nPulpExecutionException: Pulp exception occurred: PulpExecutionException\n",
"spawned_tasks": [],
"progress_report": {},
"queue": "reserved_resource_worker-0@host",
"state": "error",
"worker_name": "reserved_resource_worker-0@host",
"result": null,
"error": {
"code": "PLP0000",
"data": {},
"description": "Pulp exception occurred: PulpExecutionException",
"sub_errors": [
{
"code": "PLP0000",
"data": {},
"description": "[Errno 21] Is a directory: '/var/lib/pulp/published/yum/https/repos/var/lib/pulp/published/yum/master/yum_distributor'",
"sub_errors": []
}
]
},
"_id": {
"$oid": "54bd5c30b5a0715b62d4766a"
},
"id": "54bd5c30542c8e095ea45f20"
}

+ This bug was cloned from Bugzilla Bug #1183784 +

Related issues

Actions

Copy link

Updated by pthomas@redhat.com over 8 years ago

I checked this on rhel7 & rhel6

On Rhel7 it works fine

[root@cloud-qe-4 ~]# pulp-admin rpm repo create --repo-id zoo_1 --feed file:///var/lib/pulp/published/yum/master/yum_distributor/zoo/
Successfully created repository [zoo_1]

[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]# pulp-admin rpm repo sync run --repo-id zoo_1+----------------------------------------------------------------------+
Synchronizing Repository [zoo_1]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'///var/lib/pulp/published/yum/master/yum_distributor/zoo/repodata/repomd.xml'

Task Failed

Importer indicated a failed response

[root@cloud-qe-4 ~]# pulp-admin rpm repo delete --repo-id zoo_1
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [zoo_1] successfully deleted

[root@cloud-qe-4 ~]#

And on Rhel6 I was able to reproduce the error.

[root@gizmo ~]# pulp-admin rpm repo sync run --repo-id epel5_2

~~--------------------------------------------------------------------~~
Synchronizing Repository [epel5_2]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'//var/lib/pulp/published/yum/master/yum_distributor/epel5/repodata/repomd.xml'

Task Failed

Importer indicated a failed response

[root@gizmo ~]# pulp-admin rpm repo list
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

~~--------------------------------------------------------------------~~
RPM Repositories
~~--------------------------------------------------------------------~~

Id: upload
Display Name: upload
Description: None
Content Unit Counts:
Rpm: 9

Id: rhel7
Display Name: rhel7
Description: None
Content Unit Counts:
Erratum: 176
Package Category: 9
Package Environment: 6
Package Group: 70
Rpm: 5270
Yum Repo Metadata File: 1

Id: zoo
Display Name: zoo
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Rpm: 32

Id: epel5
Display Name: epel5
Description: None
Content Unit Counts:
Erratum: 1948
Package Category: 5
Package Group: 36
Rpm: 6674
Yum Repo Metadata File: 1

Id: epel5_2
Display Name: epel5_2
Description: None
Content Unit Counts:

[root@gizmo ~]# pulp-admin rpm repo delete --repo-id epel5_2
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Failed

Pulp exception occurred: PulpExecutionException

[root@gizmo ~]#

+ This comment was cloned from Bugzilla #1183784 comment 1 +

Actions

Copy link

Updated by amacdona@redhat.com over 8 years ago

I was unable to replicate this on RHEL6 using the zoo repo.

Looking over the comments, I notice that all the failures occurred while syncing the epel5 repo.

+ This comment was cloned from Bugzilla #1183784 comment 2 +

Actions

Copy link

Updated by mhrivnak over 8 years ago

I think the failures only occurred with feeds that are not valid yum repositories at all. If I'm reading this correctly, I'd expect you could reproduce this even with --feed=http://slashdot.org

+ This comment was cloned from Bugzilla #1183784 comment 3 +

Actions

Copy link

Updated by igulina@redhat.com over 8 years ago

Austin, in my case it was syncing of epel6 (see my bug description), for Preethi it was epel5 (her comment #1), and I also wasn't able to reproduce it for local zoo repo on rh6, but with some other bad feed on https there is the same Exception on delete repo:

rpm -qa pulp-server

pulp-server-2.6.0-0.5.beta.el6.noarch

pulp-admin rpm repo create --repo-id zoo_2 --feed file:///var/lib/pulp/published/yum/master/yum_distributor/zoo/

Successfully created repository [zoo_2]

pulp-admin rpm repo sync run --repo-id zoo_2

~~--------------------------------------------------------------------~~
Synchronizing Repository [zoo_2]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'///var/lib/pulp/published/yum/master/yum_distributor/zoo/repodata/repomd.xml'

Task Failed

Importer indicated a failed response

pulp-admin rpm repo delete --repo-id zoo_2

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [zoo_2] successfully deleted

pulp-admin rpm repo create --repo-id no_repodata_repo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/

Successfully created repository [no_repodata_repo]

pulp-admin rpm repo sync run --repo-id no_repodata_repo

~~--------------------------------------------------------------------~~
Synchronizing Repository [no_repodata_repo]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

Not Found

Task Failed

Importer indicated a failed response

pulp-admin rpm repo delete --repo-id no_repodata_repo

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Failed

Pulp exception occurred: PulpExecutionException

pulp-admin rpm repo delete --repo-id no_repodata_repo

Repository [no_repodata_repo] does not exist on the server

+ This comment was cloned from Bugzilla #1183784 comment 4 +

Actions

Copy link

Updated by amacdona@redhat.com over 8 years ago

I was able to reproduce in fedora and RHEL.

This bug is a result of not cleaning up /var/lib/pulp/published/yum/ directory when a repo is deleted. This allows the relative path of a new repo to conflict with the relative url of deleted repos.

Steps to replicate:
1. Create, sync, and delete a repo with a relative feed without specifying a relative url.
2. Create a repo, again without a relative url, that has a feed that is a sub-url of the original repo.
3. $ pulp-admin rpm repo delete --repo-id=fail

Expected Result:
Repository [fail] successfully deleted
Actual Result:
Task Failed
Pulp exception occurred: PulpExecutionException

Since this one is weird, here's a script!
$ cat 1183784.sh
pulp-admin rpm repo create --repo-id=zoo --feed=http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo
pulp-admin rpm repo sync run --repo-id=zoo
pulp-admin rpm repo delete --repo-id=zoo
pulp-admin rpm repo create --repo-id=fail --feed=http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/
pulp-admin rpm repo delete --repo-id=fail

Two things need to be fixed with this one:
1. When a repo is deleted, the directory structure in /var/lib/pulp/published/yum/ should be removed.
2. To makeWhen a repo is created, the relative url validation should validate against the directory structure.

+ This comment was cloned from Bugzilla #1183784 comment 5 +

Actions

Copy link

Updated by amacdona@redhat.com over 8 years ago

***** Bug 1188406 has been marked as a duplicate of this bug. ***

+ This comment was cloned from Bugzilla #1183784 comment 6 +

Actions

Copy link

Updated by amacdona@redhat.com over 8 years ago

Status changed from ASSIGNED to MODIFIED
Platform Release set to 2.6.1

https://github.com/pulp/pulp_rpm/pull/648

Actions

Copy link

Updated by amacdona@redhat.com about 8 years ago

Related to Issue #522: Changing relative URL doesn't remove old URL. added

Actions

Copy link

#10

Updated by bmbouter about 8 years ago

Severity changed from Low to 1. Low

Actions

Copy link

#11

Updated by bcourt about 8 years ago

Status changed from MODIFIED to 5

Actions

Copy link

#12

Updated by igulina@redhat.com about 8 years ago

Status changed from 5 to 6

rpm -qa pulp-server

pulp-server-2.6.1-0.2.beta.el6.noarch

pulp-admin rpm repo sync run --repo-id epel_6_2

~~--------------------------------------------------------------------~~
Synchronizing Repository [epel_6_2]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'///var/lib/pulp/published/yum/master/yum_distributor/epel_6_1/repodata/repomd.
xml'

Task Failed

Importer indicated a failed response

pulp-admin rpm repo delete --repo-id epel_6_2

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [epel_6_2] successfully deleted

pulp-admin rpm repo create --repo-id test_gena --feed file:///var/lib/pulp/published/yum/master/yum_distributor/gena/

Successfully created repository [test_gena]

pulp-admin rpm repo sync run --repo-id test_gena

~~--------------------------------------------------------------------~~
Synchronizing Repository [test_gena]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... failed

[Errno 2] No such file or directory:
u'///var/lib/pulp/published/yum/master/yum_distributor/gena/repodata/repomd.xml'

Task Failed

Importer indicated a failed response

pulp-admin rpm repo delete --repo-id test_gena

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [test_gena] successfully deleted

pulp-admin rpm repo list

~~--------------------------------------------------------------------~~
RPM Repositories
~~--------------------------------------------------------------------~~

Id: epel6_1
Display Name: epel6_1
Description: None
Content Unit Counts:
Erratum: 3635
Package Category: 3
Package Group: 208
Rpm: 11178
Yum Repo Metadata File: 1

Id: gena
Display Name: gena
Description: None
Content Unit Counts:

Id: epel_6_1
Display Name: epel_6_1
Description: None
Content Unit Counts:
Erratum: 3782
Package Category: 3
Package Group: 208
Rpm: 11365
Yum Repo Metadata File: 1

pulp-admin rpm repo create --repo-id no_repodata_repo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/

Successfully created repository [no_repodata_repo]

pulp-admin rpm repo sync run --repo-id no_repodata_repo

~~--------------------------------------------------------------------~~
Synchronizing Repository [no_repodata_repo]
~~--------------------------------------------------------------------~~

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

Not Found

Task Failed

Importer indicated a failed response

pulp-admin rpm repo delete --repo-id no_repodata_repo

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [no_repodata_repo] successfully deleted

pulp-admin rpm repo list

~~--------------------------------------------------------------------~~
RPM Repositories
~~--------------------------------------------------------------------~~

Id: epel6_1
Display Name: epel6_1
Description: None
Content Unit Counts:
Erratum: 3635
Package Category: 3
Package Group: 208
Rpm: 11178
Yum Repo Metadata File: 1

Id: gena
Display Name: gena
Description: None
Content Unit Counts:

Id: epel_6_1
Display Name: epel_6_1
Description: None
Content Unit Counts:
Erratum: 3782
Package Category: 3
Package Group: 208
Rpm: 11365
Yum Repo Metadata File: 1