Project

Profile

Help

Issue #1489

closed

Unit copy not copying package groups or categories

Added by paji@redhat.com about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I have not been able to reproduce all the steps via pulp-admin but seems to readily happen in the API when called in runcible.

1) Checkout -> https://github.com/Katello/runcible, we need the contents of https://github.com/Katello/runcible/tree/master/test/fixtures/repositories/zoo5 and below in some directory. Steps below will assume file:///var/www/repositories/zoo5 is the destination

2) Create a repo
RestClient.post "https://<PULP>/pulp/api/v2/repositories/", "{\"id\":\"integration_test_id\",\"importer_type_id\":\"yum_importer\",\"importer_config\":{\"feed\":\"file:///var/www/repositories/zoo5\"}}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"125", "accept"=>"application/json", "content_type"=>"application/json"

3) Sync as follows
RestClient.post "https://katello-pulp-nightly.example.com/pulp/api/v2/repositories/integration_test_id/actions/sync/", "{}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"2", "accept"=>"application/json", "content_type"=>"application/json"
\n# => 202 Accepted | application/json 172 bytes

4) Next create a new repo
RestClient.post "https://katello-pulp-nightly.example.com/pulp/api/v2/repositories/", "{\"id\":\"integration_test_id_clone\",\"importer_type_id\":\"yum_importer\",\"importer_config\":{}}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"89", "accept"=>"application/json", "content_type"=>"application/json"

5) Copy contents of integration_test_id_to integration_test_id_clone

 DEBUG -- : RestClient.post "https://katello-pulp-nightly.example.com/pulp/api/v2/repositories/integration_test_id_clone/actions/associate/", "{\"source_repo_id\":\"integration_test_id\"}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"40", "accept"=>"application/json", "content_type"=>"application/json"

Notice that package groups and package categories do not get copied. Infact you can try this part with pulp-admin also and the see the same behaviour

pulp-admin rpm repo create --repo-id=recontextualize
pulp-admin rpm repo copy all --from-repo-id=integration_test_id --to-repo-id=recontextualize

The package groups are not copied over.
Actions #1

Updated by bmbouter about 8 years ago

For the steps that are not able to be done using pulp-admin, can those be rewritten into httpie[0] commands. You can `pip install httpie` or yum/dnf httpie to get it onto your system. For example, for localhost with admin:admin credentials and having the client not verify the SSL hostname, I think the first one is:

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ id=integration_test_id importer_type_id=yum_importer importer_config:='{"feed": "file:///var/www/repositories/zoo5"}'

FWIW, I also thing this would be accomplished with pulp-admin using:

pulp-admin rpm repo create --repo-id integration_test_id --feed file:///var/www/repositories/zoo5

[0]: https://github.com/jkbrzt/httpie

Actions #2

Updated by ipanova@redhat.com about 8 years ago

Was not able to reproduce this, both pulp-admin and API work:

$ pulp-admin -u admin -p admin repo list --repo-id zoo
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                   zoo
Display Name:         None
Description:          None
Content Unit Counts:  
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

[ipanova@ina pulp]$ pulp-admin -u admin -p admin rpm repo copy all  --from-repo-id zoo --to-repo-id myrepo
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
 erratum:
  RHEA-2012:0001
  RHEA-2012:0002
  RHEA-2012:0003
  RHEA-2012:0004
 package_category:
  all
 package_group:
  birds
  mammals
 rpm:
  bear-4.1-1-noarch
  camel-0.1-1-noarch
  cat-1.0-1-noarch
  cheetah-1.25.3-5-noarch
  chimpanzee-0.21-1-noarch
  cockateel-3.1-1-noarch
  cow-2.2-3-noarch
  crow-0.8-1-noarch
  dog-4.23-1-noarch
  dolphin-3.10.232-1-noarch
  duck-0.6-1-noarch
  elephant-8.3-1-noarch
  fox-1.1-2-noarch
  frog-0.1-1-noarch
  giraffe-0.67-2-noarch
  gorilla-0.62-1-noarch
  horse-0.22-2-noarch
  kangaroo-0.2-1-noarch
  lion-0.4-1-noarch
  mouse-0.1.12-1-noarch
  penguin-0.9.1-1-noarch
  pike-2.2-1-noarch
  shark-0.1-1-noarch
  squirrel-0.1-1-noarch
  stork-0.12-2-noarch
  tiger-1.0-4-noarch
  trout-0.12-1-noarch
  walrus-0.71-1-noarch
  walrus-5.21-1-noarch
  whale-0.2-1-noarch
  wolf-9.4-2-noarch
  zebra-0.1-2-noarch

[ipanova@ina pulp]$ pulp-admin -u admin -p admin repo list --repo-id myrepo
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                   myrepo
Display Name:         None
Description:          None
Content Unit Counts:  
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

$ curl -H "Accept: application/json" -X POST -k -u admin:admin -d '{"source_repo_id": "zoo"}' 'https://localhost/pulp/api/v2/repositories/test/actions/associate/'|python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   197  100   172  100    25    618     89 --:--:-- --:--:-- --:--:--   620
{
    "error": null,
    "result": null,
    "spawned_tasks": [
        {
            "_href": "/pulp/api/v2/tasks/dde39cdc-910c-406f-b6f4-0be6eb1f983e/",
            "task_id": "dde39cdc-910c-406f-b6f4-0be6eb1f983e"
        }
    ]
}
[ipanova@ina pulp]$ pulp-admin -u admin -p admin repo list --repo-id test
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                   test
Display Name:         None
Description:          None
Content Unit Counts:  
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32
Actions #3

Updated by mhrivnak about 8 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes
Actions #4

Updated by semyers about 8 years ago

  • Related to Issue #1478: POST request to /pulp/api/v2/repositories/<reponame>/search/units/ does not transform id correctly when unit results returned added
Actions #5

Updated by semyers about 8 years ago

Possibly a duplicate of #1478, but if not it's definitely related.

Edit: I put this on the wrong issue, this comment was meant for #1490

Actions #6

Updated by semyers about 8 years ago

  • Related to deleted (Issue #1478: POST request to /pulp/api/v2/repositories/<reponame>/search/units/ does not transform id correctly when unit results returned)
Actions #7

Updated by paji@redhat.com about 8 years ago

Just saw https://pulp.plan.io/issues/1489#note-2
So I was unable to reproduce this when using pulp-admin. I think the runcible api doesnt set things like notes fields with __repo_type:yum etc. It could be that. I will try to get you the exact params we send over to pulp when we create repo in a few

Actions #8

Updated by paji@redhat.com about 8 years ago

Actually I think comment #1 completely captures the post request we send to pulp and Dr Bouters' variation should be an accurate repro

Client.post "https://<PULP>/pulp/api/v2/repositories/", "{\"id\":\"integration_test_id\",\"importer_type_id\":\"yum_importer\",\"importer_config\":{\"feed\":\"file:///var/www/repositories/zoo5\"}}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"125", "accept"=>"application/json", "content_type"=>"application/json" 

Dr Bouters' variation

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ id=integration_test_id importer_type_id=yum_importer importer_config:='{"feed": "file:///var/www/repositories/zoo5"}'
Actions #9

Updated by mhrivnak about 8 years ago

I don't think anything special in the notes field will matter. That is just a hint for pulp-admin to quickly know what type a repo is.

I'm also not able to reproduce. I'm using the repo you referenced. If you spot any differences in my requests that you think are worth noting, please point them out, but I tried to make near-identical requests.

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ id=integration_test_id importer_type_id=yum_importer importer_config:='{"feed": "file:///home/vagrant/runcible/test/fixtures/repositories/zoo5"}'

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/integration_test_id/actions/sync/

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ id=integration_test_id_clone importer_type_id=yum_importer importer_config:='{}'

http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/integration_test_id_clone/actions/associate/ source_repo_id=integration_test_id

After that, I can see that everything got copied:

$ pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  integration_test_id
Display Name:        integration_test_id
Description:         None
Content Unit Counts: 
  Distribution:     1
  Erratum:          2
  Package Category: 1
  Package Group:    2
  Rpm:              8

Id:                  integration_test_id_clone
Display Name:        integration_test_id_clone
Description:         None
Content Unit Counts: 
  Distribution:     1
  Erratum:          2
  Package Category: 1
  Package Group:    2
  Rpm:              8

Partha, do you have any suggestions for how we should proceed?

How are you determining that the groups didn't get copied? Maybe there's a bug in that workflow? If runcible was doing a search for groups based on "id", our current open bug #1478 about that attribute being renamed to "group_id" could make runcible think the groups didn't get copied.

Actions #10

Updated by ipanova@redhat.com about 8 years ago

So after some time i was able to re-produce the issue with both(CLI and API) methods.
Steps are:

1 create repoA with feed and sync it
2 create repoB
3 copy all units from repoA to repoB
------------------------------------
see all units copied( included package groups and categories)

4 remove repos( but don't clean orphans)
5 re-do steps 1-3( and use same names)
---------------------------------------
see the package groups and categories not copied

There is some issue with situation when unit key is already present and we need a deeper investigation

Actions #11

Updated by mhrivnak about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak

Added by mhrivnak about 8 years ago

Revision 250ba2f5 | View on GitHub

existing group and category units get associated correctly during sync

Previously in 2.8 development code, when a group or category existed as an orphan, the sync would incorrectly create the association so that the unit_id was not unit_id of the existing unit in the DB. This caused search to break for those units in that repo, which in turn caused copy to break.

https://pulp.plan.io/issues/1489 fixes #1489

Actions #12

Updated by mhrivnak about 8 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100
Actions #13

Updated by rbarlow about 8 years ago

  • Status changed from MODIFIED to 5
Actions #14

Updated by pthomas@redhat.com about 8 years ago

  • Status changed from 5 to 6

verified

Followed the steps from #10

[root@mgmt5 ~]# pulp-admin rpm repo create --repo-id zoo-copy\

Successfully created repository [zoo-copy]

[root@mgmt5 ~]# pulp-admin rpm repo copy all -f zoo -t zoo-copy
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
erratum:
RHEA-2012:0001
RHEA-2012:0002
RHEA-2012:0003
RHEA-2012:0004
package_category:
all
package_group:
birds
mammals
rpm:
bear-4.1-1-noarch
camel-0.1-1-noarch
cat-1.0-1-noarch
cheetah-1.25.3-5-noarch
chimpanzee-0.21-1-noarch
cockateel-3.1-1-noarch
cow-2.2-3-noarch
crow-0.8-1-noarch
dog-4.23-1-noarch
dolphin-3.10.232-1-noarch
duck-0.6-1-noarch
elephant-8.3-1-noarch
fox-1.1-2-noarch
frog-0.1-1-noarch
giraffe-0.67-2-noarch
gorilla-0.62-1-noarch
horse-0.22-2-noarch
kangaroo-0.2-1-noarch
lion-0.4-1-noarch
mouse-0.1.12-1-noarch
penguin-0.9.1-1-noarch
pike-2.2-1-noarch
shark-0.1-1-noarch
squirrel-0.1-1-noarch
stork-0.12-2-noarch
tiger-1.0-4-noarch
trout-0.12-1-noarch
walrus-0.71-1-noarch
walrus-5.21-1-noarch
whale-0.2-1-noarch
wolf-9.4-2-noarch
zebra-0.1-2-noarch

[root@mgmt5 ~]#

[root@mgmt5 ~]# pulp-admin rpm repo content group --repo-id zoo-copy
Conditional Package Names:
Default Package Names:
Id: birds
Mandatory Package Names: cockateel, duck, penguin, stork
Name: birds
Optional Package Names:
User Visible: True

Conditional Package Names:
Default Package Names:
Id: mammals
Mandatory Package Names: bear, camel, cat, cheetah, chimpanzee, cow, dog,
dolphin, elephant, fox, giraffe, gorilla, horse,
kangaroo, lion, mouse, squirrel, tiger, walrus,
whale, wolf, zebra
Name: mammals
Optional Package Names:
User Visible: True

[root@mgmt5 ~]#
[root@mgmt5 ~]#

[root@mgmt5 ~]# pulp-admin rpm repo delete --repo-id zoo
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [zoo] successfully deleted

[root@mgmt5 ~]# pulp-admin rpm repo delete --repo-id zoo-copy
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Repository [zoo-copy] successfully deleted

[root@mgmt5 ~]#
[root@mgmt5 ~]#
[root@mgmt5 ~]# pulp-admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
Successfully created repository [zoo]

[root@mgmt5 ~]# pulp-admin rpm repo sync run --repo-id zoo
--------------------------------------------------------------------
Synchronizing Repository [zoo]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[/]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@mgmt5 ~]# pulp-admin rpm repo content group --repo-id zoo
Conditional Package Names:
Default Package Names:
Id: mammals
Mandatory Package Names: bear, camel, cat, cheetah, chimpanzee, cow, dog,
dolphin, elephant, fox, giraffe, gorilla, horse,
kangaroo, lion, mouse, squirrel, tiger, walrus,
whale, wolf, zebra
Name: mammals
Optional Package Names:
User Visible: True

Conditional Package Names:
Default Package Names:
Id: birds
Mandatory Package Names: cockateel, duck, penguin, stork
Name: birds
Optional Package Names:
User Visible: True

[root@mgmt5 ~]# pulp-admin rpm repo create --repo-id zoo-copy
Successfully created repository [zoo-copy]

[root@mgmt5 ~]# pulp-admin rpm repo copy all -f zoo -t zoo-copy
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
erratum:
RHEA-2012:0001
RHEA-2012:0002
RHEA-2012:0003
RHEA-2012:0004
package_category:
all
package_group:
birds
mammals
rpm:
bear-4.1-1-noarch
camel-0.1-1-noarch
cat-1.0-1-noarch
cheetah-1.25.3-5-noarch
chimpanzee-0.21-1-noarch
cockateel-3.1-1-noarch
cow-2.2-3-noarch
crow-0.8-1-noarch
dog-4.23-1-noarch
dolphin-3.10.232-1-noarch
duck-0.6-1-noarch
elephant-8.3-1-noarch
fox-1.1-2-noarch
frog-0.1-1-noarch
giraffe-0.67-2-noarch
gorilla-0.62-1-noarch
horse-0.22-2-noarch
kangaroo-0.2-1-noarch
lion-0.4-1-noarch
mouse-0.1.12-1-noarch
penguin-0.9.1-1-noarch
pike-2.2-1-noarch
shark-0.1-1-noarch
squirrel-0.1-1-noarch
stork-0.12-2-noarch
tiger-1.0-4-noarch
trout-0.12-1-noarch
walrus-0.71-1-noarch
walrus-5.21-1-noarch
whale-0.2-1-noarch
wolf-9.4-2-noarch
zebra-0.1-2-noarch

[root@mgmt5 ~]# pulp-admin rpm repo content group --repo-id zoo-copy
Conditional Package Names:
Default Package Names:
Id: mammals
Mandatory Package Names: bear, camel, cat, cheetah, chimpanzee, cow, dog,
dolphin, elephant, fox, giraffe, gorilla, horse,
kangaroo, lion, mouse, squirrel, tiger, walrus,
whale, wolf, zebra
Name: mammals
Optional Package Names:
User Visible: True

Conditional Package Names:
Default Package Names:
Id: birds
Mandatory Package Names: cockateel, duck, penguin, stork
Name: birds
Optional Package Names:
User Visible: True

[root@mgmt5 ~]#

Actions #15

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #17

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF