Project

Profile

Help

Issue #1478

POST request to /pulp/api/v2/repositories/<reponame>/search/units/ does not transform id correctly when unit results returned

Added by bmbouter over 5 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

On an empty/fresh pulp system, run the following:

pulp-admin login -u admin -p admin

pulp-admin rpm repo create --repo-id rawhide --feed http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/x86_64/os/ --skip=rpm,drpm,erratum

pulp-admin rpm repo sync run --repo-id rawhide

Now try to search for all units of type distribution by running:

pulp-admin -vv rpm repo content distribution --repo-id rawhide

You'll receive a traceback as pulp-admin tries to parse the results:

Traceback (most recent call last):
  File "/home/vagrant/devel/pulp/client_lib/pulp/client/extensions/core.py", line 474, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.7/site-packages/okaara/cli.py", line 974, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/home/vagrant/devel/pulp/client_lib/pulp/client/extensions/extensions.py", line 210, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/home/vagrant/devel/pulp_rpm/extensions_admin/pulp_rpm/extensions/admin/contents.py", line 337, in distribution
    self.run_search([TYPE_DISTRIBUTION], self.write_distro, **kwargs)
  File "/home/vagrant/devel/pulp_rpm/extensions_admin/pulp_rpm/extensions/admin/contents.py", line 171, in run_search
    out_func(units)
  File "/home/vagrant/devel/pulp_rpm/extensions_admin/pulp_rpm/extensions/admin/contents.py", line 356, in write_distro
    'id': distro['id'],
KeyError: 'id'

The issue is not in Pulp admin. The data returned by the server is not correctly transforming the 'distribution_id' field back into the expected 'id' field. See this snip of data returned:

<snip>
      "family": "Fedora", 
      "distribution_id": "ks-Fedora--rawhide-x86_64", 
      "timestamp": 1451988072.6876094, 
      "_last_updated": 1452026990, 
      "packagedir": "", 
      "variant": "", 
      "downloaded": true, 
      "version": "rawhide", 
      "version_sort_index": "$rawhide", 
      "pulp_user_metadata": {}, 
      "_content_type_id": "distribution", 
      "_id": "4432836c-b6f4-45ea-b579-1ae1350a7ff9", 
      "arch": "x86_64", 
      "_ns": "units_distribution"
</snip>

The field renaming occurs with the 2.8.0 release so it is new since then. The fix is the have the server transform the renamed fields correctly.


Related issues

Related to RPM Support - Issue #1490: Errata IDs not showing up correctly in the APICLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Issue #1505: fields missing when pulp-admin tries to present a DistributionCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 6fdf3c7d View on GitHub
Added by semyers over 5 years ago

content unit search views use content unit serializers

Recent changes in the pymongo > mongoengine conversion have resulted in different ways to represent mongo documents in the REST api views; the correct way is to use the mongoengine Document serializers wherever possible. When that isn't possible, remap_fields_with_serializer exists to use the information in a mongoengine serializer to modify serialized results properly.

Anything that is using the serialization functions in pulp.server.webservices.views.serializers.content should automaticaly benefit from this change, which implicitly fixes redmine #1490.

These changes can also be used to benefit views that don't call the existing serialization functions by explicitly using the new remap_fields_with_serializer function as-needed, such as RepoUnitSearch, which will help (but not fix) redmine #1478.

fixes #1490 https://pulp.plan.io/issues/1490

re #1478 https://pulp.plan.io/issues/1478

Revision 6fdf3c7d View on GitHub
Added by semyers over 5 years ago

content unit search views use content unit serializers

Recent changes in the pymongo > mongoengine conversion have resulted in different ways to represent mongo documents in the REST api views; the correct way is to use the mongoengine Document serializers wherever possible. When that isn't possible, remap_fields_with_serializer exists to use the information in a mongoengine serializer to modify serialized results properly.

Anything that is using the serialization functions in pulp.server.webservices.views.serializers.content should automaticaly benefit from this change, which implicitly fixes redmine #1490.

These changes can also be used to benefit views that don't call the existing serialization functions by explicitly using the new remap_fields_with_serializer function as-needed, such as RepoUnitSearch, which will help (but not fix) redmine #1478.

fixes #1490 https://pulp.plan.io/issues/1490

re #1478 https://pulp.plan.io/issues/1478

Revision c7e79237 View on GitHub
Added by semyers over 5 years ago

RepoUnitSearch remaps unit fields using legacy field remapper

The reproducer in #1478 will still fail, but with a problem that appears to be specific to the distribution type in pulp_rpm. That problem is being tracked in https://pulp.plan.io/issues/1505

https://pulp.plan.io/issues/1478 fixes #1478

re #1505

Revision c7e79237 View on GitHub
Added by semyers over 5 years ago

RepoUnitSearch remaps unit fields using legacy field remapper

The reproducer in #1478 will still fail, but with a problem that appears to be specific to the distribution type in pulp_rpm. That problem is being tracked in https://pulp.plan.io/issues/1505

https://pulp.plan.io/issues/1478 fixes #1478

re #1505

History

#1 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#2 Updated by paji@redhat.com over 5 years ago

To add to this issue the response returned by

RestClient.post "https://katello-pulp-nightly.example.com/pulp/api/v2/repositories/integration_test_id/search/units/", "{\"criteria\":{\"type_ids\":[\"erratum\"]}}", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"37", "accept"=>"application/json", "content_type"=>"application/json"
\n# => 200 OK | application/json 645 bytes

contains a bunch of errata with no "id" column under the "metadata" section. In other words

x = respons(post request to search errata)
x[0]["metadata"]["id"]

does not exist in 2.8 while it did in 2.6.

#3 Updated by mhrivnak over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to semyers
  • Triaged changed from No to Yes

#4 Updated by semyers over 5 years ago

search views don't appear to respect model serializers by default, but I think I see where to hook them

#5 Updated by semyers over 5 years ago

  • Related to Issue #1489: Unit copy not copying package groups or categories added

#6 Updated by semyers over 5 years ago

  • Related to deleted (Issue #1489: Unit copy not copying package groups or categories)

#7 Updated by semyers over 5 years ago

  • Related to Issue #1490: Errata IDs not showing up correctly in the API added

#8 Updated by semyers over 5 years ago

  • Status changed from ASSIGNED to MODIFIED

https://github.com/pulp/pulp/pull/2310

"old" pymongo objects weren't been serialized when coming out of some search api views dealing with content units. The solution (after normalizing the model serializer location to model.SERIALIZER) was to create a helper function to serialize those objects using the field remapping rules from the newer mongoengine serializers. Most search views will transparently benefit from this change, with the issue in #1490 being a noted exception.

#9 Updated by semyers over 5 years ago

  • Status changed from MODIFIED to ASSIGNED

Doh, I have the issues backwards. Sorry!

This is the notable exception to the fix in #1490, and I'm still working on this one.

Also, #1478 and #1490 aren't duplicates. :)

#10 Updated by semyers over 5 years ago

  • Related to Issue #1505: fields missing when pulp-admin tries to present a Distribution added

#11 Updated by semyers over 5 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#12 Updated by rbarlow about 5 years ago

  • Status changed from MODIFIED to 5

#13 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#14 Updated by bmbouter about 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF