Project

Profile

Help

Issue #1931

FieldDoesNotExist: The field '_rpm_references' does not exist on the document 'Errata'

Added by daviddavis over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.0
Platform Release:
2.8.4
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 3
Quarter:

Description

Hitting an error when trying to copy errata:

May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) Exception from importer [yum_importer] while importing units into repository [mydomain_com-Production_7-13_0-CentOS_7-x86_64_extras]
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) Traceback (most recent call last):
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     units=transfer_units)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 57, in import_units
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     return associate.associate(source_repo, dest_repo, import_conduit, config, units)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 53, in associate
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 227, in find_repo_content_units
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     for unit in qs:
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1411, in next
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 730, in _from_son
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 52, in __init__
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     super(AutoRetryDocument, self).__init__(*args, **kwargs)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)   File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 80, in __init__
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264)     raise FieldDoesNotExist(msg)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) FieldDoesNotExist: The field '_rpm_references' does not exist on the document 'Errata'
May 19 09:49:08 puppet01 pulp: pulp.server.async.tasks:INFO: Task failed : [19c9d4bb-306d-4d7d-8023-f7498c9f267e]

Looks like some errata do contain the field:

> db.units_erratum.find({ _rpm_references: { $exists: false}});
{ "_id" : "1705defe-9db4-438d-87c7-994988d2fc4e", "status" : "stable", "updated" : "", "description" : "Latest upstream.\n\n----\n\nDB upgrade adds taskotron rules to the defaults.", "issued" : "2016-03-22 01:28:15", "_last_updated" : 1463464999, "pushcount" : "", "references" : [ ], "_content_type_id" : "erratum", "_storage_path" : null, "reboot_suggested" : false, "severity" : "", "rights" : "Copyright (C) 2015 Red Hat, Inc. and others.", "_ns" : "units_erratum", "title" : "python-fmn-lib-0.8.1-1.el7", "solution" : "", "summary" : "python-fmn-lib-0.8.1-1.el7 bugfix update", "version" : "2.0", "release" : "Fedora EPEL 7", "type" : "bugfix", "pkglist" : [ { "packages" : [ { "src" : "https://download.fedoraproject.org/pub/fedora/linux/updates/7/i386/p/python-fmn-lib-0.8.1-1.el7.noarch.rpm", "name" : "python-fmn-lib", "sum" : null, "filename" : "python-fmn-lib-0.8.1-1.el7.noarch.rpm", "epoch" : "0", "version" : "0.8.1", "release" : "1.el7", "arch" : "noarch" }, { "src" : "https://download.fedoraproject.org/pub/fedora/linux/updates/7/SRPMS/p/python-fmn-lib-0.8.1-1.el7.src.rpm", "name" : "python-fmn-lib", "sum" : null, "filename" : "python-fmn-lib-0.8.1-1.el7.src.rpm", "epoch" : "0", "version" : "0.8.1", "release" : "1.el7", "arch" : "src" } ], "name" : "Fedora EPEL 7", "short" : "EPEL-7" } ], "pulp_user_metadata" : {  }, "errata_id" : "FEDORA-EPEL-2016-80c92ab4c6", "errata_from" : "updates@fedoraproject.org" }

> db.units_erratum.find({ _rpm_references: { $exists: true}});
{ "_id" : "a8dfa22f-5130-498e-85be-8585d8110b15", "_rpm_references" : [ "32112e39-33c5-48a0-8167-7fbb01012e5a", "9465eb4a-08f1-44ef-8115-66c0c92d7ec3", "7ff0d24e-3612-4b96-bfad-36cd2ced9716", "e12650eb-97c3-41aa-9a26-07f45ef594a7", "de48cb48-0069-4e37-b25b-1d833da348ef", "5fbafa80-8dd4-4806-95a0-22d285d0fb29", "c6f81ae2-9f4d-4b8c-bde2-3f8c1c9e2d2b", "ce24eaf5-8e7e-4804-8b2a-fd66b3dd0154", "90093189-ed3f-4919-9acf-823ca38e4cb8", "72388ca9-7fdf-4947-bd02-9b5e718b6bfa", "e2e6e66d-ec25-49f6-ae7b-85723b16a7d9", "3d6e84ce-f95a-4e83-8958-a3f1f0129079" ], "issued" : "2015-12-09 19:18:47", "references" : [ { "href" : "https://rhn.redhat.com/errata/RHSA-2015-2552.html", "type" : "Security Advisory", "id" : "CESA-2015:2552", "title" : "Important CentOS kernel Security Update" }, { "href" : "http://lists.centos.org/pipermail/centos-cr-announce/2015-December/002732.html", "type" : "Security Advisory", "id" : "CESA-2015:2552", "title" : "Important CentOS kernel Security Update" } ], "_content_type_id" : "erratum", "severity" : null, "title" : "Important CentOS kernel Security Update", "_ns" : "units_erratum", "version" : "1", "reboot_suggested" : false, "type" : "security", "pkglist" : [ { "packages" : [ { "src" : "N/A", "name" : "kernel-tools", "arch" : "x86_64", "sums" : "b9e1ce5fe49ea410f9261ed6ca0e73cf3910e481030a7552d41cb6e3a51a299a", "filename" : "kernel-tools-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-abi-whitelists", "arch" : "noarch", "sums" : "f43d8ed284936508b8934b9907a65899f7a06e61aa84253e6187f65b0177450b", "filename" : "kernel-abi-whitelists-3.10.0-327.3.1.el7.noarch.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-debug-devel", "arch" : "x86_64", "sums" : "ad3dcdd7fdbfb72378d3f87aed0b0c72059afe71ed97fbd05a1fab154714c0d0", "filename" : "kernel-debug-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "perf", "arch" : "x86_64", "sums" : "c159c6e6ce7627c52bafddd60d5c382fd299de843b96b319abc948acb55ec8ea", "filename" : "perf-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-headers", "arch" : "x86_64", "sums" : "7da46d6669072c7851c909587b57ad3fe6b2e3914da0b6003d5043a7995a6921", "filename" : "kernel-headers-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-doc", "arch" : "noarch", "sums" : "e020ed78ae5aac637e02bec645e2e39fa07be658554599ea6eed77fb97c558a5", "filename" : "kernel-doc-3.10.0-327.3.1.el7.noarch.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-tools-libs-devel", "arch" : "x86_64", "sums" : "72f1f15c613957961d4900eb0bf0722069cb51010627ae4291440d3fb110575b", "filename" : "kernel-tools-libs-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel", "arch" : "x86_64", "sums" : "ba971bc760b43b09045ecce3cc189c2288d38a4ac234cd61038fdb82f4c69a2c", "filename" : "kernel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-debug", "arch" : "x86_64", "sums" : "879edb6ccdeffefe015cab4f38b7624e6e9e6411641e202c8f05a7cccd258c54", "filename" : "kernel-debug-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "python-perf", "arch" : "x86_64", "sums" : "d7bf5f61e0808e77b9507eceacd6a9028f2e6d4e939d80a65670df5126bd7df6", "filename" : "python-perf-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-devel", "arch" : "x86_64", "sums" : "a7a2c92aca038115241fe0eed65a70060100a5843acf15dfd4c7b902c0fcc59a", "filename" : "kernel-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-tools-libs", "arch" : "x86_64", "sums" : "597a75e56ec41fba5078e562c2da0e493c0178fc64c0c853d092899cfc770e06", "filename" : "kernel-tools-libs-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" } ], "name" : "327.3.1.el7", "short" : "" } ], "status" : "final", "updated" : "2015-12-09 19:18:47", "description" : "Not available", "_last_updated" : 1454411526, "pushcount" : "1", "_storage_path" : null, "rights" : null, "solution" : null, "summary" : null, "release" : "327.3.1.el7", "pulp_user_metadata" : {  }, "errata_id" : "CESA-2015:2552", "errata_from" : "centos-announce@centos.org" }

Associated revisions

Revision 558cb7fb View on GitHub
Added by ipanova@redhat.com over 4 years ago

Remove unused fields from units_erratum collection

closes #1931 https://pulp.plan.io/issues/1931

The _rpm_references field was present in earlier versions of Pulp but is no longer used. It is not included in the mongoengine Errata model anymore.

Revision bba907a2 View on GitHub
Added by ipanova@redhat.com over 4 years ago

Remove unused fields from units_erratum collection

closes #1931 https://pulp.plan.io/issues/1931

The _rpm_references field was present in earlier versions of Pulp but is no longer used. It is not included in the mongoengine Errata model anymore.

History

#2 Updated by daviddavis over 4 years ago

  • Severity changed from 2. Medium to 3. High

#3 Updated by trunet over 4 years ago

I just saw that I'm also getting this when trying to sync a repository, note the:
The field '_erratum_references' does not exist on the document 'RPM'

{"contents_changed"=>true,
 "pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/43187cb3-7637-4450-a5ac-d8d98196323f/",
    "task_id"=>"43187cb3-7637-4450-a5ac-d8d98196323f",
    "tags"=>
     ["pulp:repository:mydomain_com-CentOS_7-x86_64_extras",
      "pulp:action:sync"],
    "finish_time"=>"2016-05-24T06:02:28Z",
    "_ns"=>"task_status",
    "start_time"=>"2016-05-24T06:02:17Z",
    "traceback"=>
     "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 473, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 103, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 437, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 810, in sync\n    raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\nPulpExecutionException: Importer indicated a failed response\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"size_total"=>0,
          "items_left"=>0,
          "items_total"=>0,
          "state"=>"FAILED",
          "size_left"=>0,
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "error"=>
           "The field '_erratum_references' does not exist on the document 'RPM'",
          "error_details"=>[]},
        "comps"=>{"state"=>"NOT_STARTED"},
        "purge_duplicates"=>{"state"=>"NOT_STARTED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"NOT_STARTED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"NOT_STARTED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"reserved_resource_worker-0@puppet01.mydomain.com.dq",
    "state"=>"error",
    "worker_name"=>"reserved_resource_worker-0@puppet01.mydomain.com",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>"Importer indicated a failed response",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5743ede07c9ac706a09a9230"},
    "id"=>"5743ede07c9ac706a09a9230"}]}

#4 Updated by bmbouter over 4 years ago

  • Sprint/Milestone set to 21
  • Platform Release set to 2.8.4

#5 Updated by bmbouter over 4 years ago

  • Priority changed from Normal to High
  • Platform Release deleted (2.8.4)

Raising the priority since this is an issue that affects both users and Katello. This regression was introduced with 2.8.0.

Strangely, the _rpm_references field is not referenced anywhere in pulp_rpm or platform's git history. I checked by running this command on each repo with up-to-date refs:

git grep _rpm_references $(git rev-list --all)

Did this come through in through the pre-mongoengine, dict-to-mongo code? If not, how could it have gotten into the database?

#6 Updated by mhrivnak over 4 years ago

I did a similar search and came up empty. I suspect the attribute name was generated dynamically. Fun!

If you look at old sync code and find where errata were handled, that may give you some clues. You'll hopefully see something that creates a link.

Also, I think the sync conduit might have (or used to have) a function that linked units together. If you find that, it might explain things, and I bet that's what the old sync code used.

#7 Updated by bmbouter over 4 years ago

There is no reading of this attribute today. Given that, I think writing a migration to remove that field from all Errata Documents would resolve it.

What do you think?

#8 Updated by mhrivnak over 4 years ago

That much is definitely true, and I agree a migration to remove the field would solve the problem. Knowing how the problem manifested would just be a bonus. :)

#9 Updated by trunet over 4 years ago

I'm using https://github.com/brdude/pulp_centos_errata_import to import erratas for my CentOS repositories. It always worked(previous version was katello 2.4 and I think pulp 2.6), but it's failing now.

#10 Updated by dkliban@redhat.com over 4 years ago

  • Triaged changed from No to Yes

#11 Updated by bmbouter over 4 years ago

Fixing this is a very simple migration. It would be just like this one[0] except for the Errata type and the field '_rpm_references'.

[0]: https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/migrations/0026_remove_distribution_field.py

#12 Updated by ipanova@redhat.com over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#13 Updated by ipanova@redhat.com over 4 years ago

  • Status changed from ASSIGNED to POST
  • Platform Release set to 2.8.5

#14 Updated by ipanova@redhat.com over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#15 Updated by semyers over 4 years ago

  • Priority changed from High to Urgent
  • Platform Release changed from 2.8.5 to 2.8.4

#16 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to 5

#17 Updated by pthomas@redhat.com over 4 years ago

  • Status changed from 5 to 6

verified

[root@mgmt8 ~]# pulp-admin rpm repo copy errata -f rhel6 -t errata-copy --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
  erratum: 3325

[root@mgmt8 ~]# pulp-admin rpm repo publish run --repo-id errata-copy
+----------------------------------------------------------------------+
                  Publishing Repository [errata-copy]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[-]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3325 of 3325 items
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@mgmt8 ~]# 

#18 Updated by semyers over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#22 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 3

#23 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (21)

#24 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF