Actions
Issue #1931
closed
FieldDoesNotExist: The field '_rpm_references' does not exist on the document 'Errata'
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.0
Platform Release:
2.8.4
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 3
Quarter:
Description
Hitting an error when trying to copy errata:
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) Exception from importer [yum_importer] while importing units into repository [mydomain_com-Production_7-13_0-CentOS_7-x86_64_extras]
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) Traceback (most recent call last):
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) units=transfer_units)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 57, in import_units
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) return associate.associate(source_repo, dest_repo, import_conduit, config, units)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 53, in associate
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) associated_units = set([_associate_unit(dest_repo, unit) for unit in units])
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 227, in find_repo_content_units
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) for unit in qs:
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1411, in next
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 730, in _from_son
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 52, in __init__
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) super(AutoRetryDocument, self).__init__(*args, **kwargs)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 80, in __init__
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) raise FieldDoesNotExist(msg)
May 19 09:49:08 puppet01 pulp: pulp.server.managers.repo.unit_association:ERROR: (26475-59264) FieldDoesNotExist: The field '_rpm_references' does not exist on the document 'Errata'
May 19 09:49:08 puppet01 pulp: pulp.server.async.tasks:INFO: Task failed : [19c9d4bb-306d-4d7d-8023-f7498c9f267e]
Looks like some errata do contain the field:
> db.units_erratum.find({ _rpm_references: { $exists: false}});
{ "_id" : "1705defe-9db4-438d-87c7-994988d2fc4e", "status" : "stable", "updated" : "", "description" : "Latest upstream.\n\n----\n\nDB upgrade adds taskotron rules to the defaults.", "issued" : "2016-03-22 01:28:15", "_last_updated" : 1463464999, "pushcount" : "", "references" : [ ], "_content_type_id" : "erratum", "_storage_path" : null, "reboot_suggested" : false, "severity" : "", "rights" : "Copyright (C) 2015 Red Hat, Inc. and others.", "_ns" : "units_erratum", "title" : "python-fmn-lib-0.8.1-1.el7", "solution" : "", "summary" : "python-fmn-lib-0.8.1-1.el7 bugfix update", "version" : "2.0", "release" : "Fedora EPEL 7", "type" : "bugfix", "pkglist" : [ { "packages" : [ { "src" : "https://download.fedoraproject.org/pub/fedora/linux/updates/7/i386/p/python-fmn-lib-0.8.1-1.el7.noarch.rpm", "name" : "python-fmn-lib", "sum" : null, "filename" : "python-fmn-lib-0.8.1-1.el7.noarch.rpm", "epoch" : "0", "version" : "0.8.1", "release" : "1.el7", "arch" : "noarch" }, { "src" : "https://download.fedoraproject.org/pub/fedora/linux/updates/7/SRPMS/p/python-fmn-lib-0.8.1-1.el7.src.rpm", "name" : "python-fmn-lib", "sum" : null, "filename" : "python-fmn-lib-0.8.1-1.el7.src.rpm", "epoch" : "0", "version" : "0.8.1", "release" : "1.el7", "arch" : "src" } ], "name" : "Fedora EPEL 7", "short" : "EPEL-7" } ], "pulp_user_metadata" : { }, "errata_id" : "FEDORA-EPEL-2016-80c92ab4c6", "errata_from" : "updates@fedoraproject.org" }
> db.units_erratum.find({ _rpm_references: { $exists: true}});
{ "_id" : "a8dfa22f-5130-498e-85be-8585d8110b15", "_rpm_references" : [ "32112e39-33c5-48a0-8167-7fbb01012e5a", "9465eb4a-08f1-44ef-8115-66c0c92d7ec3", "7ff0d24e-3612-4b96-bfad-36cd2ced9716", "e12650eb-97c3-41aa-9a26-07f45ef594a7", "de48cb48-0069-4e37-b25b-1d833da348ef", "5fbafa80-8dd4-4806-95a0-22d285d0fb29", "c6f81ae2-9f4d-4b8c-bde2-3f8c1c9e2d2b", "ce24eaf5-8e7e-4804-8b2a-fd66b3dd0154", "90093189-ed3f-4919-9acf-823ca38e4cb8", "72388ca9-7fdf-4947-bd02-9b5e718b6bfa", "e2e6e66d-ec25-49f6-ae7b-85723b16a7d9", "3d6e84ce-f95a-4e83-8958-a3f1f0129079" ], "issued" : "2015-12-09 19:18:47", "references" : [ { "href" : "https://rhn.redhat.com/errata/RHSA-2015-2552.html", "type" : "Security Advisory", "id" : "CESA-2015:2552", "title" : "Important CentOS kernel Security Update" }, { "href" : "http://lists.centos.org/pipermail/centos-cr-announce/2015-December/002732.html", "type" : "Security Advisory", "id" : "CESA-2015:2552", "title" : "Important CentOS kernel Security Update" } ], "_content_type_id" : "erratum", "severity" : null, "title" : "Important CentOS kernel Security Update", "_ns" : "units_erratum", "version" : "1", "reboot_suggested" : false, "type" : "security", "pkglist" : [ { "packages" : [ { "src" : "N/A", "name" : "kernel-tools", "arch" : "x86_64", "sums" : "b9e1ce5fe49ea410f9261ed6ca0e73cf3910e481030a7552d41cb6e3a51a299a", "filename" : "kernel-tools-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-abi-whitelists", "arch" : "noarch", "sums" : "f43d8ed284936508b8934b9907a65899f7a06e61aa84253e6187f65b0177450b", "filename" : "kernel-abi-whitelists-3.10.0-327.3.1.el7.noarch.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-debug-devel", "arch" : "x86_64", "sums" : "ad3dcdd7fdbfb72378d3f87aed0b0c72059afe71ed97fbd05a1fab154714c0d0", "filename" : "kernel-debug-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "perf", "arch" : "x86_64", "sums" : "c159c6e6ce7627c52bafddd60d5c382fd299de843b96b319abc948acb55ec8ea", "filename" : "perf-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-headers", "arch" : "x86_64", "sums" : "7da46d6669072c7851c909587b57ad3fe6b2e3914da0b6003d5043a7995a6921", "filename" : "kernel-headers-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-doc", "arch" : "noarch", "sums" : "e020ed78ae5aac637e02bec645e2e39fa07be658554599ea6eed77fb97c558a5", "filename" : "kernel-doc-3.10.0-327.3.1.el7.noarch.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-tools-libs-devel", "arch" : "x86_64", "sums" : "72f1f15c613957961d4900eb0bf0722069cb51010627ae4291440d3fb110575b", "filename" : "kernel-tools-libs-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel", "arch" : "x86_64", "sums" : "ba971bc760b43b09045ecce3cc189c2288d38a4ac234cd61038fdb82f4c69a2c", "filename" : "kernel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-debug", "arch" : "x86_64", "sums" : "879edb6ccdeffefe015cab4f38b7624e6e9e6411641e202c8f05a7cccd258c54", "filename" : "kernel-debug-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "python-perf", "arch" : "x86_64", "sums" : "d7bf5f61e0808e77b9507eceacd6a9028f2e6d4e939d80a65670df5126bd7df6", "filename" : "python-perf-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-devel", "arch" : "x86_64", "sums" : "a7a2c92aca038115241fe0eed65a70060100a5843acf15dfd4c7b902c0fcc59a", "filename" : "kernel-devel-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" }, { "src" : "N/A", "name" : "kernel-tools-libs", "arch" : "x86_64", "sums" : "597a75e56ec41fba5078e562c2da0e493c0178fc64c0c853d092899cfc770e06", "filename" : "kernel-tools-libs-3.10.0-327.3.1.el7.x86_64.rpm", "epoch" : "0", "version" : "3.10.0", "release" : "327.3.1.el7", "type" : "sha256" } ], "name" : "327.3.1.el7", "short" : "" } ], "status" : "final", "updated" : "2015-12-09 19:18:47", "description" : "Not available", "_last_updated" : 1454411526, "pushcount" : "1", "_storage_path" : null, "rights" : null, "solution" : null, "summary" : null, "release" : "327.3.1.el7", "pulp_user_metadata" : { }, "errata_id" : "CESA-2015:2552", "errata_from" : "centos-announce@centos.org" }
Updated by daviddavis over 8 years ago
Katello bug:
Updated by daviddavis over 8 years ago
- Severity changed from 2. Medium to 3. High
Updated by trunet over 8 years ago
I just saw that I'm also getting this when trying to sync a repository, note the:
The field '_erratum_references' does not exist on the document 'RPM'
{"contents_changed"=>true,
"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.sync.sync",
"_href"=>"/pulp/api/v2/tasks/43187cb3-7637-4450-a5ac-d8d98196323f/",
"task_id"=>"43187cb3-7637-4450-a5ac-d8d98196323f",
"tags"=>
["pulp:repository:mydomain_com-CentOS_7-x86_64_extras",
"pulp:action:sync"],
"finish_time"=>"2016-05-24T06:02:28Z",
"_ns"=>"task_status",
"start_time"=>"2016-05-24T06:02:17Z",
"traceback"=>
"Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 473, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 103, in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 437, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 810, in sync\n raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\nPulpExecutionException: Importer indicated a failed response\n",
"spawned_tasks"=>[],
"progress_report"=>
{"yum_importer"=>
{"content"=>
{"size_total"=>0,
"items_left"=>0,
"items_total"=>0,
"state"=>"FAILED",
"size_left"=>0,
"details"=>
{"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
"error"=>
"The field '_erratum_references' does not exist on the document 'RPM'",
"error_details"=>[]},
"comps"=>{"state"=>"NOT_STARTED"},
"purge_duplicates"=>{"state"=>"NOT_STARTED"},
"distribution"=>
{"items_total"=>0,
"state"=>"NOT_STARTED",
"error_details"=>[],
"items_left"=>0},
"errata"=>{"state"=>"NOT_STARTED"},
"metadata"=>{"state"=>"FINISHED"}}},
"queue"=>"reserved_resource_worker-0@puppet01.mydomain.com.dq",
"state"=>"error",
"worker_name"=>"reserved_resource_worker-0@puppet01.mydomain.com",
"result"=>nil,
"error"=>
{"code"=>"PLP0000",
"data"=>{},
"description"=>"Importer indicated a failed response",
"sub_errors"=>[]},
"_id"=>{"$oid"=>"5743ede07c9ac706a09a9230"},
"id"=>"5743ede07c9ac706a09a9230"}]}
Updated by bmbouter over 8 years ago
- Sprint/Milestone set to 21
- Platform Release set to 2.8.4
Updated by bmbouter over 8 years ago
- Priority changed from Normal to High
- Platform Release deleted (
2.8.4)
Raising the priority since this is an issue that affects both users and Katello. This regression was introduced with 2.8.0.
Strangely, the _rpm_references field is not referenced anywhere in pulp_rpm or platform's git history. I checked by running this command on each repo with up-to-date refs:
git grep _rpm_references $(git rev-list --all)
Did this come through in through the pre-mongoengine, dict-to-mongo code? If not, how could it have gotten into the database?
Updated by mhrivnak over 8 years ago
I did a similar search and came up empty. I suspect the attribute name was generated dynamically. Fun!
If you look at old sync code and find where errata were handled, that may give you some clues. You'll hopefully see something that creates a link.
Also, I think the sync conduit might have (or used to have) a function that linked units together. If you find that, it might explain things, and I bet that's what the old sync code used.
Updated by bmbouter over 8 years ago
There is no reading of this attribute today. Given that, I think writing a migration to remove that field from all Errata Documents would resolve it.
What do you think?
Updated by mhrivnak over 8 years ago
That much is definitely true, and I agree a migration to remove the field would solve the problem. Knowing how the problem manifested would just be a bonus. :)
Updated by trunet over 8 years ago
I'm using https://github.com/brdude/pulp_centos_errata_import to import erratas for my CentOS repositories. It always worked(previous version was katello 2.4 and I think pulp 2.6), but it's failing now.
Updated by bmbouter over 8 years ago
Fixing this is a very simple migration. It would be just like this one[0] except for the Errata type and the field '_rpm_references'.
Updated by ipanova@redhat.com over 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ipanova@redhat.com
Added by ipanova@redhat.com over 8 years ago
Updated by ipanova@redhat.com over 8 years ago
- Status changed from ASSIGNED to POST
- Platform Release set to 2.8.5
Updated by ipanova@redhat.com over 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset 558cb7fb84daa1e4e7b2718f07c6b34929cee10e.
Updated by semyers over 8 years ago
- Priority changed from High to Urgent
- Platform Release changed from 2.8.5 to 2.8.4
Added by ipanova@redhat.com over 8 years ago
Revision bba907a2 | View on GitHub
Remove unused fields from units_erratum collection
closes #1931 https://pulp.plan.io/issues/1931
The _rpm_references field was present in earlier versions of Pulp
but is no longer used. It is not included in the mongoengine
Errata model anymore.
Updated by pthomas@redhat.com over 8 years ago
- Status changed from 5 to 6
verified
[root@mgmt8 ~]# pulp-admin rpm repo copy errata -f rhel6 -t errata-copy --recursive
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
Copied:
erratum: 3325
[root@mgmt8 ~]# pulp-admin rpm repo publish run --repo-id errata-copy
+----------------------------------------------------------------------+
Publishing Repository [errata-copy]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
3325 of 3325 items
... completed
Publishing Comps file
[-]
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
[root@mgmt8 ~]#
Updated by semyers over 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
.
Remove unused fields from units_erratum collection
closes #1931 https://pulp.plan.io/issues/1931
The
_rpm_referencesfield was present in earlier versions of Pulp but is no longer used. It is not included in the mongoengine Errata model anymore.