Actions
Issue #1754
closed
Repo sync with fails on an upgraded pulp server
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.8.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
Syncing a repo with on_demand download policy on an upgraded pulp-server fails
1. Install and configure a prior pulp release. (In my case 2.6)
2. I had some repos created and synced
3. Stop the services and update to 2.8
4. Run migration
5. Restart services
6. Install pulp streamer & squid
7. Create a repo with on_demand download policy
8. Run repo sync on that repo
Pulp 2.6 setup http://pulp-qe.etherpad.corp.redhat.com/102
After upgrade
[root@mgmt5 ~]# rpm -qa |grep pulp
pulp-server-2.8.0-0.8.beta.el7.noarch
pulp-puppet-consumer-extensions-2.8.0-0.8.beta.el7.noarch
python-pulp-streamer-2.8.0-0.8.beta.el7.noarch
pulp-admin-client-2.8.0-0.8.beta.el7.noarch
python-pulp-oid_validation-2.8.0-0.8.beta.el7.noarch
pulp-rpm-plugins-2.8.0-0.8.beta.el7.noarch
pulp-puppet-handlers-2.8.0-0.8.beta.el7.noarch
python-pulp-common-2.8.0-0.8.beta.el7.noarch
python-pulp-agent-lib-2.8.0-0.8.beta.el7.noarch
pulp-consumer-client-2.8.0-0.8.beta.el7.noarch
pulp-nodes-admin-extensions-2.8.0-0.8.beta.el7.noarch
pulp-puppet-admin-extensions-2.8.0-0.8.beta.el7.noarch
python-pulp-bindings-2.8.0-0.8.beta.el7.noarch
python-pulp-puppet-common-2.8.0-0.8.beta.el7.noarch
python-pulp-docker-common-2.0.0-0.8.beta.el7.noarch
python-pulp-repoauth-2.8.0-0.8.beta.el7.noarch
python-kombu-3.0.33-4.pulp.el7.noarch
pulp-docker-plugins-2.0.0-0.8.beta.el7.noarch
pulp-selinux-2.8.0-0.8.beta.el7.noarch
pulp-docker-admin-extensions-2.0.0-0.8.beta.el7.noarch
pulp-rpm-admin-extensions-2.8.0-0.8.beta.el7.noarch
pulp-rpm-yumplugins-2.8.0-0.8.beta.el7.noarch
python-pulp-client-lib-2.8.0-0.8.beta.el7.noarch
pulp-puppet-plugins-2.8.0-0.8.beta.el7.noarch
pulp-rpm-handlers-2.8.0-0.8.beta.el7.noarch
python-pulp-rpm-common-2.8.0-0.8.beta.el7.noarch
pulp-nodes-common-2.8.0-0.8.beta.el7.noarch
pulp-nodes-parent-2.8.0-0.8.beta.el7.noarch
pulp-agent-2.8.0-0.8.beta.el7.noarch
pulp-rpm-consumer-extensions-2.8.0-0.8.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
[root@mgmt5 ~]#
[root@mgmt5 ~]# pulp-admin rpm repo create --repo-id pulp --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --download-policy=on_demand
07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) The field 'checksum_type' does not exist on the document 'RPM'
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) Traceback (most recent call last):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 220, in run
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) self.update_content(metadata_files, url)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 447, in update_content
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) rpms_to_download, drpms_to_download = self._decide_what_to_download(metadata_files)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 468, in _decide_what_to_download
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) self._decide_rpms_to_download(metadata_files)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 504, in _decide_rpms_to_download
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) wanted.iterkeys(), self.conduit, self.download_deferred)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 118, in check_all_and_associate
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) for unit in units_controller.find_units(unit_generator):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/units.py", line 40, in find_units
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) for found_unit in query:
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1411, in next
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 730, in _from_son
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 362, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) super(RpmBase, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 135, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) super(NonMetadataPackage, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 52, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) super(AutoRetryDocument, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 80, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) raise FieldDoesNotExist(msg)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) FieldDoesNotExist: The field 'checksum_type' does not exist on the document 'RPM'
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp.server.async.tasks:INFO: Task failed : [d1f517ea-5d52-4d01-9a11-54a6ebc0ca31]
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) Task pulp.server.managers.repo.sync.sync[d1f517ea-5d52-4d01-9a11-54a6ebc0ca31] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) Traceback (most recent call last):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) R = retval = fun(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) return super(Task, self).__call__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) return super(PulpTask, self).__call__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) return self.run(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 807, in sync
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) PulpExecutionException: Importer indicated a failed response
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_re
Files
Related issues
Updated by pthomas@redhat.com over 8 years ago
- File pulp26.tar.gz pulp26.tar.gz added
- Priority changed from Normal to High
2.6 mongodump attached.
Updated by ipanova@redhat.com over 8 years ago
looking at the traceback those bugs could be related or be duplicate https://pulp.plan.io/issues/1749
Updated by ipanova@redhat.com over 8 years ago
And seems like it does not matter if the policy is on_demand or immediate, it just happens in the upgraded environment.
https://github.com/MongoEngine/mongoengine/issues/970
Updated by ipanova@redhat.com over 8 years ago
- Related to Issue #1749: Centos6 64b updates sync failed with The field '_erratum_references' does not exist on the document 'RPM' added
Updated by jcline@redhat.com over 8 years ago
I poked around the database a bit last night:
> db.units_rpm.find({"name": "pulp-test-package"}).pretty()
{
"_id" : "d0f05b70-da4f-4d78-93f1-38796356d4b6",
"vendor" : null,
"description" : "Test package. Nothing to see here.",
"_last_updated" : 1457364164,
"build_time" : NumberLong(1273087108),
"buildhost" : "gibson",
"relativepath" : "pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
"_content_type_id" : "rpm",
"repodata" : {
"filelists" : "\n<package pkgid=\"4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7\" name=\"pulp-test-package\" arch=\"x86_64\">\n <version epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n <file>/etc/pulp-test-file.txt</file>\n</package>",
"other" : "\n<package pkgid=\"4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7\" name=\"pulp-test-package\" arch=\"x86_64\">\n <version epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n <changelog author=\"<mmccune@redhat.com> - 0.1.1\" date=\"1273060800\">- Initial rev</changelog>\n <changelog author=\"Mike McCune <mmccune@redhat.com> 0.2.1-1\" date=\"1273060801\">- new package</changelog>\n</package>",
"primary" : "\n<package type=\"rpm\">\n <name>pulp-test-package</name>\n <arch>x86_64</arch>\n <version epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n <checksum type=\"sha256\" pkgid=\"YES\">4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7</checksum>\n <summary>Test package</summary>\n <description>Test package. Nothing to see here.</description>\n <packager></packager>\n <url>https://fedorahosted.org/pulp/</url>\n <time file=\"1457109770\" build=\"1273087108\"/>\n <size package=\"2216\" installed=\"5\" archive=\"268\"/>\n <location href=\"pulp-test-package-0.2.1-1.fc11.x86_64.rpm\"/>\n <format>\n <rpm:license>MIT</rpm:license>\n <rpm:vendor/>\n <rpm:group>Development/Libraries</rpm:group>\n <rpm:buildhost>gibson</rpm:buildhost>\n <rpm:sourcerpm>pulp-test-package-0.2.1-1.fc11.src.rpm</rpm:sourcerpm>\n <rpm:header-range start=\"280\" end=\"2092\"/>\n <rpm:provides>\n <rpm:entry name=\"config(pulp-test-package)\" flags=\"EQ\" epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n <rpm:entry name=\"pulp-test-package\" flags=\"EQ\" epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n <rpm:entry name=\"pulp-test-package(x86-64)\" flags=\"EQ\" epoch=\"0\" ver=\"0.2.1\" rel=\"1.fc11\"/>\n </rpm:provides>\n <file>/etc/pulp-test-file.txt</file>\n </format>\n</package>"
},
"checksum" : "4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7",
"arch" : "x86_64",
"_ns" : "units_rpm",
"checksumtype" : "sha256",
"_storage_path" : "/var/lib/pulp/content/rpm/pulp-test-package/0.2.1/1.fc11/x86_64/4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7/pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
"name" : "pulp-test-package",
"license" : "MIT",
"release_sort_index" : "01-1.$fc.02-11",
"filename" : "pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
"epoch" : "0",
"version" : "0.2.1",
"version_sort_index" : "01-0.01-2.01-1",
"provides" : [
{
"release" : "1.fc11",
"epoch" : "0",
"version" : "0.2.1",
"flags" : "EQ",
"name" : "config(pulp-test-package)"
},
{
"release" : "1.fc11",
"epoch" : "0",
"version" : "0.2.1",
"flags" : "EQ",
"name" : "pulp-test-package"
},
{
"release" : "1.fc11",
"epoch" : "0",
"version" : "0.2.1",
"flags" : "EQ",
"name" : "pulp-test-package(x86-64)"
}
],
"time" : 1457109770,
"release" : "1.fc11",
"checksum_type" : null,
"requires" : [ ],
"pulp_user_metadata" : {
}
}
Note that there's a field "checksumtype" and a field "checksum_type". MongoEngine will raise the exception in the log if there's a key in a document that's not on the model (http://docs.mongoengine.org/en/latest/apireference.html#mongoengine.FieldDoesNotExist). One option looks to be setting `strict = False` on the model meta dictionary. I don't know if that's something we want to do or not, though.
Updated by jcline@redhat.com over 8 years ago
- Subject changed from Repo sync with on_demand policy fails on an upgraded pulp server to Repo sync with fails on an upgraded pulp server
Updated by dkliban@redhat.com over 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
Updated by pthomas@redhat.com over 8 years ago
It may be worth noting that
The same repo that failed on the upgraded pulp from 2.6 ->2.8 was successful on 2.7->2.8
pulp-admin rpm repo create --repo-id pulp --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest
Added by dkliban@redhat.com over 8 years ago
Updated by dkliban@redhat.com over 8 years ago
- Status changed from ASSIGNED to POST
Updated by dkliban@redhat.com over 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp_rpm:f4633c672ba60ed04cfc48046945883f74f890e3.
Added by dkliban@redhat.com over 8 years ago
Revision 1afa2931 | View on GitHub
Removes checksum_type field from RPM unit documents in Mongo
After migrating from 2.6 to 2.8, users could have database records that contain 'checksum_type' attribute that is set to null. This migration removes this field from the documents.
Updated by dkliban@redhat.com over 8 years ago
- Status changed from MODIFIED to 5
Updated by pthomas@redhat.com over 8 years ago
- Status changed from 5 to 6
verified
[root@celeno ~]# rpm -qa |grep pulp-server
pulp-server-2.8.0-0.9.rc.el6.noarch
[root@celeno ~]#
[root@celeno ~]#pulp-admin rpm repo create --repo-id pulp --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/
[root@celeno ~]#
[root@celeno ~]# pulp-admin rpm repo sync run --repo-id pulp
--------------------------------------------------------------------
Synchronizing Repository [pulp]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
[root@celeno ~]# rpm -qa |grep pulp-server
pulp-server-2.8.0-0.9.rc.el6.noarch
[root@celeno ~]#
Downloading metadata...
[\]
... completed
Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Cleaning duplicate packages...
[-]
... completed
Task Succeeded
Copying files
[-]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
52 of 52 items
... completed
Publishing Comps file
[==================================================] 100%
5 of 5 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
Updated by dkliban@redhat.com over 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
Updated by bmbouter about 8 years ago
- Related to Issue #2278: Remove checksum_type from the srpm and drpm collections added
Actions
.
Removes checksum_type field from RPM unit documents in Mongo
After migrating from 2.6 to 2.8, users could have database records that contain 'checksum_type' attribute that is set to null. This migration removes this field from the documents.
https://pulp.plan.io/issues/1754 closes #1754