Pulp

Syncing a repo with on_demand download policy on an upgraded pulp-server fails

1. Install and configure a prior pulp release. (In my case 2.6)
2. I had some repos created and synced
3. Stop the services and update to 2.8
4. Run migration
5. Restart services
6. Install pulp streamer & squid
7. Create a repo with on_demand download policy
8. Run repo sync on that repo

Pulp 2.6 setup http://pulp-qe.etherpad.corp.redhat.com/102

After upgrade

[root@mgmt5 ~]# rpm -qa |grep pulp
pulp-server-2.8.0-0.8.beta.el7.noarch
pulp-puppet-consumer-extensions-2.8.0-0.8.beta.el7.noarch
python-pulp-streamer-2.8.0-0.8.beta.el7.noarch
pulp-admin-client-2.8.0-0.8.beta.el7.noarch
python-pulp-oid_validation-2.8.0-0.8.beta.el7.noarch
pulp-rpm-plugins-2.8.0-0.8.beta.el7.noarch
pulp-puppet-handlers-2.8.0-0.8.beta.el7.noarch
python-pulp-common-2.8.0-0.8.beta.el7.noarch
python-pulp-agent-lib-2.8.0-0.8.beta.el7.noarch
pulp-consumer-client-2.8.0-0.8.beta.el7.noarch
pulp-nodes-admin-extensions-2.8.0-0.8.beta.el7.noarch
pulp-puppet-admin-extensions-2.8.0-0.8.beta.el7.noarch
python-pulp-bindings-2.8.0-0.8.beta.el7.noarch
python-pulp-puppet-common-2.8.0-0.8.beta.el7.noarch
python-pulp-docker-common-2.0.0-0.8.beta.el7.noarch
python-pulp-repoauth-2.8.0-0.8.beta.el7.noarch
python-kombu-3.0.33-4.pulp.el7.noarch
pulp-docker-plugins-2.0.0-0.8.beta.el7.noarch
pulp-selinux-2.8.0-0.8.beta.el7.noarch
pulp-docker-admin-extensions-2.0.0-0.8.beta.el7.noarch
pulp-rpm-admin-extensions-2.8.0-0.8.beta.el7.noarch
pulp-rpm-yumplugins-2.8.0-0.8.beta.el7.noarch
python-pulp-client-lib-2.8.0-0.8.beta.el7.noarch
pulp-puppet-plugins-2.8.0-0.8.beta.el7.noarch
pulp-rpm-handlers-2.8.0-0.8.beta.el7.noarch
python-pulp-rpm-common-2.8.0-0.8.beta.el7.noarch
pulp-nodes-common-2.8.0-0.8.beta.el7.noarch
pulp-nodes-parent-2.8.0-0.8.beta.el7.noarch
pulp-agent-2.8.0-0.8.beta.el7.noarch
pulp-rpm-consumer-extensions-2.8.0-0.8.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
[root@mgmt5 ~]# 

[root@mgmt5 ~]# pulp-admin rpm repo create --repo-id pulp --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --download-policy=on_demand

07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) The field 'checksum_type' does not exist on the document 'RPM'
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) Traceback (most recent call last):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 220, in run
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     self.update_content(metadata_files, url)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 447, in update_content
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     rpms_to_download, drpms_to_download = self._decide_what_to_download(metadata_files)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 468, in _decide_what_to_download
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     self._decide_rpms_to_download(metadata_files)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 504, in _decide_rpms_to_download
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     wanted.iterkeys(), self.conduit, self.download_deferred)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 118, in check_all_and_associate
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     for unit in units_controller.find_units(unit_generator):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/units.py", line 40, in find_units
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     for found_unit in query:
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1411, in next
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 730, in _from_son
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 362, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     super(RpmBase, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 135, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     super(NonMetadataPackage, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 52, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     super(AutoRetryDocument, self).__init__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)   File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 80, in __init__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296)     raise FieldDoesNotExist(msg)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp_rpm.plugins.importers.yum.sync:ERROR: (9248-95296) FieldDoesNotExist: The field 'checksum_type' does not exist on the document 'RPM'
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9248]: pulp.server.async.tasks:INFO: Task failed : [d1f517ea-5d52-4d01-9a11-54a6ebc0ca31]
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) Task pulp.server.managers.repo.sync.sync[d1f517ea-5d52-4d01-9a11-54a6ebc0ca31] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) Traceback (most recent call last):
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)     R = retval = fun(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)     return super(Task, self).__call__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)     return self.run(*args, **kwargs)
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 807, in sync
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:ERROR: (9041-95296) PulpExecutionException: Importer indicated a failed response
Mar 07 15:52:35 mgmt5.rhq.lab.eng.bos.redhat.com pulp[9041]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_re