Pulp

Something else that's odd is that, to the best of my knowledge, we didn't see then when upgrading from 1.1.z to 1.2.z in 2.12.0 or 2.12.1. Because of this, and the fact that it appears to be fixed again in Fedora 25, leads me to thinking that this is a maybe a dnf bug that got fixed, but for whatever reason isn't fixed in Fedora 24. There might be things we can do to fix it, but it also might just fix itself.

I was wrong, we don't yet know if fedora 25 is affected or not. If this is affecting fedora 25 as well as fedora 24, I think that would have a significant impact on the severity of this. Some more investigation is needed.

Looking in Jenkins for upgrade jobs 2.10 -> 2.12.2 beta and 2.11 ->2.12.2 beta

2.10 -> 2.12.2 beta failed on F24

fatal: [localhost]: FAILED! => {"changed": true, "cmd": "dnf -y update", "delta": "0:00:30.450133", "end": "2017-03-16 23:42:58.461305", "failed": true, "rc": 1, "start": "2017-03-16 23:42:28.011172", "stderr": "Error: Transaction check error:\n  file /usr/lib/python2.7/site-packages/pulp_ostree/__init__.pyc from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/__init__.pyc from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/extensions/__init__.pyc from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/__init__.pyo from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/__init__.pyo from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/extensions/__init__.pyo from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/constants.py from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/constants.pyc from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/constants.pyo from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/errors.py from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/errors.pyc from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n  file /usr/lib/python2.7/site-packages/pulp_ostree/common/errors.pyo from install of python-pulp-ostree-common-1.2.1-0.1.beta.fc24.noarch conflicts with file from package python2-pulp-ostree-common-1.1.2-1.fc24.noarch\n\nError Summary\n-------------", "stdout": "Last metadata expiration check: 0:00:06 ago on Thu Mar 16 23:42:47 2017.\nDependencies 

2.11 -> 2.12.2 beta on F25

TASK [pulp-upgrade : Update packages] ******************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "dnf -y update", "delta": "0:00:30.402367", "end": "2017-03-20 08:35:51.346206", "failed": true, "rc": 1, "start": "2017-03-20 08:35:20.943839", "stderr": "/usr/share/locale/de/LC_MESSAGES/cryptsetup.mo: No such file or directory\ncannot reconstruct rpm from disk files\nError: Transaction check error:\n  file /usr/lib/python2.7/site-packages/pulp_docker/__init__.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/__init__.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/extensions/__init__.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/__init__.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/__init__.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/extensions/__init__.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/constants.py from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/constants.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/constants.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/error_codes.py from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/error_codes.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/error_codes.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/tags.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/tags.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/tarutils.pyc from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n  file /usr/lib/python2.7/site-packages/pulp_docker/common/tarutils.pyo from install of python-pulp-docker-common-2.3.0-1.fc25.noarch conflicts with file from package python2-pulp-docker-common-2.1.0-2.fc25.noarch\n\nError Summary\n-------------", "stdout": "Last metadata expiration check: 0:00:17 ago on Mon Mar 20 08:35:29 2017.\nDependencies 

And those upgrades are not from the versions available in fedora, correct?

Talking with elyezer, we might also be seeing these failures with the python plugin. I suspect that at least python, ostree, and docker are affected, but dnf is maybe giving up after the first package is found with apparently broken dependencies, so we don't see all affected packages. This is my guess at why we aren't seeing the same failure on different platforms: for whatever reason dnf processes different packages first on different platforms when going through the upgrade transaction.

This issue seems similar to one we had on pulp platform that was resolved (for platform) in 2.10. The fix at the time was to have upstream 'provide' python2-* packages in the spec file.

https://github.com/pulp/pulp/pull/2791

This should be evaluated to see if doing this across the board for all plugins will resolve these issues.

This was affecting more than ostree. The following PRs address this for all plugins

https://github.com/pulp/pulp_docker/pull/192
https://github.com/pulp/pulp_ostree/pull/85
https://github.com/pulp/pulp_puppet/pull/257
https://github.com/pulp/pulp_python/pull/104
https://github.com/pulp/pulp_rpm/pull/1045