RPM Support

Copying a modulemd_defaults unit from one repo to another, where a modulemd_defaults unit with the same "name" was formerly uploaded into the target repo, fails with "[Errno 20] Not a directory".

Steps to reproduce¶

In dev env:

0. Prepare two modulemd_defaults YAML with same name:

[vagrant@pulp2 ~]$ cat mod1.yaml 
---
document: modulemd-defaults
version: 1
data:
  module: varnish
  stream: 6
  profiles:
    6: [common]
...

[vagrant@pulp2 ~]$ cat mod2.yaml 
---
document: modulemd-defaults
version: 1
data:
  module: varnish
  stream: 7
  profiles:
    7: [common]
...

And ensure a clean RPM repo exists for target of copy (e.g. zoo2):

pulp-admin  rpm repo create --repo-id zoo2

1. Upload a modulemd_defaults to zoo

$ curl -k -u admin:admin -X POST https://localhost/pulp/api/v2/content/uploads/
{"upload_id": "7a55199f-59aa-49e0-b808-26bee49b0c01", "_href": "/pulp/api/v2/content/uploads/7a55199f-59aa-49e0-b808-26bee49b0c01/"}

$ curl -k -u admin:admin -X PUT --data-binary @mod1.yaml https://localhost/pulp/api/v2/content/uploads/7a55199f-59aa-49e0-b808-26bee49b0c01/0/

$ curl -v -k -u admin:admin -X POST -H 'Content-Type: application/json' -d '{"upload_id":"7a55199f-59aa-49e0-b808-26bee49b0c01", "unit_type_id": "modulemd_defaults", "unit_key": {}}' https://localhost/pulp/api/v2/repositories/zoo/actions/import_upload/

# ...and wait for task to succeed

2. Upload a modulemd_defaults to zoo2

Similar commands as step 1, but use the other YAML file and repo.

$ curl -k -u admin:admin -X POST https://localhost/pulp/api/v2/content/uploads/
$ curl -k -u admin:admin -X PUT --data-binary @mod2.yaml https://localhost/pulp/api/v2/content/uploads/4c445ccc-a96d-4212-97d9-d60bddd642ce/0/
$ curl -v -k -u admin:admin -X POST -H 'Content-Type: application/json' -d '{"upload_id":"4c445ccc-a96d-4212-97d9-d60bddd642ce", "unit_type_id":"modulemd_defaults", "unit_key": {}}' https://localhost/pulp/api/v2/repositories/zoo2/actions/import_upload/
# ...and wait for task to succeed

After this step, zoo2 contains this association:

    {
        "_id": {
            "$oid": "5dbf63be741ab32483de58b5"
        },
        "created": "2019-11-03T23:33:18Z",
        "metadata": {
            "_content_type_id": "modulemd_defaults",
            "_id": "ff24ba39-925b-42fd-af6e-19cfdb74eac4",
            "_last_updated": 1572823998,
            "_ns": "units_modulemd_defaults",
            "_storage_path": "/var/lib/pulp/content/units/modulemd_defaults/75/593dd2cf41e10676befac02877d4f4da4d854cbc648d4dfb1a61ca19986dff",
            "checksum": "152166f328559f2c1b4bcd403e47801a1092bc86d93560cff723653b9cb57c16",
            "downloaded": true,
            "name": "varnish",
            "profiles": {
                "7": [
                    "common"
                ]
            },
            "pulp_user_metadata": {},
            "repo_id": "zoo2",
            "stream": "7"
        },
        "repo_id": "zoo2",
        "unit_id": "ff24ba39-925b-42fd-af6e-19cfdb74eac4",
        "unit_type_id": "modulemd_defaults",
        "updated": "2019-11-03T23:33:18Z"
    }

3. Copy from zoo to zoo2

$ pulp-admin  rpm repo copy all --from-repo-id zoo --to-repo-id zoo2
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Failed

[Errno 20] Not a directory:
'/var/lib/pulp/content/units/modulemd_defaults/75/593dd2cf41e10676befac02877d4f4
da4d854cbc648d4dfb1a61ca19986dff/tmpxUWq1E'

Actual results¶

1. Copy task fails:

pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416) Exception from importer [yum_importer] while importing units into repository [zoo2]
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416) Traceback (most recent call last):
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     units=transfer_units)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/importer.py", line 60, in import_units
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     return associate.associate(source_repo, dest_repo, import_conduit, config, units)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/associate.py", line 72, in associate
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     for unit in units if not isinstance(unit, models.RPM)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/associate.py", line 471, in _associate_unit
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     return associate_copy_for_repo(unit, dest_repo, True)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/associate.py", line 526, in associate_copy_for_repo
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     new_unit.safe_import_content(unit._storage_path)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/db/model/__init__.py", line 941, in safe_import_content
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     self.import_content(path, location)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/db/model/__init__.py", line 915, in import_content
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     storage.put(self, path, location)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/content/storage.py", line 128, in put
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     fd, temp_destination = tempfile.mkstemp(dir=os.path.dirname(destination))
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/usr/lib64/python2.7/tempfile.py", line 314, in mkstemp
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     return _mkstemp_inner(dir, prefix, suffix, flags)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)   File "/usr/lib64/python2.7/tempfile.py", line 244, in _mkstemp_inner
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416)     fd = _os.open(file, flags, 0600)
pulp.server.managers.repo.unit_association:ERROR: [b368ddc9] (20873-28416) OSError: [Errno 20] Not a directory: '/var/lib/pulp/content/units/modulemd_defaults/75/593dd2cf41e10676befac02877d4f4da4d854cbc648d4dfb1a61ca19986dff/tmpxUWq1E'
pulp.server.async.tasks:INFO: [b368ddc9] Task failed : [b368ddc9-4701-45e4-8eb2-32b5fa9c6fa2]
celery.app.trace:ERROR: [b368ddc9] (20873-28416) Task pulp.server.managers.repo.unit_association.associate_from_repo[b368ddc9-4701-45e4-8eb2-32b5fa9c6fa2] raised unexpected: OSError(20, 'Not a directory')
celery.app.trace:ERROR: [b368ddc9] (20873-28416) Traceback (most recent call last):
celery.app.trace:ERROR: [b368ddc9] (20873-28416)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [b368ddc9] (20873-28416)     R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 686, in __call__
celery.app.trace:ERROR: [b368ddc9] (20873-28416)     return super(Task, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 108, in __call__
celery.app.trace:ERROR: [b368ddc9] (20873-28416)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.app.trace:ERROR: [b368ddc9] (20873-28416)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
celery.app.trace:ERROR: [b368ddc9] (20873-28416)     return self.run(*args, **kwargs)
celery.app.trace:ERROR: [b368ddc9] (20873-28416)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo
celery.app.trace:ERROR: [b368ddc9] (20873-28416)     raise (e, None, sys.exc_info()[2])
celery.app.trace:ERROR: [b368ddc9] (20873-28416) OSError: [Errno 20] Not a directory: '/var/lib/pulp/content/units/modulemd_defaults/75/593dd2cf41e10676befac02877d4f4da4d854cbc648d4dfb1a61ca19986dff/tmpxUWq1E'

2. Target repo is left with no modulemd_defaults unit for the relevant "name" (i.e. the unit which was formerly present, is removed).

3. Filesystem is left with a file under content/units/modulemd_defaults which is not referenced by any existing unit (and so can't be cleaned up using Pulp's API).

Expected results¶

Copy task succeeds.

Additional info¶

The issue seems to be caused by the fact that two different storage path structures are used for modulemd_defaults units, depending on whether a unit was uploaded or associated.

Where "digest" is derived from the unit key (repo_id, name), then:

- if unit was uploaded, it's stored at content/units/<unit_type>/<digest[:2]>/<digest[2:]>
- if unit was associated, it's stored at content/units/<unit_type>/<digest[:2]>/<digest[2:]>/<basename-from-src>

So, in the upload case, the leading portion of the path "content/units/<unit_type>/<digest[0:2]>/<digest[2:]>" is used as a file, while in the associate case it's used as a directory (with no logic for detecting if the same path was formerly used as a file).

Tested with pulp 706d6c14b8adb9db034866, pulp_rpm cc35af04da2b97.