Pulp

On the RHEL6 server, with a repo created from:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/

pulp-admin rpm repo create \
    --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/ \
    --relative-url rpm-signed \
    --repo-id rpm-signed \
    --serve-http true \
    --serve-https true
pulp-admin rpm repo sync run --repo-id rpm-signed
pulp-admin rpm repo publish run --repo-id rpm-signed

Once the repo is published these links are made:

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm 
lrwxrwxrwx. 1 apache apache 118 Jul 26 17:20 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/14/65deedb21279f6ab176ed87b93a24ce49d49344eeb2beb2eeeb749d3c53d2a/cow-2.2-3.noarch.rpm

You can successfully search for a package:

pulp-admin rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
Arch:         noarch
Buildhost:    smqe-ws15
Checksum:     3e71d4bb14e20d0e140e2d45e8b074d70caa6291edcaf7e9c5e3e334bbb30092
Checksumtype: sha256
Description:  A dummy package of cow
Epoch:        0
Filename:     cow-2.2-3.noarch.rpm
License:      GPLv2
Name:         cow
Provides:     cow = 2.2-3-0
Release:      3
Requires:     
Version:      2.2

The rhel6 machine is backed up by using mongodb's backup utilities and all files in /var/lib/pulp are compressed via tar -cjf (which preserves symlinks)

But when the backup is restored on RHEL7 machine with pulp 2.13.3 cannot search for the unit

Restore process:
0) services are stopped:

- httpd
- pulp_celerybeat
- pulp_resource_manager
- pulp_streamer
- pulp_workers
- qpidd

1) archives are unpacked
2) restorcon is run to relabel files
3) Old database dropped
4) mongodb backup is restored
5) pulp-manage-db run
6) services that were stopped are restarted

Files are there, repos are there, but can't search.

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm
lrwxrwxrwx. 1 apache apache 118 Jul 17 09:11 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/a7/852ce199034863275655b090b4e046aaf00f854198fa20b29ba72a7c53ac5f/cow-2.2-3.noarch.rpm

# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rpm-signed
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               32

pulp-admin -vvv rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
2017-07-27 16:42:47,297 - DEBUG - sending POST request to /pulp/api/v2/repositories/rpm-signed/search/units/
2017-07-27 16:42:47,470 - INFO - POST request to /pulp/api/v2/repositories/rpm-signed/search/units/ with parameters {"criteria": {"type_ids": ["rpm"], "filters": {"unit": {"name": {"$regex": "cow"}}}}}
2017-07-27 16:42:47,470 - INFO - Response status : 500 

2017-07-27 16:42:47,470 - INFO - Response body :
 {
  "exception": [
    "error: Error -3 while decompressing data: incorrect header check\n"
  ], 
  "traceback": [
    "  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n    return func(*args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py\", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit['metadata'])\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py\", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py\", line 40, in serialize\n    unit['repodata'][metadata_type] = gzip.zlib.decompress(metadata)\n"
  ], 
  "_href": "/pulp/api/v2/repositories/rpm-signed/search/units/", 
  "error_message": "Error -3 while decompressing data: incorrect header check", 
  "http_request_method": "POST", 
  "http_status": 500
}

2017-07-27 16:42:47,470 - ERROR - Exception occurred:
        href:      /pulp/api/v2/repositories/rpm-signed/search/units/
        method:    POST
        status:    500
        error:     Error -3 while decompressing data: incorrect header check
        traceback: [u'  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper\n    return func(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit[\'metadata\'])\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n', u'  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py", line 40, in serialize\n    unit[\'repodata\'][metadata_type] = gzip.zlib.decompress(metadata)\n']
        data:      {}

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/rpm-signed/search/units/ failed with 500 - Error -3
while decompressing data: incorrect header check

Error is "incorrect header check"

Is this a problem with how I am restoring the backup, or is it a problem in pulp?