Project

Profile

Help

Issue #2944

Story #2901: As the admin user for a Pulp 2.11.z server running on RHEL 6.9, I am able to backup my database and import into a new Pulp 2.13.z server running on RHEL 7.3+

"incorrect header check" when searching for files restored from backup

Added by kdelee@redhat.com about 3 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.13.3
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

On the RHEL6 server, with a repo created from:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/


pulp-admin rpm repo create \
    --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/ \
    --relative-url rpm-signed \
    --repo-id rpm-signed \
    --serve-http true \
    --serve-https true
pulp-admin rpm repo sync run --repo-id rpm-signed
pulp-admin rpm repo publish run --repo-id rpm-signed

Once the repo is published these links are made:

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm 
lrwxrwxrwx. 1 apache apache 118 Jul 26 17:20 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/14/65deedb21279f6ab176ed87b93a24ce49d49344eeb2beb2eeeb749d3c53d2a/cow-2.2-3.noarch.rpm

You can successfully search for a package:

pulp-admin rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
Arch:         noarch
Buildhost:    smqe-ws15
Checksum:     3e71d4bb14e20d0e140e2d45e8b074d70caa6291edcaf7e9c5e3e334bbb30092
Checksumtype: sha256
Description:  A dummy package of cow
Epoch:        0
Filename:     cow-2.2-3.noarch.rpm
License:      GPLv2
Name:         cow
Provides:     cow = 2.2-3-0
Release:      3
Requires:     
Version:      2.2

The rhel6 machine is backed up by using mongodb's backup utilities and all files in /var/lib/pulp are compressed via tar -cjf (which preserves symlinks)

But when the backup is restored on RHEL7 machine with pulp 2.13.3 cannot search for the unit

Restore process:
0) services are stopped:

- httpd
- pulp_celerybeat
- pulp_resource_manager
- pulp_streamer
- pulp_workers
- qpidd

1) archives are unpacked
2) restorcon is run to relabel files
3) Old database dropped
4) mongodb backup is restored
5) pulp-manage-db run
6) services that were stopped are restarted

Files are there, repos are there, but can't search.

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm
lrwxrwxrwx. 1 apache apache 118 Jul 17 09:11 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/a7/852ce199034863275655b090b4e046aaf00f854198fa20b29ba72a7c53ac5f/cow-2.2-3.noarch.rpm

# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rpm-signed
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               32
pulp-admin -vvv rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
2017-07-27 16:42:47,297 - DEBUG - sending POST request to /pulp/api/v2/repositories/rpm-signed/search/units/
2017-07-27 16:42:47,470 - INFO - POST request to /pulp/api/v2/repositories/rpm-signed/search/units/ with parameters {"criteria": {"type_ids": ["rpm"], "filters": {"unit": {"name": {"$regex": "cow"}}}}}
2017-07-27 16:42:47,470 - INFO - Response status : 500 

2017-07-27 16:42:47,470 - INFO - Response body :
 {
  "exception": [
    "error: Error -3 while decompressing data: incorrect header check\n"
  ], 
  "traceback": [
    "  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n    return func(*args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py\", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit['metadata'])\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py\", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py\", line 40, in serialize\n    unit['repodata'][metadata_type] = gzip.zlib.decompress(metadata)\n"
  ], 
  "_href": "/pulp/api/v2/repositories/rpm-signed/search/units/", 
  "error_message": "Error -3 while decompressing data: incorrect header check", 
  "http_request_method": "POST", 
  "http_status": 500
}

2017-07-27 16:42:47,470 - ERROR - Exception occurred:
        href:      /pulp/api/v2/repositories/rpm-signed/search/units/
        method:    POST
        status:    500
        error:     Error -3 while decompressing data: incorrect header check
        traceback: [u'  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper\n    return func(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit[\'metadata\'])\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n', u'  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py", line 40, in serialize\n    unit[\'repodata\'][metadata_type] = gzip.zlib.decompress(metadata)\n']
        data:      {}

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/rpm-signed/search/units/ failed with 500 - Error -3
while decompressing data: incorrect header check

Error is "incorrect header check"

Is this a problem with how I am restoring the backup, or is it a problem in pulp?

mongo_info_broken_restore.txt (2.45 KB) mongo_info_broken_restore.txt output from mongo shell kdelee@redhat.com, 07/28/2017 07:41 PM
logs_manual_sync.txt (26.6 KB) logs_manual_sync.txt kdelee@redhat.com, 07/28/2017 08:13 PM
logs_manual_sync_search.txt (2.01 KB) logs_manual_sync_search.txt kdelee@redhat.com, 07/28/2017 08:14 PM

History

#1 Updated by kdelee@redhat.com about 3 years ago

  • Version set to 2.13.3
  • Platform Release set to 2.13.3
  • OS set to RHEL 7

#2 Updated by ttereshc about 3 years ago

It seems like you haven't run migrations (sudo -u apache pulp-manage-db) after restoring mongodb. So your DB data is compatible with Pulp 2.11.z but not with 2.13.3. Let me know if it's not the case.

#3 Updated by ttereshc about 3 years ago

  • Platform Release deleted (2.13.3)

#4 Updated by kdelee@redhat.com about 3 years ago

I had run migrations, you can find the ansible tasks run on this machine to restore the backup on my github [0].

The tasks run to create the backup files are also found on that same branch. [1]

[0] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-restore-backup/tasks/main.yaml#L48
[1] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-backup/tasks/main.yaml

#5 Updated by ttereshc about 3 years ago

The issue can't be triaged without more information.
ttereshc will try to help to figure out if the problem is present in Pulp or in the restoring steps.

#6 Updated by kdelee@redhat.com about 3 years ago

Output of db.migration_trackers.find() in mongo shell makes it appear that migrations were successful.

Also, it appears that repodata was gzipped.

See attached for output.

#7 Updated by kdelee@redhat.com about 3 years ago

  • Description updated (diff)

#8 Updated by kdelee@redhat.com about 3 years ago

  • Description updated (diff)

#9 Updated by kdelee@redhat.com about 3 years ago

On a fresh install of pulp-server-2.13.3-1.el7, confirmed that when creating this repo and syncing it manually, it is searchable. (So nothing wrong with this repo or 2.13.3 using it)

See attached logs.

#10 Updated by kdelee@redhat.com about 3 years ago

  • Status changed from NEW to CLOSED - NOTABUG

On RHEL7 server with error, pulp-manage-db was run twice without dropping the db inbetween.

This caused the error.

Thanks ttereshc !!!

Restore should work with following steps:
1) stop httpd, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, qpidd
2) Restore files in /var/lib/pulp /etc/pulp /etc/pki/pulp
3) Run restorecon -r on these directories
4) Drop the pulp_database
5) Restore database from backup
6) run pulp-manage-db
7) Restart stopped services

#11 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF