Project

Profile

Help

Issue #2944

closed

Story #2901: As the admin user for a Pulp 2.11.z server running on RHEL 6.9, I am able to backup my database and import into a new Pulp 2.13.z server running on RHEL 7.3+

"incorrect header check" when searching for files restored from backup

Added by kdelee@redhat.com over 5 years ago. Updated over 3 years ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.13.3
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

On the RHEL6 server, with a repo created from:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/


pulp-admin rpm repo create \
    --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/ \
    --relative-url rpm-signed \
    --repo-id rpm-signed \
    --serve-http true \
    --serve-https true
pulp-admin rpm repo sync run --repo-id rpm-signed
pulp-admin rpm repo publish run --repo-id rpm-signed

Once the repo is published these links are made:

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm 
lrwxrwxrwx. 1 apache apache 118 Jul 26 17:20 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/14/65deedb21279f6ab176ed87b93a24ce49d49344eeb2beb2eeeb749d3c53d2a/cow-2.2-3.noarch.rpm

You can successfully search for a package:

pulp-admin rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
Arch:         noarch
Buildhost:    smqe-ws15
Checksum:     3e71d4bb14e20d0e140e2d45e8b074d70caa6291edcaf7e9c5e3e334bbb30092
Checksumtype: sha256
Description:  A dummy package of cow
Epoch:        0
Filename:     cow-2.2-3.noarch.rpm
License:      GPLv2
Name:         cow
Provides:     cow = 2.2-3-0
Release:      3
Requires:     
Version:      2.2

The rhel6 machine is backed up by using mongodb's backup utilities and all files in /var/lib/pulp are compressed via tar -cjf (which preserves symlinks)

But when the backup is restored on RHEL7 machine with pulp 2.13.3 cannot search for the unit

Restore process:
0) services are stopped:

- httpd
- pulp_celerybeat
- pulp_resource_manager
- pulp_streamer
- pulp_workers
- qpidd

1) archives are unpacked
2) restorcon is run to relabel files
3) Old database dropped
4) mongodb backup is restored
5) pulp-manage-db run
6) services that were stopped are restarted

Files are there, repos are there, but can't search.

# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm
lrwxrwxrwx. 1 apache apache 118 Jul 17 09:11 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/a7/852ce199034863275655b090b4e046aaf00f854198fa20b29ba72a7c53ac5f/cow-2.2-3.noarch.rpm

# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rpm-signed
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               32
pulp-admin -vvv rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
2017-07-27 16:42:47,297 - DEBUG - sending POST request to /pulp/api/v2/repositories/rpm-signed/search/units/
2017-07-27 16:42:47,470 - INFO - POST request to /pulp/api/v2/repositories/rpm-signed/search/units/ with parameters {"criteria": {"type_ids": ["rpm"], "filters": {"unit": {"name": {"$regex": "cow"}}}}}
2017-07-27 16:42:47,470 - INFO - Response status : 500 

2017-07-27 16:42:47,470 - INFO - Response body :
 {
  "exception": [
    "error: Error -3 while decompressing data: incorrect header check\n"
  ], 
  "traceback": [
    "  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n    return func(*args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py\", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit['metadata'])\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py\", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n", 
    "  File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py\", line 40, in serialize\n    unit['repodata'][metadata_type] = gzip.zlib.decompress(metadata)\n"
  ], 
  "_href": "/pulp/api/v2/repositories/rpm-signed/search/units/", 
  "error_message": "Error -3 while decompressing data: incorrect header check", 
  "http_request_method": "POST", 
  "http_status": 500
}

2017-07-27 16:42:47,470 - ERROR - Exception occurred:
        href:      /pulp/api/v2/repositories/rpm-signed/search/units/
        method:    POST
        status:    500
        error:     Error -3 while decompressing data: incorrect header check
        traceback: [u'  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper\n    return func(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py", line 127, in post\n    return self._generate_response(query, options, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 293, in _generate_response\n    content.serialize_unit_with_serializer(unit[\'metadata\'])\n', u'  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py", line 49, in serialize_unit_with_serializer\n    serializer.serialize(content_unit)\n', u'  File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py", line 40, in serialize\n    unit[\'repodata\'][metadata_type] = gzip.zlib.decompress(metadata)\n']
        data:      {}

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/rpm-signed/search/units/ failed with 500 - Error -3
while decompressing data: incorrect header check

Error is "incorrect header check"

Is this a problem with how I am restoring the backup, or is it a problem in pulp?


Files

mongo_info_broken_restore.txt (2.45 KB) mongo_info_broken_restore.txt output from mongo shell kdelee@redhat.com, 07/28/2017 07:41 PM
logs_manual_sync.txt (26.6 KB) logs_manual_sync.txt kdelee@redhat.com, 07/28/2017 08:13 PM
logs_manual_sync_search.txt (2.01 KB) logs_manual_sync_search.txt kdelee@redhat.com, 07/28/2017 08:14 PM
Actions #1

Updated by kdelee@redhat.com over 5 years ago

  • Version set to 2.13.3
  • Platform Release set to 2.13.3
  • OS set to RHEL 7
Actions #2

Updated by ttereshc over 5 years ago

It seems like you haven't run migrations (sudo -u apache pulp-manage-db) after restoring mongodb. So your DB data is compatible with Pulp 2.11.z but not with 2.13.3. Let me know if it's not the case.

Actions #3

Updated by ttereshc over 5 years ago

  • Platform Release deleted (2.13.3)
Actions #4

Updated by kdelee@redhat.com over 5 years ago

I had run migrations, you can find the ansible tasks run on this machine to restore the backup on my github [0].

The tasks run to create the backup files are also found on that same branch. [1]

[0] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-restore-backup/tasks/main.yaml#L48
[1] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-backup/tasks/main.yaml

Actions #5

Updated by ttereshc over 5 years ago

The issue can't be triaged without more information.
ttereshc will try to help to figure out if the problem is present in Pulp or in the restoring steps.

Actions #6

Updated by kdelee@redhat.com over 5 years ago

Output of db.migration_trackers.find() in mongo shell makes it appear that migrations were successful.

Also, it appears that repodata was gzipped.

See attached for output.

Actions #7

Updated by kdelee@redhat.com over 5 years ago

  • Description updated (diff)
Actions #8

Updated by kdelee@redhat.com over 5 years ago

  • Description updated (diff)
Actions #9

Updated by kdelee@redhat.com over 5 years ago

On a fresh install of pulp-server-2.13.3-1.el7, confirmed that when creating this repo and syncing it manually, it is searchable. (So nothing wrong with this repo or 2.13.3 using it)

See attached logs.

Actions #10

Updated by kdelee@redhat.com over 5 years ago

  • Status changed from NEW to CLOSED - NOTABUG

On RHEL7 server with error, pulp-manage-db was run twice without dropping the db inbetween.

This caused the error.

Thanks ttereshc !!!

Restore should work with following steps:
1) stop httpd, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, qpidd
2) Restore files in /var/lib/pulp /etc/pulp /etc/pki/pulp
3) Run restorecon -r on these directories
4) Drop the pulp_database
5) Restore database from backup
6) run pulp-manage-db
7) Restart stopped services

Actions #11

Updated by bmbouter over 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF