Issue #2944
Story #2901: As the admin user for a Pulp 2.11.z server running on RHEL 6.9, I am able to backup my database and import into a new Pulp 2.13.z server running on RHEL 7.3+
"incorrect header check" when searching for files restored from backup
Description
On the RHEL6 server, with a repo created from:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/
pulp-admin rpm repo create \
--feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/ \
--relative-url rpm-signed \
--repo-id rpm-signed \
--serve-http true \
--serve-https true
pulp-admin rpm repo sync run --repo-id rpm-signed
pulp-admin rpm repo publish run --repo-id rpm-signed
Once the repo is published these links are made:
# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm
lrwxrwxrwx. 1 apache apache 118 Jul 26 17:20 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/14/65deedb21279f6ab176ed87b93a24ce49d49344eeb2beb2eeeb749d3c53d2a/cow-2.2-3.noarch.rpm
You can successfully search for a package:
pulp-admin rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
Arch: noarch
Buildhost: smqe-ws15
Checksum: 3e71d4bb14e20d0e140e2d45e8b074d70caa6291edcaf7e9c5e3e334bbb30092
Checksumtype: sha256
Description: A dummy package of cow
Epoch: 0
Filename: cow-2.2-3.noarch.rpm
License: GPLv2
Name: cow
Provides: cow = 2.2-3-0
Release: 3
Requires:
Version: 2.2
The rhel6 machine is backed up by using mongodb's backup utilities and all files in /var/lib/pulp are compressed via tar -cjf
(which preserves symlinks)
But when the backup is restored on RHEL7 machine with pulp 2.13.3 cannot search for the unit
Restore process:
0) services are stopped:
- httpd
- pulp_celerybeat
- pulp_resource_manager
- pulp_streamer
- pulp_workers
- qpidd
1) archives are unpacked
2) restorcon is run to relabel files
3) Old database dropped
4) mongodb backup is restored
5) pulp-manage-db run
6) services that were stopped are restarted
Files are there, repos are there, but can't search.
# ls -al /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm
lrwxrwxrwx. 1 apache apache 118 Jul 17 09:11 /var/lib/pulp/published/yum/https/repos/rpm-signed/cow-2.2-3.noarch.rpm -> /var/lib/pulp/content/units/rpm/a7/852ce199034863275655b090b4e046aaf00f854198fa20b29ba72a7c53ac5f/cow-2.2-3.noarch.rpm
# pulp-admin rpm repo list
+----------------------------------------------------------------------+
RPM Repositories
+----------------------------------------------------------------------+
Id: rpm-signed
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 32
pulp-admin -vvv rpm repo content rpm --repo-id rpm-signed --match 'name=cow'
2017-07-27 16:42:47,297 - DEBUG - sending POST request to /pulp/api/v2/repositories/rpm-signed/search/units/
2017-07-27 16:42:47,470 - INFO - POST request to /pulp/api/v2/repositories/rpm-signed/search/units/ with parameters {"criteria": {"type_ids": ["rpm"], "filters": {"unit": {"name": {"$regex": "cow"}}}}}
2017-07-27 16:42:47,470 - INFO - Response status : 500
2017-07-27 16:42:47,470 - INFO - Response body :
{
"exception": [
"error: Error -3 while decompressing data: incorrect header check\n"
],
"traceback": [
" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n return handler(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n value = method(self, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n return func(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py\", line 127, in post\n return self._generate_response(query, options, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 293, in _generate_response\n content.serialize_unit_with_serializer(unit['metadata'])\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py\", line 49, in serialize_unit_with_serializer\n serializer.serialize(content_unit)\n",
" File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py\", line 40, in serialize\n unit['repodata'][metadata_type] = gzip.zlib.decompress(metadata)\n"
],
"_href": "/pulp/api/v2/repositories/rpm-signed/search/units/",
"error_message": "Error -3 while decompressing data: incorrect header check",
"http_request_method": "POST",
"http_status": 500
}
2017-07-27 16:42:47,470 - ERROR - Exception occurred:
href: /pulp/api/v2/repositories/rpm-signed/search/units/
method: POST
status: 500
error: Error -3 while decompressing data: incorrect header check
traceback: [u' File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch\n return handler(request, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth\n value = method(self, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper\n return func(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py", line 127, in post\n return self._generate_response(query, options, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 293, in _generate_response\n content.serialize_unit_with_serializer(unit[\'metadata\'])\n', u' File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/serializers/content.py", line 49, in serialize_unit_with_serializer\n serializer.serialize(content_unit)\n', u' File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/serializers.py", line 40, in serialize\n unit[\'repodata\'][metadata_type] = gzip.zlib.decompress(metadata)\n']
data: {}
An internal error occurred on the Pulp server:
RequestException: POST request
on /pulp/api/v2/repositories/rpm-signed/search/units/ failed with 500 - Error -3
while decompressing data: incorrect header check
Error is "incorrect header check"
Is this a problem with how I am restoring the backup, or is it a problem in pulp?
History
#1
Updated by kdelee@redhat.com over 3 years ago
- Version set to 2.13.3
- Platform Release set to 2.13.3
- OS set to RHEL 7
#2
Updated by ttereshc over 3 years ago
It seems like you haven't run migrations (sudo -u apache pulp-manage-db
) after restoring mongodb. So your DB data is compatible with Pulp 2.11.z but not with 2.13.3. Let me know if it's not the case.
#3
Updated by ttereshc over 3 years ago
- Platform Release deleted (
2.13.3)
#4
Updated by kdelee@redhat.com over 3 years ago
I had run migrations, you can find the ansible tasks run on this machine to restore the backup on my github [0].
The tasks run to create the backup files are also found on that same branch. [1]
[0] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-restore-backup/tasks/main.yaml#L48
[1] https://github.com/kdelee/pulp_packaging/blob/backup_pulp/ci/ansible/roles/pulp-backup/tasks/main.yaml
#5
Updated by ttereshc over 3 years ago
The issue can't be triaged without more information.
ttereshc will try to help to figure out if the problem is present in Pulp or in the restoring steps.
#6
Updated by kdelee@redhat.com over 3 years ago
Output of db.migration_trackers.find()
in mongo shell makes it appear that migrations were successful.
Also, it appears that repodata was gzipped.
See attached for output.
#7
Updated by kdelee@redhat.com over 3 years ago
- Description updated (diff)
#8
Updated by kdelee@redhat.com over 3 years ago
- Description updated (diff)
#9
Updated by kdelee@redhat.com over 3 years ago
- File logs_manual_sync.txt logs_manual_sync.txt added
- File logs_manual_sync_search.txt logs_manual_sync_search.txt added
On a fresh install of pulp-server-2.13.3-1.el7, confirmed that when creating this repo and syncing it manually, it is searchable. (So nothing wrong with this repo or 2.13.3 using it)
See attached logs.
#10
Updated by kdelee@redhat.com over 3 years ago
- Status changed from NEW to CLOSED - NOTABUG
On RHEL7 server with error, pulp-manage-db was run twice without dropping the db inbetween.
This caused the error.
Thanks ttereshc !!!
Restore should work with following steps:
1) stop httpd, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, qpidd
2) Restore files in /var/lib/pulp /etc/pulp /etc/pki/pulp
3) Run restorecon -r on these directories
4) Drop the pulp_database
5) Restore database from backup
6) run pulp-manage-db
7) Restart stopped services
#11
Updated by bmbouter almost 2 years ago
- Tags Pulp 2 added
Please register to edit this issue