Issue #2666
closed
Rsync publish for RPM repo fails to run in fast forward mode
Description
Reproducing steps:
1. Create a RPM repo with default configurations. Upload some rpms/srpms to the repo.
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo38", "description": "test repo", "distributors": [{"distributor_id": "yum_distributor", "distributor_type_id": "yum_distributor", "distributor_config": {"http": true, "https": false, "relative_url": "content/dist/test-repo38"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "rpm_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "yum_distributor"}}], "notes": {"_repo-type": "rpm-repo", "relative_url": "content/dist/test-repo38"}, "importer_type_id": "yum_importer", "id": "test-repo38"}'
$ pulp-admin rpm repo uploads rpm --repo-id test-repo38 --file fox-1.1-2.noarch.rpm
$ pulp-admin rpm repo uploads srpm --repo-id test-repo38 --file test-srpm01-1.0-1.src.rpm
2. Run yum publish and rsync publish.
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo38/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "yum_distributor", "override_config": {}}'
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo38/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'
3. Repeat step 2, and check status of the 2nd rsync publish task.
$ pulp-admin tasks details --task-id c64f9c6f-d150-4c4f-b8e2-67888f8380d3
+----------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------+
Operations: publish
Resources: test-repo38 (repository)
State: Successful
Start Time: 2017-03-27T04:08:19Z
Finish Time: 2017-03-27T04:08:20Z
Result:
Completed: 2017-03-27T04:08:20Z
Details:
Description:
Details:
Error Details:
Items Total: 2
Num Failures: 0
Num Processed: 2
Num Success: 2
No new unit has been added to the repo since last publish, and default publish mode should be fast forward. However, all units in the repo are processed here. The problem might be last_published is incorrectly set to None when last_deleted has no value.
/usr/lib/python2.7/site-packages/pulp/plugins/rsync/publish.py
348 if self.last_deleted:
349 last_deleted = self.last_deleted.replace(tzinfo=None)
350 else:
351 last_published = None
Related issues
Updated by ipanova@redhat.com over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ipanova@redhat.com
Added by ipanova@redhat.com over 7 years ago
Updated by ipanova@redhat.com over 7 years ago
Updated by ipanova@redhat.com over 7 years ago
- Status changed from ASSIGNED to POST
Updated by jortel@redhat.com over 7 years ago
- Sprint/Milestone changed from 37 to 38
Updated by bmbouter over 7 years ago
- Has duplicate Issue #2743: If unit has never been deleted from a repo, rsync distributor incorrectly forces full publish added
Updated by ipanova@redhat.com over 7 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp:pulp|5343c4fa04b88f7cfd2b847320f1d1ec0f044873.
Updated by Ichimonji10 over 7 years ago
- Status changed from 5 to ASSIGNED
I tested this issue by creating a test case which will do the following when executed:
1. Create a yum repository with a num and rsync distributor.
2. Add some content to the repository.
3. Publish the repository with the yum and rsync distributors, respectively.
4. Publish the repository with the yum and rsync distributors, respectively, again.
The second publish with the rsync distributor should be a fast-forward publish, as no new units were added to the repository between the first and second rsync publishes.
If I understand correctly, one can figure out whether or not a fast-forward publish occurred by examining the Num Processed: value printed by the CLI, or the num_processed field included in task reports. Thus, "num processed" should be non-zero for the first publish, and zero for the second publish. Unfortunately, "num published" is non-zero for both publishes.
The test case I created talks directly to the API. Here's a concrete example of the task spawned by the first rsync publish. (Both rsync publishes spawn only a single task.) The important section is the list of ['result']['details']. As you can see, num_processed is non-zero for each step, which is correct.
{'_href': '/pulp/api/v2/tasks/81e36a7b-fd34-4fec-bd44-716414a0969b/',
'_id': {'$oid': '591f3a55cfcc4780d24f5caa'},
'_ns': 'task_status',
'error': None,
'exception': None,
'finish_time': '2017-05-19T18:32:57Z',
'id': '591f3a55cfcc4780d24f5caa',
'progress_report': {'b50995fe-84f0-4eef-b471-6679bfe27582': [{'description': '',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '28e506e8-f8d6-4abf-893f-adb50616d2ac',
'step_type': 'Unit query step (rpm, drpm, srpm)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '3e60a4ce-d946-41be-b2d8-ccaa315f8934',
'step_type': 'Rsync step (origin)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': 'c006aea8-e156-4bcf-abb6-ded9aec3e443',
'step_type': 'Rsync step (content)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '69eb7d60-2797-4017-9b41-7b64eab07be3',
'step_type': 'Rsync step (repodata)'}]},
'queue': 'reserved_resource_worker-0@rhel-7-3-pulp-2-13-beta.dq',
'result': {'_ns': 'repo_publish_results',
'completed': '2017-05-19T18:32:57Z',
'details': [{'description': '',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '28e506e8-f8d6-4abf-893f-adb50616d2ac',
'step_type': 'Unit query step (rpm, drpm, srpm)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '3e60a4ce-d946-41be-b2d8-ccaa315f8934',
'step_type': 'Rsync step (origin)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': 'c006aea8-e156-4bcf-abb6-ded9aec3e443',
'step_type': 'Rsync step (content)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '69eb7d60-2797-4017-9b41-7b64eab07be3',
'step_type': 'Rsync step (repodata)'}],
'distributor_id': 'b50995fe-84f0-4eef-b471-6679bfe27582',
'distributor_type_id': 'rpm_rsync_distributor',
'error_message': None,
'exception': None,
'id': '591f3a5949a6ca531aecc0fa',
'repo_id': '0ec8b400-3a65-443b-83b0-0dc50401d6fe',
'result': 'success',
'started': '2017-05-19T18:32:54Z',
'summary': {'Rsync step (content)': 'FINISHED',
'Rsync step (origin)': 'FINISHED',
'Rsync step (repodata)': 'FINISHED',
'Unit query step (rpm, drpm, srpm)': 'FINISHED'},
'traceback': None},
'spawned_tasks': [],
'start_time': '2017-05-19T18:32:54Z',
'state': 'finished',
'tags': ['pulp:repository:0ec8b400-3a65-443b-83b0-0dc50401d6fe',
'pulp:action:publish'],
'task_id': '81e36a7b-fd34-4fec-bd44-716414a0969b',
'task_type': 'pulp.server.managers.repo.publish.publish',
'traceback': None,
'worker_name': 'reserved_resource_worker-0@rhel-7-3-pulp-2-13-beta'}
Here's the list of ['result']['details'] for the second rsync publish:
'details': [{'description': '',
'details': '',
'error_details': [],
'items_total': 0,
'num_failures': 0,
'num_processed': 0,
'num_success': 0,
'state': 'FINISHED',
'step_id': '19d78ef8-fdc6-4680-a38d-d256f3b8a36b',
'step_type': 'Unit query step (rpm, drpm, srpm)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': 'c8be9373-dc7a-46be-b2de-7a47586ab7d7',
'step_type': 'Rsync step (origin)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': 'd1e5f97e-6d2b-4082-8ac3-d4446e4a1db6',
'step_type': 'Rsync step (content)'},
{'description': 'Rsync files to remote destination',
'details': '',
'error_details': [],
'items_total': 1,
'num_failures': 0,
'num_processed': 1,
'num_success': 1,
'state': 'FINISHED',
'step_id': '94664ae6-3aac-46e1-8b1a-63b25d3dbb54',
'step_type': 'Rsync step (repodata)'}],
The 'Unit query step (rpm, drpm, srpm)' now has a num_processed of 0, but the other three steps all have non-zero values:
- Rsync step (origin)
- Rsync step (content)
- Rsync step (repodata)
As far as I can tell, this indicates that the issue has not been fixed, and fast-forward publishes are not occurring when they should be. If my test procedure is incorrect, let me know, and I'll update my test procedure.
Updated by rmcgover over 7 years ago
I think that the observed behavior is correct. Internally, publish steps can be written to consume items via an iterator, but they don't have to, and this affects the numbers.
The "Unit query step" uses an iterator. The count should be the number of units processed. So the first publish report should mean one RPM was processed by that step, and the second publish report should mean that no RPMs were processed. This seems OK (if "Add some content to the repository" meant one RPM).
The "Rsync step" doesn't use an iterator, and would always appear as handling a total of 1 items regardless of how many files were copied in the sync, or whether rsync was run at all.
Updated by ipanova@redhat.com over 7 years ago
rmcgover is right, the only thing which you need to check is 'Unit query step' which is responsible for the processing of content units, that can be run in fast forward mode.
Updated by Ichimonji10 over 7 years ago
- Status changed from ASSIGNED to 5
Got it. I'll update the test case. See: https://github.com/PulpQE/pulp-smash/pull/648
if "Add some content to the repository" meant one RPM
Correct. I've written the test to upload just one RPM. Uploading just one RPM to a repository speeds up the test case.
Updated by bizhang over 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Updated by Ichimonji10 over 7 years ago
- Copied to Issue #2844: Rsync publish for RPM repo fails to run in fast forward mode added
.
Fix fast forward mode for rsync publish.
closes #2666 https://pulp.plan.io/issues/2666