Project

Profile

Help

Issue #9266

closed

repository 'modify' touches all the content being added one by one

Added by dkliban@redhat.com over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

Bulk touch feature of the BulkQuerySet should be used to reduce number of queries that are executed in the viewset handling the Repository Modify API.

Removing the iteration of the content in the viewset will require the task to determine if any of the content to be added doesn't exist. The task needs to raise a RuntimeError with a list of pulp_href's for the content that doesn't exist.


Related issues

Related to Pulp - Issue #9243: 30-50% re-sync performance regression due to touch() of content and artifacts during syncCLOSED - CURRENTRELEASEdkliban@redhat.comActions
Copied to Pulp - Backport #9401: Backport #9266 "repository 'modify' touches all the content being added one by one" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by dkliban@redhat.com over 3 years ago

  • Related to Issue #9243: 30-50% re-sync performance regression due to touch() of content and artifacts during sync added
Actions #2

Updated by dkliban@redhat.com over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 104
Actions #3

Updated by dalley over 3 years ago

Additionally, this line is problematic for the same reason, one query per content unit. The same issue led to https://pulp.plan.io/issues/7483 for the RPM plugin

https://github.com/pulp/pulpcore/blob/de67d41c39d9d4bde2e8f86e4bf3f51d6cc774e7/pulpcore/plugin/actions.py#L47

There was some previous discussion about this on Matrix covering some of the competing requirements: https://riot.im/app/#/room/#pulp-dev:matrix.org/$8_vHR2IhnUvnjSU5XeimYEKXov-ed9u_Us0YtMzqvPE

Actions #4

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #5

Updated by dalley about 3 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High

Bumping the prio because with a lot of content in the operation, it can easily time out the whole request. We don't have any tests running at that scale but Katello is running into this issue.

Actions #6

Updated by lmjachky about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky
Actions #7

Updated by dalley about 3 years ago

  • Sprint/Milestone set to 3.16.0
Actions #8

Updated by dalley about 3 years ago

  • Copied to Backport #9401: Backport #9266 "repository 'modify' touches all the content being added one by one" to 3.14.z added
Actions #9

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST

Added by Lubos Mjachky about 3 years ago

Revision 4023fce7 | View on GitHub

Do bulk_touch instead of touching one by one

closes #9266

Actions #10

Updated by Anonymous about 3 years ago

  • Status changed from POST to MODIFIED
Actions #12

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF