Issue #9266: repository 'modify' touches all the content being added one by one - Pulp

Agile board
Agile charts

Agile boards

Pulp 2 QE Agile Board
Pulp 2 QE Agile Board - Priority View
Pulp 3 QE Agile Board
Pulp 3 RPM Tests Agile Board
Sprint 112 Agile Board

Custom queries

[All Projects] All Test Trackers
2.21.4 - Next Bugfix Release
2.21.4 MODIFIED
2.21.5
3.14.3
CI/CD
Easy Fix
FIPS open
Functional Tests
Galaxy NG
Groomed
Installer - Triage
Installer Items
Issues for Triage
Katello Integration
Next Docs Day
Operator issues
POST/ASSIGNED - no Sprint - Pulp2
POST/ASSIGNED - no Sprint - Pulp3
Prioritized Bugs
Pulp 3 Docs issues
Pulp2 issues available for next release
Pulpcore
SELinux related
Sprint 111
Sprint 112
Sprint Candidates (also groomed)
Un-Triaged Bugs
Verification Required

Actions

Send by e-mail Copy link

Issue #9266

closed

repository 'modify' touches all the content being added one by one

Added by dkliban@redhat.com over 3 years ago. Updated about 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

lmjachky

Category:

Sprint/Milestone:

3.16.0

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Sprint 105

Quarter:

Description

Bulk touch feature of the BulkQuerySet should be used to reduce number of queries that are executed in the viewset handling the Repository Modify API.

Removing the iteration of the content in the viewset will require the task to determine if any of the content to be added doesn't exist. The task needs to raise a RuntimeError with a list of pulp_href's for the content that doesn't exist.

Related issues

Related to Pulp - Issue #9243: 30-50% re-sync performance regression due to touch() of content and artifacts during sync

CLOSED - CURRENTRELEASE

dkliban@redhat.com

Actions

Copied to Pulp - Backport #9401: Backport #9266 "repository 'modify' touches all the content being added one by one" to 3.14.z

CLOSED - CURRENTRELEASE

dalley

Actions

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by dkliban@redhat.com over 3 years ago

Related to Issue #9243: 30-50% re-sync performance regression due to touch() of content and artifacts during sync added

Actions

Copy link

Updated by dkliban@redhat.com about 3 years ago

Triaged changed from No to Yes
Sprint set to Sprint 104

Actions

Copy link

Updated by dalley about 3 years ago

Additionally, this line is problematic for the same reason, one query per content unit. The same issue led to https://pulp.plan.io/issues/7483 for the RPM plugin

https://github.com/pulp/pulpcore/blob/de67d41c39d9d4bde2e8f86e4bf3f51d6cc774e7/pulpcore/plugin/actions.py#L47

There was some previous discussion about this on Matrix covering some of the competing requirements: https://riot.im/app/#/room/#pulp-dev:matrix.org/$8_vHR2IhnUvnjSU5XeimYEKXov-ed9u_Us0YtMzqvPE

Actions

Copy link

Updated by rchan about 3 years ago

Sprint changed from Sprint 104 to Sprint 105

Actions

Copy link

Updated by dalley about 3 years ago

Priority changed from Normal to High
Severity changed from 2. Medium to 3. High

Bumping the prio because with a lot of content in the operation, it can easily time out the whole request. We don't have any tests running at that scale but Katello is running into this issue.

Actions

Copy link