Project

Profile

Help

Story #2965

As a user, I can use pulp_example plugin to import content

Added by dkliban@redhat.com over 2 years ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 26

Description

The Example plugin will provide a setup.py for installing the plugin. The install will create an entry point that will make the plugin discover-able by Pulp.

The Example plugin will provide ExampleContent content type to Pulp. ExampleContent will be exactly the same as FileContent from file plugin.

The Example plugin will also provide an ExampleImporter. The importer will

- perform downloads asyncronously and in parallel
- minimize number of select queries executed to look up existing content and artifacts
- use separate database transactions for each content unit created to avoid having to perform database cleanup when failures occur

The pulp_example repository will also include a README that will provide documentation about the plugin.

History

#2 Updated by bmbouter over 2 years ago

Will this also include a setup.py?
Will this also include an entrypoint to be installed with?

Also a basic README in the repo would be nice.

#3 Updated by dkliban@redhat.com over 2 years ago

  • Description updated (diff)

#4 Updated by dkliban@redhat.com over 2 years ago

Yes and yes. I updated the description.

#5 Updated by jortel@redhat.com over 2 years ago

When using transactions, it's very important to insert in the same order to prevent deadlock (table locks).

The ChangeSet inserts in this order:
  • content (base)
  • content (concrete)
  • artifact
  • content-artifact
  • deferred-artifact

Suggest this plugin do the same. We can discuss the order itself (a little) but Pulp must require plugins (using transactions, and they all should) to insert in a known order.

#6 Updated by bmbouter over 2 years ago

I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.

Can you describe this table lock in more detail? An example would also be great.

#7 Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#8 Updated by jortel@redhat.com over 2 years ago

  • Groomed changed from No to Yes

#9 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone set to 43

#10 Updated by mhrivnak over 2 years ago

  • Priority changed from Normal to High

#11 Updated by jortel@redhat.com over 2 years ago

bmbouter wrote:

I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.

Can you describe this table lock in more detail? An example would also be great.

Right, the DB will enforce referential integrity based on constraint created by django. I'm talking about something else. While in a transaction, if an insert is attempted, the RDBMS will acquire a table lock and insert the row but not commit it. That lock will be held for the duration of the transaction. Inserts are performed on 2+ tables in opposite order in different transactions results in deadlock.

Much of the order I mentioned in comment 5 is dictated by DB constraints. However, Content and Artifact can be inserted in any order.

Example:

T = transaction

T-1: insert into Content
  Content table lock held by T-1

T-2: insert into Artifact
  Artifact table lock held by T-2

T-1: insert into Artifact
  Wait on Artifact lock held by T-2

T-2: insert into Content
  Wait on Content lock held by T-1

This is a classic RDBMS problem and the traditional solution is to ensure that operations (like insert) that results in table locks are performed in a consistent order. PostgreSQL may some algorithm to sort this out but I doubt it. I can run a quick test to substantiate this issue if needed.

#12 Updated by jortel@redhat.com over 2 years ago

  • Sprint/Milestone changed from 43 to 44

#13 Updated by mhrivnak about 2 years ago

  • Sprint/Milestone changed from 44 to 45

#14 Updated by dkliban@redhat.com about 2 years ago

  • Status changed from ASSIGNED to MODIFIED

#15 Updated by bmbouter almost 2 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

#16 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 26

#17 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (45)

#18 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#19 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF