Project

Profile

Help

Story #2965

closed

As a user, I can use pulp_example plugin to import content

Added by dkliban@redhat.com about 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 26
Quarter:

Description

The Example plugin will provide a setup.py for installing the plugin. The install will create an entry point that will make the plugin discover-able by Pulp.

The Example plugin will provide ExampleContent content type to Pulp. ExampleContent will be exactly the same as FileContent from file plugin.

The Example plugin will also provide an ExampleImporter. The importer will

- perform downloads asyncronously and in parallel
- minimize number of select queries executed to look up existing content and artifacts
- use separate database transactions for each content unit created to avoid having to perform database cleanup when failures occur

The pulp_example repository will also include a README that will provide documentation about the plugin.

Actions #2

Updated by bmbouter about 7 years ago

Will this also include a setup.py?
Will this also include an entrypoint to be installed with?

Also a basic README in the repo would be nice.

Actions #3

Updated by dkliban@redhat.com about 7 years ago

  • Description updated (diff)
Actions #4

Updated by dkliban@redhat.com about 7 years ago

Yes and yes. I updated the description.

Actions #5

Updated by jortel@redhat.com about 7 years ago

When using transactions, it's very important to insert in the same order to prevent deadlock (table locks).

The ChangeSet inserts in this order:

  • content (base)
  • content (concrete)
  • artifact
  • content-artifact
  • deferred-artifact

Suggest this plugin do the same. We can discuss the order itself (a little) but Pulp must require plugins (using transactions, and they all should) to insert in a known order.

Actions #6

Updated by bmbouter about 7 years ago

I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.

Can you describe this table lock in more detail? An example would also be great.

Actions #7

Updated by dkliban@redhat.com about 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions #8

Updated by jortel@redhat.com about 7 years ago

  • Groomed changed from No to Yes
Actions #9

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone set to 43
Actions #10

Updated by mhrivnak about 7 years ago

  • Priority changed from Normal to High
Actions #11

Updated by jortel@redhat.com about 7 years ago

bmbouter wrote:

I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.

Can you describe this table lock in more detail? An example would also be great.

Right, the DB will enforce referential integrity based on constraint created by django. I'm talking about something else. While in a transaction, if an insert is attempted, the RDBMS will acquire a table lock and insert the row but not commit it. That lock will be held for the duration of the transaction. Inserts are performed on 2+ tables in opposite order in different transactions results in deadlock.

Much of the order I mentioned in comment 5 is dictated by DB constraints. However, Content and Artifact can be inserted in any order.

Example:

T = transaction

T-1: insert into Content
  Content table lock held by T-1

T-2: insert into Artifact
  Artifact table lock held by T-2

T-1: insert into Artifact
  Wait on Artifact lock held by T-2

T-2: insert into Content
  Wait on Content lock held by T-1

This is a classic RDBMS problem and the traditional solution is to ensure that operations (like insert) that results in table locks are performed in a consistent order. PostgreSQL may some algorithm to sort this out but I doubt it. I can run a quick test to substantiate this issue if needed.

Actions #12

Updated by jortel@redhat.com about 7 years ago

  • Sprint/Milestone changed from 43 to 44
Actions #13

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone changed from 44 to 45
Actions #14

Updated by dkliban@redhat.com about 7 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #15

Updated by bmbouter almost 7 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

Actions #16

Updated by bmbouter over 6 years ago

  • Sprint set to Sprint 26
Actions #17

Updated by bmbouter over 6 years ago

  • Sprint/Milestone deleted (45)
Actions #18

Updated by daviddavis over 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #19

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #20

Updated by bmbouter almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF