Story #2965
closedAs a user, I can use pulp_example plugin to import content
0%
Description
The Example plugin will provide a setup.py for installing the plugin. The install will create an entry point that will make the plugin discover-able by Pulp.
The Example plugin will provide ExampleContent content type to Pulp. ExampleContent will be exactly the same as FileContent from file plugin.
The Example plugin will also provide an ExampleImporter. The importer will
- perform downloads asyncronously and in parallel
- minimize number of select queries executed to look up existing content and artifacts
- use separate database transactions for each content unit created to avoid having to perform database cleanup when failures occur
The pulp_example repository will also include a README that will provide documentation about the plugin.
Updated by mhrivnak about 7 years ago
Here you go! https://github.com/pulp/pulp_example
Updated by bmbouter about 7 years ago
Will this also include a setup.py?
Will this also include an entrypoint to be installed with?
Also a basic README in the repo would be nice.
Updated by dkliban@redhat.com about 7 years ago
Yes and yes. I updated the description.
Updated by jortel@redhat.com about 7 years ago
When using transactions, it's very important to insert in the same order to prevent deadlock (table locks).
The ChangeSet inserts in this order:
- content (base)
- content (concrete)
- artifact
- content-artifact
- deferred-artifact
Suggest this plugin do the same. We can discuss the order itself (a little) but Pulp must require plugins (using transactions, and they all should) to insert in a known order.
Updated by bmbouter about 7 years ago
I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.
Can you describe this table lock in more detail? An example would also be great.
Updated by dkliban@redhat.com about 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
Updated by jortel@redhat.com about 7 years ago
bmbouter wrote:
I think Django will fail to save if the order isn't correct because the remote objects won't have primary keys set yet. If that is true then during development this should be easy to get correct because it work otherwise.
Can you describe this table lock in more detail? An example would also be great.
Right, the DB will enforce referential integrity based on constraint created by django. I'm talking about something else. While in a transaction, if an insert is attempted, the RDBMS will acquire a table lock and insert the row but not commit it. That lock will be held for the duration of the transaction. Inserts are performed on 2+ tables in opposite order in different transactions results in deadlock.
Much of the order I mentioned in comment 5 is dictated by DB constraints. However, Content and Artifact can be inserted in any order.
Example:
T = transaction
T-1: insert into Content
Content table lock held by T-1
T-2: insert into Artifact
Artifact table lock held by T-2
T-1: insert into Artifact
Wait on Artifact lock held by T-2
T-2: insert into Content
Wait on Content lock held by T-1
This is a classic RDBMS problem and the traditional solution is to ensure that operations (like insert) that results in table locks are performed in a consistent order. PostgreSQL may some algorithm to sort this out but I doubt it. I can run a quick test to substantiate this issue if needed.
Updated by jortel@redhat.com about 7 years ago
- Sprint/Milestone changed from 43 to 44
Updated by dkliban@redhat.com about 7 years ago
- Status changed from ASSIGNED to MODIFIED
Updated by bmbouter almost 7 years ago
- Tags deleted (
Pulp 3 Plugin Writer Alpha)
Cleaning up Redmine tags
Updated by bmbouter almost 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE