Story #4294
Updated by bmbouter over 5 years ago
h2. The Problem Consider Docker which uses the ArtifactDownloader + ArtifactSaver to download a metadata file and save it. That metadata file identifies N more downloads, and those downloads identify more things, and so one forming an arbitrarily large tree data structure. For terminology purposes, call each level of these a "layer" of discovery. Currently the only way to have the ArtifactDownloader + ArtifactSaver stages download objects in subsequent "layers" is to make the pipeline longer and have more instances of the ArtifactDownloader + ArtifactSaver. This solution won't work in cases where the number of layers is not known and can't be known before runtime. This is the case with Maven for example. h3. The Solution 1. Extend <code>DeclarativeContent</code> to allow for Add a <code>asyncio.Future</code> "loop" feature to be attached. This future can be known about in FirstStage when it creates the the DeclarativeContent. Having the reference to the Future ArtifactDownloader stage that feeds from FirstStage allows the plugin writer to await on it for a result. 2. Create a new stage in second Queue called <code>loop_in_q</code>. This is distinct from the pipeline existing Queue called <code>ResolveContentFutures</code> that will set <code>in_q</code>. See the result of the <code>DeclarativeContent.future</code> to the <code>DeclarativeContent.content</code> diagram below: 3. When <code>FirstStage</code> awaits on <code>DeclarativeContent.get_future()</code> it will receive the fully downloaded Content Unit which it can then use to generate more <code>DeclarativeContent</code> objects to send down the pipeline. This solution was originally inspired by an idea from @mdellweg and suggested by @gmbnomis. !loop_diagram.jpg!
.