« Previous | Next » 

Revision 1b36eeee

Added by bmbouter over 3 years ago

Introducing asyncio downloaders

Adds a HttpDownloader

  • handles both synchronous and asynchronous downloads
  • designed for subclassing to make custom downloader
  • size validation works
  • digest validation works
  • has a well defined return interface DownloadResult

Adds a FileDownloader

Adds a BaseDownloader

Reworks the Factory

  • Updates the Factory to use HttpDownloader and FileDownloader
  • Handles http, https, and file
  • Fully configures HttpDownloader with all importer settings including: ssl client and server options, basic auth, proxy, proxy_auth
  • Can be reused with custom downloaders

Adds a GroupDownloader

  • Downloads files in parallel but only return back to the use when any group of files are fully downloaded.
  • Drivable with or with generators
  • Constrains in-memory objects through generator use

General Updates

  • Updates the importer to use the Factory
  • Updates the docs to use docs from the new code closes #2951