Puppet Support

Just to ask the silly question: is there any chance the call to delete the repo is happening before the task finishes?

The delete call is triggered before the spawned publish task. From the provided info i assume you got auto-publish set to true.

Just to ask the silly question: is there any chance the call to delete the repo is happening before the task finishes?

I don't think this is the case, based on how the test is written. I'll take another look at journalctl's output and make sure.

The delete call is triggered before the spawned publish task.

What makes you think this?

From the provided info i assume you got auto-publish set to true.

Yes. I'm letting pulp-admin do whatever it does by default, which includes publishing.

This issue definitely has nothing to do with repositories being deleted. I'm running tests with the following modification to Pulp Smash:

diff --git a/pulp_smash/tests/puppet/cli/test_sync.py b/pulp_smash/tests/puppet/cli/test_sync.py
index bc6bb9e..c22772e 100644
--- a/pulp_smash/tests/puppet/cli/test_sync.py
+++ b/pulp_smash/tests/puppet/cli/test_sync.py
@@ -74,9 +74,6 @@ class SyncDownloadedContentTestCase(unittest.TestCase):
                 'pulp-admin puppet repo create '
                 '--repo-id {} --feed {} --queries {}'
             ).format(repo_id, PUPPET_FEED, PUPPET_QUERY).split())
-            self.addCleanup(client.run, (
-                'pulp-admin puppet repo delete --repo-id {}'
-            ).format(repo_id).split())
             client.run((
                 'pulp-admin puppet repo sync run --repo-id {}'
             ).format(repo_id).split())
@@ -131,7 +128,6 @@ class SyncFromPuppetForgeTestCase(unittest.TestCase):
         ).format(repo_id, PUPPET_FEED, PUPPET_QUERY)
         client.run(cmd.split())
         cmd = 'pulp-admin puppet repo delete --repo-id {}'.format(repo_id)
-        self.addCleanup(client.run, cmd.split())

         # Sync the repository.
         cmd = 'pulp-admin puppet repo sync run --repo-id {}'.format(repo_id)

...and the publish error still shows up.

I think I can explain the test failure. Consider the test script posted in the issue:

#!/usr/bin/env bash

pulp-admin puppet repo create --repo-id foo --feed http://forge.puppetlabs.com --queries "pulp-pulp-1.0.0" 
pulp-admin puppet repo sync run --repo-id foo
pulp-admin puppet repo delete --repo-id foo

When pulp-admin puppet repo sync run ... is executed, pulp-admin performs both a sync and a publish. Furthermore, pulp-admin waits for each of these tasks to complete before returning (zero) to the user. But what would happen if the wait-and-poll logic in pulp-admin was broken and pulp-admin returned while a sync was occurring? In that case, the pulp-admin puppet repo delete ... would execute. The whole sequence of events would look like this:

1. pulp-admin issues a synchronous "create repository" command.
2. pulp-admin issues asynchronous "sync repository" and "publish repository" commands. pulp-admin starts polling those tasks, encounters an error while polling the "sync repository" tasks, and returns.
3. pulp-admin issues a synchronous "delete repository" command. As soon as the "sync repository" command completes, the repository is deleted.
4. Pulp attempts to service the "publish repository" command, but the target repository is gone.

Much circumstantial evidence fits with this theory. First, when the following diff is applied to Pulp Smash...

diff --git a/pulp_smash/tests/puppet/cli/test_sync.py b/pulp_smash/tests/puppet/cli/test_sync.py
index bc6bb9e..1f89728 100644
--- a/pulp_smash/tests/puppet/cli/test_sync.py
+++ b/pulp_smash/tests/puppet/cli/test_sync.py
@@ -78,7 +78,7 @@ class SyncDownloadedContentTestCase(unittest.TestCase):
                 'pulp-admin puppet repo delete --repo-id {}'
             ).format(repo_id).split())
             client.run((
-                'pulp-admin puppet repo sync run --repo-id {}'
+                'pulp-admin -v puppet repo sync run --repo-id {}'
             ).format(repo_id).split())

         # Verify the number of puppet modules in each repository.
@@ -134,7 +134,7 @@ class SyncFromPuppetForgeTestCase(unittest.TestCase):
         self.addCleanup(client.run, cmd.split())

         # Sync the repository.
-        cmd = 'pulp-admin puppet repo sync run --repo-id {}'.format(repo_id)
+        cmd = 'pulp-admin -v puppet repo sync run --repo-id {}'.format(repo_id)
         completed_proc = client.run(cmd.split())

         # Verify no errors were reported.

...I get the following output:

$ python -m unittest pulp_smash.tests.puppet.cli.test_sync
/home/ichimonji10/.venvs/pulp-smash/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
E.
======================================================================
ERROR: test_sync_downloaded_content (pulp_smash.tests.puppet.cli.test_sync.SyncDownloadedContentTestCase)
Create two repositories with the same feed, and sync them serially.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/puppet/cli/test_sync.py", line 79, in test_sync_downloaded_content
    ).format(repo_id).split())
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/cli.py", line 242, in run
    return self.response_handler(completed_process)
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/cli.py", line 78, in code_handler
    completed_proc.check_returncode()
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/cli.py", line 153, in check_returncode
    self.stderr,
pulp_smash.exceptions.CalledProcessError: Command ['pulp-admin', '-v', 'puppet', 'repo', 'sync', 'run', '--repo-id', 'f603e575-1165-4afb-9f2e-af84c6790160'] returned non-zero exit status 70.

stdout: +----------------------------------------------------------------------+
    Synchronizing Repository [f603e575-1165-4afb-9f2e-af84c6790160]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
An unexpected error has occurred.More information may be found using the -vv
flag.

stderr: 2017-02-08 14:18:34,134 - ERROR - Client-side exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py", line 474, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.7/site-packages/okaara/cli.py", line 974, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/usr/lib/python2.7/site-packages/pulp/client/extensions/extensions.py", line 210, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/client/commands/repo/sync_publish.py", line 203, in run
    self.poll([sync_task], kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/client/commands/polling.py", line 120, in poll
    task = self._poll_task(task)
  File "/usr/lib/python2.7/site-packages/pulp/client/commands/polling.py", line 211, in _poll_task
    self.progress(task, running_spinner)
  File "/usr/lib/python2.7/site-packages/pulp/client/commands/repo/sync_publish.py", line 102, in progress
    self.renderer.display_report(task.progress_report)
  File "/usr/lib/python2.7/site-packages/pulp_puppet/extensions/admin/repo/status.py", line 38, in display_report
    self._display_sync_metadata_step(sync_report)
  File "/usr/lib/python2.7/site-packages/pulp_puppet/extensions/admin/repo/status.py", line 66, in _display_sync_metadata_step
    item_type, self.sync_metadata_bar, sync_report.metadata_state)
  File "/usr/lib/python2.7/site-packages/pulp_puppet/extensions/admin/repo/status.py", line 240, in _render_itemized_in_progress_state
    progress_bar.render(items_done, items_total, message=bar_message)
  File "/usr/lib/python2.7/site-packages/okaara/progress.py", line 100, in render
    percentage = float(step) / float(total)
TypeError: float() argument must be a string or a number

----------------------------------------------------------------------
Ran 2 tests in 18.491s

FAILED (errors=1)

Also, all of the API tests succeed. Only CLI tests fail.