Issue #4795
Updated by bherring over 5 years ago
h2. Problem
When attempting to sync a feed for a second time in automation/CI (not manually), the sync will return incorrect/invalid information for the task details failing the test "often".
h2. Problem Breakout
There are two discrete issues here:
* `pulp-smash` is now/recently updated polling at a 0.3s rate that. For a test including a second repo sync, this will occasionally returning a task "COMPLETE" but the values for the task details had yet to be reset. The resulting return of non-zero values causes test failures. It appears that the ACTUAL change is none, however the task_details are reporting incorrectly. This was supposedly discovered long ago, but the legacy issue could not be found.
* Due to the above, the values of 'added_count', 'removed_count', 'updated_count' are being permanently updated to values in the db that are incorrect. Meaning, running `pulp-admin tasks details --task-id` of the automated task will result in the invalid values still being populated in the historic record. This appears to be related to legacy issue #4428
h2. Recreation
The RCA seems to be from updating pulp-smash polling to a much quicker rate:
https://github.com/PulpQE/pulp-smash/blob/e385e6685a82709fd167b5374d19c4dfcfdb78c5/pulp_smash/api.py#L687
To test this, locally have pulp-smash installed and set to a VERY low polling rate. At the time of this writing, anything around `0.3` would fail 50-80% of the time. Adjusting to an even smaller value should increase the probabiltiy of failure on a non-patched system.
Note: This MUST be done in an automated fashion with pulp-smash due to the api polling. Testing this by hand or manually will NOT result in issue recreation.
Run this test through pytest with the following syntax in an appropriately prepared virtenv:
<pre>
(pulp2) [herring@redherring api_v2]$ pwd
/home/herring/git/Pulp-2-Tests/pulp_2_tests/tests/rpm/api_v2
(pulp2) [herring@redherring api_v2]$ count=1; while [ $count -le 10 ]; do echo -e "Iteration: $count\n"; pytest -svv test_sync_publish.py::SyncRpmRepoTestCase::test_no_change_in_second_sync --disable-warnings; ((count++)); done
</pre>
A loop value of 10 was chosen to ensure math was easy.
It is expected there will be a 100% pass rate. At this time, even with larger values around 2 seconds, there is still ~10% chance of failure.
Also, running the following on a passing Task ID should NOT result in incorrect values for the task:
<pre>
[root@rhel76 ~]# pulp-admin tasks details --task-id d0140034-e1ca-4d1e-a34f-7fdf1d17a697
</pre>
Where the `--task-id` is appropriate for the jobs being ran.
h2. QE Workaround in place
For the time being, the values for pulp-smash when used on Pulp2 polling can be increase to keep these failures from happening.
Once this task issues are resolved, pulp-smash can have the values restored to lower polling values.
h2. References
* Additional Reference needed here from @dkliban about how the values are calculated
https://pulp.plan.io/issues/4428#note-17
* Original investigation information moved down in other Notes to retain investigation history.
.