Migration Plugin

Reproduced

In [23]: Pulp2Content.objects.count()                                                                                                                                                                                                         
Out[23]: 150267

In [24]: FileContent.objects.count()                                                                                                                                                                                                          
Out[24]: 90978

In [25]: Pulp2Content.objects.filter(pulp3_content_id=None).count()                                                                                                                                                                           
Out[25]: 59289

It looks like the number of migrated units is different each time, and whether you do a catch-all or a specific migration plan doesn't matter.

Re-running the migration plan does continue adding more FileContent to the database FWIW, so there doesn't appear to be any fundamental reason why the content is being skipped. It's just random.

After a second migration I now have 130,000 FileContent (out of 150,000 expected) instead of the 109,000 I had before or the 91,000 I got in the separate test up above.

I post the observations here for historical purpose

In [96]: batch_count = 36                                                                                                                                                                                          

In [97]: batch_size = 4167                                                                                                                                                                                         

In [98]: pulp2content_qs = Pulp2Content.objects.filter(pulp2_content_type_id='iso')                                                                                                                                

In [99]: migrators = []                                                                                                                                                                                            

In [100]: s = set()                                                                                                                                                                                                

In [101]: for batch_idx in range(batch_count): 
     ...:     start = batch_idx * batch_size 
     ...:     end = (batch_idx + 1) * batch_size 
     ...:     batch = pulp2content_qs[start:end] 
     ...:     migrators.append(batch) 
                                                                                                                                                                                                   

In [102]:  
     ...: for i in migrators: 
     ...:     s.update(set(i)) 
                                                                                                                                                                                                   

In [103]: len(s)                                                                                                                                                                                                   
Out[103]: 101194

In [114]: pulp2content_qs.count()                                                                                                                                                                                  
Out[114]: 149997

1. we needed to remove slicing, another option is if call list() on the qs which will evaluate it, but as a side effect it will need to evaluate all the values at once..
2. seems like because qs is not evaluated the sliced queries are getting evaluated as they are requested and this produces different results. This behaviour was not observed in a small qs, so maybe the the batching_size has some impact

In [1]: pulp2content_qs = Pulp2Content.objects.filter(pulp2_content_type_id='iso')                                                                                                                                 

In [2]: a=pulp2content_qs[0:10]                                                                                                                                                                                    

In [3]: b=pulp2content_qs[0:10]                                                                                                                                                                                    

In [11]: list(a)==list(b)                                                                                                                                                                                          
Out[11]: True

In [14]: a=pulp2content_qs[0:10000]                                                                                                                                                                                

In [15]: b=pulp2content_qs[0:10000]                                                                                                                                                                                

In [16]: list(a)==list(b)                                                                                                                                                                                          
Out[16]: False

Applied in changeset pulp-2to3-migration|97a5d04bc926d0fde2919ffaf12c359ddbc71054.

Applied in changeset pulp:pulp-2to3-migrate|97a5d04bc926d0fde2919ffaf12c359ddbc71054.