-
Type:
Spec Change
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Component/s: CRUD
-
None
-
Needed
In PYTHON-2055, we discovered that PyMongo could generate a batched bulk write larger than maxMessageSizeBytes. Specifically, when using OP_MSG with OP_COMPRESSED we did not factor in the 16-byte message header.
In response I added this test to ensure that batch sizes are calculated appropriately:
@client_context.require_version_min(3, 6)
def test_bulk_max_message_size(self):
self.coll.delete_many({})
self.addCleanup(self.coll.delete_many, {})
_16_MB = 16 * 1000 * 1000
# Generate a list of documents such that the first batched OP_MSG is
# as close as possible to the 48MB limit.
docs = [
{'_id': 1, 'l': 's' * _16_MB},
{'_id': 2, 'l': 's' * _16_MB},
{'_id': 3, 'l': 's' * (_16_MB - 10000)},
]
# Fill in the remaining ~10000 bytes with small documents.
for i in range(4, 10000):
docs.append({'_id': i})
result = self.coll.insert_many(docs)
self.assertEqual(len(docs), len(result.inserted_ids))
I propose we turn this into a prose test to help avoid this bug in other drivers.
- related to
-
PYTHON-2055 Large bulk inserts with compressor throws DBException error
-
- Closed
-