Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-3233

Improved Bulk Write API

XMLWordPrintableJSON

    • Type: Icon: Epic Epic
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.9
    • Affects Version/s: None
    • Component/s: Spec Comp
    • None
    • Needed
    • Hide
      1. What would you like to communicate to the user about this feature?
      Server 8.0 introduces a new bulkWrite command. It is expected to require less network round trips than existing driver bulk write API.
      Suggest updating the Bulk Write Operations to suggest users use the new {{MongoClient.bulk_write}} if using server 8.0+.
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      Yes, here is our current examples page: https://github.com/mongodb/mongo-python-driver/blob/master/doc/examples/client_bulk.rst
      3. Which versions of the driver/connector does this apply to?
      PyMongo 4.9
      Show
      1. What would you like to communicate to the user about this feature? Server 8.0 introduces a new bulkWrite command. It is expected to require less network round trips than existing driver bulk write API. Suggest updating the Bulk Write Operations to suggest users use the new {{MongoClient.bulk_write}} if using server 8.0+. 2. Would you like the user to see examples of the syntax and/or executable code and its output? Yes, here is our current examples page: https://github.com/mongodb/mongo-python-driver/blob/master/doc/examples/client_bulk.rst 3. Which versions of the driver/connector does this apply to? PyMongo 4.9
    • Done
    • Improved Bulk Write API
    • 6
    • 6
    • 8
    • Hide

      Engineer: Shruti Sridhar, Steven Silvester

      Summary:
      2024-08-29

      • What was completed over the last two weeks?
        • Addressed follow up bugs and performance regressions
      • What's the focus over the next two weeks?
        • Finish remaining bugs and add C extension for performance. 
        • Finish epic in time for 8.0 compatibility release (PyMongo 4.9)

      Engineer: Shruti Sridhar

      Summary:
      2024-08-15

      • What was completed over the last two weeks?
        • Merged the PR for the main Python implementation
        • Added performance benchmarks and documentation
      • What's the focus over the next two weeks?
        • Address any test failures that come up, explore how to improve performance

      Summary:
      2024-08-02

      • What was completed over the last two weeks?
        • Opened PR, addressed code review comments
      • What's the focus over the next two weeks?
        • Address any additional comments and merge PR

      Summary:
      2024-07-19

      • What was completed over the last two weeks?
        • Implemented necessary functionality (sending server commands, processing response cursor, batch splitting, error handling) to get all spec tests to pass (only testing the sync version of the API)
        • Wrote most prose tests, got them to pass for sync + async APIs
      • What's the focus over the next two weeks?
        • Finish implementing all the prose tests and getting them to pass
        • Run spec tests on the async version of the API once possible
        • Document, refactor, reorganize code as needed, then open PR
      • Impediments encountered over the last two weeks:
        • Some difficulties with transaction-related and server selection logging-related spec test failures, but have now been resolved

          Summary:
          2024-07-05

      • What was completed over the last two weeks?
        • Designed the user-facing MongoClient.bulkWrite interface
        • Outlined the _ClientBulk class containing the private guts of the API, designed the ClientBulkWriteContext, ClientBulkWriteResult, and ClientBulkWriteException classes, and implemented the write model classes
      • What's the focus over the next two weeks?
        • Implement methods in _ClientBulk and associated classes to enable sending bulk writes to the server, handling the response
        • Start to implement command batching and error handling
      • Impediments encountered over the last two weeks:
        • None
      Show
      Engineer: Shruti Sridhar, Steven Silvester Summary: 2024-08-29 What was completed over the last two weeks? Addressed follow up bugs and performance regressions What's the focus over the next two weeks? Finish remaining bugs and add C extension for performance.  Finish epic in time for 8.0 compatibility release (PyMongo 4.9) Engineer: Shruti Sridhar Summary: 2024-08-15 What was completed over the last two weeks? Merged the PR for the main Python implementation Added performance benchmarks and documentation What's the focus over the next two weeks? Address any test failures that come up, explore how to improve performance Summary: 2024-08-02 What was completed over the last two weeks? Opened PR, addressed code review comments What's the focus over the next two weeks? Address any additional comments and merge PR Summary: 2024-07-19 What was completed over the last two weeks? Implemented necessary functionality (sending server commands, processing response cursor, batch splitting, error handling) to get all spec tests to pass (only testing the sync version of the API) Wrote most prose tests, got them to pass for sync + async APIs What's the focus over the next two weeks? Finish implementing all the prose tests and getting them to pass Run spec tests on the async version of the API once possible Document, refactor, reorganize code as needed, then open PR Impediments encountered over the last two weeks: Some difficulties with transaction-related and server selection logging-related spec test failures, but have now been resolved Summary: 2024-07-05 What was completed over the last two weeks? Designed the user-facing MongoClient.bulkWrite interface Outlined the _ClientBulk class containing the private guts of the API, designed the ClientBulkWriteContext , ClientBulkWriteResult , and ClientBulkWriteException classes, and implemented the write model classes What's the focus over the next two weeks? Implement methods in _ClientBulk and associated classes to enable sending bulk writes to the server, handling the response Start to implement command batching and error handling Impediments encountered over the last two weeks: None
    • Hide

      DRIVERS-716:
      Summary of necessary driver changes

      • Implement the new bulk write API and tests defined in the specification.

      Commits for syncing spec/prose tests

      • Sync spec tests and implement prose tests from this commit.
        • 2024-08-28 include skips on serverless from 0984b09
        • 2024-08-29 include missing skip on serverless from e9f02f3
      • Sync the unacknowledged-client-bulkWrite command monitoring test files from this commit. The outdated version of this test in the commit from the previous bullet has a potential race.
      • See DRIVERS-2993 for a fix to unacknowledged write handling.

      Reference implementations

      Show
      DRIVERS-716 : Summary of necessary driver changes Implement the new bulk write API and tests defined in the specification . Commits for syncing spec/prose tests Sync spec tests and implement prose tests from this commit . 2024-08-28 include skips on serverless from  0984b09 2024-08-29 include missing skip on serverless from e9f02f3 Sync the unacknowledged-client-bulkWrite command monitoring test files from this commit . The outdated version of this test in the commit from the previous bullet has a potential race. See DRIVERS-2993 for a fix to unacknowledged write handling. Reference implementations Rust C

      This ticket was split from DRIVERS-716, please see that ticket for a detailed description.

            Assignee:
            steve.silvester@mongodb.com Steve Silvester
            Reporter:
            dbeng-pm-bot PM Bot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:
              9 weeks, 2 days