Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-2265

Overly validating documents in bulk inserts

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.10.0
    • Affects Version/s: None
    • Component/s: libmongoc
    • None
    • Fully Compatible

        bool                                                                                 
        mongoc_bulk_operation_insert_with_opts (mongoc_bulk_operation_t *bulk,               
                                                const bson_t *document,                      
                                                const bson_t *opts,                          
                                                bson_error_t *error)                         
        {                                                                                    
      ....
           if (opts && bson_iter_init_find_case (&iter, opts, "legacyIndex") &&              
               bson_iter_as_bool (&iter)) {                                                  
              if (!_mongoc_validate_legacy_index (document, error)) {                        
                 return false;                                                               
              }                                                                              
           } else if (!_mongoc_validate_new_document (document, error)) {                    
              return false;                                                                  
           }   
      

      When creating a bulk of large number of small documents the _mongoc_validate_new_document () takes 20% of the total execution time, including a localhost roundtrip.
      When creating a bulk of few but very large documents, _mongoc_validate_new_document() takes up to 90% of the time.

      I think we can reduce this validation significantly. It should be a programming error to provide corrupt bson_t for example.
      Maybe add a flag to the opts to skip the validation, which would assume the bson_t was already bson_validate()d by the application before presented to the bulk operations.

            Assignee:
            xiangyu.yao@mongodb.com Xiangyu Yao (Inactive)
            Reporter:
            bjori Hannes Magnusson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: