Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18784

Errors during concurrent update/upsert on an existing document with a unique index

    • ALL
    • Hide

      Tested using version 2.6.3 and 3.0.3.
      Run MongoDB locally as follow (Note that the test will write about ~4 GB of data): ./mongodb-osx-x86_64-2.6.10/bin/mongod --dbpath /tmp/mongo Run the following code once, restart the database, comment out the following line: "fillUpCollection(testMongoDB.col1, value, 0, 300);" (IMPORTANT), then run the code again.

      package test;
      
      import com.mongodb.BasicDBObject;
      import com.mongodb.DBCollection;
      import com.mongodb.DBObject;
      import com.mongodb.Mongo;
      import com.mongodb.MongoClient;
      
      import java.io.IOException;
      import java.util.ArrayList;
      import java.util.List;
      import java.util.Random;
      
      public class TestMongoDB {
          public static final String DOC_ID = "docId";
          public static final String VALUE = "value";
          public static final String DB_NAME = "db1";
          public static final String UNIQUE = "unique";
          public static final String BACKGROUND = "background";
          private DBCollection col1;
          private DBCollection col2;
      
          private static DBCollection getCollection(Mongo mongo, String collectionName) {
              DBCollection col =  mongo.getDB(DB_NAME).getCollection(collectionName);
              BasicDBObject index = new BasicDBObject();
              index.append(DOC_ID, 1);
              DBObject indexOptions = new BasicDBObject();
              indexOptions.put(UNIQUE, true);
              indexOptions.put(BACKGROUND, true);
              col.createIndex(index, indexOptions);
              return col;
          }
      
          private static void storeDoc(String docId, DBObject doc, DBCollection dbCollection) throws IOException {
              BasicDBObject query = new BasicDBObject();
              query.put(DOC_ID, docId);
              dbCollection.update(query, doc, true, false);
              //dbCollection.findAndModify(query, null, null, false, doc, false, true);
          }
      
          public static void main(String[] args) throws Exception{
              final String value = new String(new char[1000000]).replace('\0', 'a');
              Mongo mongo = new MongoClient("localhost:27017");
              final TestMongoDB testMongoDB = new TestMongoDB();
              testMongoDB.col1 = getCollection(mongo, "col1");
              testMongoDB.col2 = getCollection(mongo, "col2");
      
              fillUpCollection(testMongoDB.col1, value, 0, 300);
              //restart Database, comment out previous line, and run again
              fillUpCollection(testMongoDB.col2, value, 0, 2000);
              updateExistingDocuments(testMongoDB, value);
          }
      
          private static void updateExistingDocuments(TestMongoDB testMongoDB, String value) {
              List<String> docIds = new ArrayList<String>();
              for(int i = 0; i < 10; i++) {
                  docIds.add(new Random().nextInt(300) + "");
              }
              multiThreadUpdate(testMongoDB.col1, value, docIds);
          }
      
      
          private static void multiThreadUpdate(final DBCollection col, final String value, final List<String> docIds) {
              Runnable worker = new Runnable() {
                  @Override
                  public void run() {
                      try {
                          System.out.println("Started Thread");
                          for(String id : docIds) {
                              storeDoc(id, getDbObject(value, id), col);
                          }
                      } catch (Exception e) {
                          System.out.println(e);
                      } finally {
                          System.out.println("Completed");
                      }
                  }
              };
      
              for(int i = 0; i < 8; i++) {
                  new Thread(worker).start();
              }
          }
      
          private static DBObject getDbObject(String value, String docId) {
              final DBObject object2 = new BasicDBObject();
              object2.put(DOC_ID, docId);
              object2.put(VALUE, value);
              return object2;
          }
      
          private static void fillUpCollection(DBCollection col, String value, int from, int to) throws IOException {
              for(int i = from ; i <= to; i++) {
                  storeDoc(i + "", getDbObject(value, i + ""), col);
              }
          }
      }
      

      Sample Output on the second run:

      Started Thread
      Started Thread
      Started Thread
      Started Thread
      Started Thread
      Started Thread
      Started Thread
      Started Thread
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "290" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "170" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "241" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "127" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "120" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "91" }'
      Completed
      com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "136" }'
      Completed
      Completed
      
      Show
      Tested using version 2.6.3 and 3.0.3. Run MongoDB locally as follow (Note that the test will write about ~4 GB of data): ./mongodb-osx-x86_64-2.6.10/bin/mongod --dbpath /tmp/mongo Run the following code once, restart the database, comment out the following line: "fillUpCollection(testMongoDB.col1, value, 0, 300);" (IMPORTANT), then run the code again. package test; import com.mongodb.BasicDBObject; import com.mongodb.DBCollection; import com.mongodb.DBObject; import com.mongodb.Mongo; import com.mongodb.MongoClient; import java.io.IOException; import java.util.ArrayList; import java.util.List; import java.util.Random; public class TestMongoDB { public static final String DOC_ID = "docId" ; public static final String VALUE = "value" ; public static final String DB_NAME = "db1" ; public static final String UNIQUE = "unique" ; public static final String BACKGROUND = "background" ; private DBCollection col1; private DBCollection col2; private static DBCollection getCollection(Mongo mongo, String collectionName) { DBCollection col = mongo.getDB(DB_NAME).getCollection(collectionName); BasicDBObject index = new BasicDBObject(); index.append(DOC_ID, 1); DBObject indexOptions = new BasicDBObject(); indexOptions.put(UNIQUE, true ); indexOptions.put(BACKGROUND, true ); col.createIndex(index, indexOptions); return col; } private static void storeDoc( String docId, DBObject doc, DBCollection dbCollection) throws IOException { BasicDBObject query = new BasicDBObject(); query.put(DOC_ID, docId); dbCollection.update(query, doc, true , false ); //dbCollection.findAndModify(query, null , null , false , doc, false , true ); } public static void main( String [] args) throws Exception{ final String value = new String ( new char [1000000]).replace( '\0' , 'a' ); Mongo mongo = new MongoClient( "localhost:27017" ); final TestMongoDB testMongoDB = new TestMongoDB(); testMongoDB.col1 = getCollection(mongo, "col1" ); testMongoDB.col2 = getCollection(mongo, "col2" ); fillUpCollection(testMongoDB.col1, value, 0, 300); //restart Database, comment out previous line, and run again fillUpCollection(testMongoDB.col2, value, 0, 2000); updateExistingDocuments(testMongoDB, value); } private static void updateExistingDocuments(TestMongoDB testMongoDB, String value) { List< String > docIds = new ArrayList< String >(); for ( int i = 0; i < 10; i++) { docIds.add( new Random().nextInt(300) + ""); } multiThreadUpdate(testMongoDB.col1, value, docIds); } private static void multiThreadUpdate( final DBCollection col, final String value, final List< String > docIds) { Runnable worker = new Runnable () { @Override public void run() { try { System .out.println( "Started Thread " ); for ( String id : docIds) { storeDoc(id, getDbObject(value, id), col); } } catch (Exception e) { System .out.println(e); } finally { System .out.println( "Completed" ); } } }; for ( int i = 0; i < 8; i++) { new Thread (worker).start(); } } private static DBObject getDbObject( String value, String docId) { final DBObject object2 = new BasicDBObject(); object2.put(DOC_ID, docId); object2.put(VALUE, value); return object2; } private static void fillUpCollection(DBCollection col, String value, int from, int to) throws IOException { for ( int i = from ; i <= to; i++) { storeDoc(i + "", getDbObject(value, i + " "), col); } } } Sample Output on the second run: Started Thread Started Thread Started Thread Started Thread Started Thread Started Thread Started Thread Started Thread com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "290" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "170" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "241" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "127" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "120" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "91" }' Completed com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1 dup key: { : "136" }' Completed Completed

      Given a collection with a unique index and when running concurrent updates (3 threads) with upsert as true on a given existing document, 1 to 2 threads raise the following exception:

      Processing failed (Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$key_1  dup key: { : 123 }'):
      

      I would expect all of the three updates to succeed because the document I am trying to update already exists. Instead, it looks like it is trying to do an insert on few or all of the update requests and few fails due to the unique index. Repeating the same concurrent update on the document does not raise any exceptions. Also, using find() on a document to bring it to the working set, then running the concurrent updates on that document also runs as expected.

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            aaskar aaskar
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: