Uploaded image for project: 'Python Integrations'
  1. Python Integrations
  2. INTPYTHON-326

LlamaIndex: Error when storing to MongoDBIndexStore and MongoDBDocumentStore for Knowledge Graph

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: None
    • Component/s: AI/ML
    • None
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      User reported an error with using LlamaIndex integrations in MongoDB

      https://github.com/run-llama/llama_index/issues/9546

      Detailed steps to reproduce the problem?

      Here's the code snippet

      documents = SimpleDirectoryReader("kg").load_data()
      graph_store = NebulaGraphStore(
      space_name=space_name,
      edge_types=edge_types,
      rel_prop_names=rel_prop_names,
      tags=tags,
      )
      docstore = MongoDocumentStore.from_uri(uri=MONGO_URI)
      index_store = MongoIndexStore.from_uri(uri=MONGO_URI)

      storage_context = StorageContext.from_defaults(
      graph_store=graph_store,
      docstore=docstore,
      index_store=index_store
      )
      index = KnowledgeGraphIndex.from_documents(
      documents,
      max_triplets_per_chunk=10,
      storage_context=storage_context,
      service_context=service_context,
      include_embeddings=True,
      )

      Stack Trace

      Traceback (most recent call last):
      File "/home/ssm-user/llama/testnebula.py", line 84, in <module>
      index = KnowledgeGraphIndex.from_documents(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/llama_index/indices/base.py", line 106, in from_documents
      return cls(
      ^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/llama_index/indices/knowledge_graph/base.py", line 81, in _init_
      super()._init_(
      File "/home/ssm-user/llama/lib/python3.11/site-packages/llama_index/indices/base.py", line 73, in _init_
      self._storage_context.index_store.add_index_struct(self._index_struct)
      File "/home/ssm-user/llama/lib/python3.11/site-packages/llama_index/storage/index_store/keyval_index_store.py", line 38, in add_index_struct
      self._kvstore.put(key, data, collection=self._collection)
      File "/home/ssm-user/llama/lib/python3.11/site-packages/llama_index/storage/kvstore/mongodb_kvstore.py",line 112, in put
      self._db[collection].replace_one(
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/collection.py", line 973, in replace_one
      self._update_retryable(
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/collection.py", line 881, in _update_retryable
      return self.__database.client._retryable_write(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1523, in _retryable_write
      return self._retry_with_session(retryable, func, s, bulk)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1421, in _retry_with_session
      return self._retry_internal(
      ^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
      return func(self, *args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1462, in _retry_internal
      ).run()
      ^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/mongo_client.py", line 2315, in run
      return self._read() if self._is_read else self._write()
      ^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/mongo_client.py", line 2423, in _write
      return self._func(self._session, conn, self._retryable) # type: ignore
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/collection.py", line 862, in _update
      return self._update(
      ^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/collection.py", line 816, in _update
      result = conn.command(
      ^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/helpers.py", line 322, in inner
      return func(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/pool.py", line 996, in command
      self._raise_connection_failure(error)
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/pool.py", line 968, in command
      return command(
      ^^^^^^^^
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/network.py", line 164, in command
      message._raise_document_too_large(name, size, max_bson_size + message._COMMAND_OVERHEAD)
      File "/home/ssm-user/llama/lib/python3.11/site-packages/pymongo/message.py", line 1182, in _raise_document_too_large
      raise DocumentTooLarge(f"

      {operation!r}

      command document too large")
      pymongo.errors.DocumentTooLarge: 'update' command document too large

      Definition of done: what must be done to consider the task complete?

      Describe how MongoDB is set up. Local vs Hosted, version, topology, load balanced, etc.

      Atlas

      The operating system and version (e.g. Windows 7, OSX 10.8, ...)

      Web framework or asynchronous network library used, if any, with version (e.g. Django 1.7, mod_wsgi 4.3.0, gevent 1.0.1, Tornado 4.0.2, ...)

      LLamaIndex

            Assignee:
            prakul.agarwal@mongodb.com Prakul Agarwal
            Reporter:
            prakul.agarwal@mongodb.com Prakul Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: