Uploaded image for project: 'Python Integrations'
  1. Python Integrations
  2. INTPYTHON-462

Converting String to ObjectId and then writing to MongoDb using PyMongoArrow

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: None
    • Component/s: pymongoarrow
    • None
    • Python Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Via https://github.com/mongodb-labs/mongo-arrow/issues/253

       

      ```

      Hi, I hope you're all having a wonderful day.

      I have a redshift table that includes 4 columns, two of the columns are string version of ObjectId.

      I load the data in polars and then apply the following code.

      ` assignment_fwks = assignment_fwks.with_columns(
                  pl.col("profile_id").map_elements(ObjectId, return_dtype=pl.Object).alias("profile_id"),
                  pl.col("framework_id").map_elements(ObjectId, return_dtype=pl.Object).alias("framework_id"))`
              
              
      However, when I do 

      `pymongoarrow.api.write(my_collection, assignment_fwks)`

      I get the error, 

      `Exception has occurred: PanicException
      called Option::unwrap() on a None value
        File "/home/ubuntu/projects/profile_assigner/src/consumption_assignments/app.py", line 49, in upsert_profile_assignment
          result = write(coll, insertion_fwk_assignments)
        File "/home/ubuntu/projects/profile_assigner/src/consumption_assignments/app.py", line 105, in client_profile_assignments
          upsert_profile_assignment(
        File "/home/ubuntu/projects/profile_assigner/src/consumption_assignments/app.py", line 136, in main
          client_error = client_profile_assignments(region, cli_region_df, credentials)
        File "/home/ubuntu/projects/profile_assigner/src/consumption_assignments/app.py", line 149, in <module>
          main()
      pyo3_runtime.PanicException: called Option::unwrap()`

       

      If i don't convert these columns to ObjectId and keep them as strings, then it works fine and inserts the data correctly into the mongo collection.

      So is there a way I can convert these string columns to ObjectIds and do the insertion to mongo collection, without explicitly having to convert to another data structure like pandas dataframe or List?

      ```

      As long as i can use the arrow format it would be great. As it is very memory and cost efficient.

            Assignee:
            steve.silvester@mongodb.com Steve Silvester
            Reporter:
            alex.clark@mongodb.com Alex Clark
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: