Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-1799

bson encoding subtypes of known bson types is not thread safe

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Minor - P4 Minor - P4
    • 3.8
    • Affects Version/s: None
    • Component/s: BSON
    • None

      Adding new entries to bson._ENCODERS is not thread safe because we also iterate it.
      The attached repro script, bson-not-thread-safe.py , shows the issue. When multiple threads are encoding custom classes one will fail with a RuntimeError:

      Exception while encoding <class '__main__.MyInt_0_4'>: dictionary changed size during iteration
      Exception in thread Thread-1:
      Traceback (most recent call last):
        File "/usr/local/Cellar/pypy/5.10.0_1/libexec/lib-python/2.7/threading.py", line 797, in __bootstrap_inner
          self.run()
        File "/usr/local/Cellar/pypy/5.10.0_1/libexec/lib-python/2.7/threading.py", line 750, in run
          self.__target(*self.__args, **self.__kwargs)
        File "bson-not-thread-safe.py", line 9, in target
          bson.BSON.encode({'my_int': my_int()})
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 1049, in encode
          return cls(_dict_to_bson(document, check_keys, codec_options))
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 831, in _dict_to_bson
          check_keys, opts))
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 816, in _element_to_bson
          return _name_value_to_bson(name, value, check_keys, opts)
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 786, in _name_value_to_bson
          for base in _ENCODERS:
      RuntimeError: dictionary changed size during iteration
      

      We could fix this by:
      1) remove the logic to cache custom subtypes, or
      2) Instead of modifying _ENCODERS we can add new entries to another dict, like _CACHED_ENCODERS. This would require a second dict lookup for each encoded value, or
      3) Instead of iterating over the keys of _ENCODERS (which may change), iterate over a global copy of the initial keys.

      3) seems like the best solution to me.

      Note: This issue is only present went the C extensions are not installed.

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: