-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 2.6.3
-
Component/s: None
-
None
-
Environment:$ uname -r
3.2.0-51-virtual
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Codename: precise
$ python -V
Python 2.7.3
$ pip freeze
Flask==0.10.1
Flask-WTF==0.9.3
Jinja2==2.7.1
MarkupSafe==0.18
SOAPpy==0.12.5
WTForms==1.0.5
Werkzeug==0.9.4
argparse==1.2.1
configparser==3.3.0r2
docutils==0.11
fpconst==0.7.2
gevent==1.0
greenlet==0.4.1
gunicorn==18.0
itsdangerous==0.23
pymongo==2.6.3
python-dateutil==2.2
pytz==2013.8
requests==2.0.1
rspy==0.1
six==1.4.1
wsgiref==0.1.2
wstools==0.4.3$ uname -r 3.2.0-51-virtual $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.2 LTS Release: 12.04 Codename: precise $ python -V Python 2.7.3 $ pip freeze Flask==0.10.1 Flask-WTF==0.9.3 Jinja2==2.7.1 MarkupSafe==0.18 SOAPpy==0.12.5 WTForms==1.0.5 Werkzeug==0.9.4 argparse==1.2.1 configparser==3.3.0r2 docutils==0.11 fpconst==0.7.2 gevent==1.0 greenlet==0.4.1 gunicorn==18.0 itsdangerous==0.23 pymongo==2.6.3 python-dateutil==2.2 pytz==2013.8 requests==2.0.1 rspy==0.1 six==1.4.1 wsgiref==0.1.2 wstools==0.4.3
Note from jesse: This is arguably a bug in Gevent, or in how people use Gevent. Now that Gevent 1.0 uses a thread pool for DNS resolution by default, one must be careful not to import a module that does network I/O when it is imported. There are a number of solid workarounds:
- Don't do create a MongoClient or MongoReplicaSetClient at the time your module is imported. Instead, create the client in your main program file.
- Or, set the GEVENT_RESOLVER environment variable to "ares".
- Or, execute unicode('foo').encode('idna') before importing a module that creates a client at import time.
- Or, use Gevent 0.13.8 instead of 1.0 until Gevent's issue 349 is resolved.
Original bug report:
Today I did a deployment of an application without pinning the virtualenv to gevent 0.13, which caused the latest version of gevent 1.0 to install into the virtualenv. I found that I'm seeing the instantiation of the MongoReplicaSetClient cause the program to hang. Running an strace on the PID, I see that the process is halted at an epoll_wait() syscall.
# strace -p 3615 Process 3615 attached - interrupt to quit clock_gettime(CLOCK_MONOTONIC, {8908174, 879475984}) = 0 gettimeofday({1386196654, 512281}, NULL) = 0 clock_gettime(CLOCK_MONOTONIC, {8908174, 879932439}) = 0 epoll_wait(4,
That file descriptor is an anonymous inode, according to lsof:
python 4913 webapp 4u 0000 0,9 0 4738 anon_inode
The traceback after interrupting the process is here:
$ python mcal-cron.py ^CKeyboardInterrupt Traceback (most recent call last): File "mcal-cron.py", line 28, in <module> from mcal.config import config File "/home/webapp/apps/mcal/mcal/mcal/__init__.py", line 19, in <module> mongo = pymongo.MongoReplicaSetClient(config.MONGO_IP, use_greenlets=False, **config.MONGO_KWARGS) File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 742, in __init__ self.refresh() File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1179, in refresh res, connection_pool, ping_time = self.__is_master(host) File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1056, in __is_master sock_info = connection_pool.get_socket() File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 339, in get_socket sock_info, from_pool = self.connect(pair), False File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 265, in connect sock = self.create_connection(pair) File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 237, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/socket.py", line 621, in getaddrinfo return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags) File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs) File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get() File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/event.py", line 226, in get result = self.hub.switch() File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch return greenlet.switch(self) KeyboardInterrupt
Rolling back the installed version of gevent to 0.13 resolves the issue. As you can see above, I did explicitly set the use_greenlets arg to False, though since the default is already false, this had no effect.
Let me know if I can help further.