Python: thread.error: can't start new thread

So the case is this: i have a python script that listens on certain port for a connection, and when a client connects, it acts as a tcp forwarder, forwarding connections accordingly. somehow, the server.py script throws exception after few client.py connection:


Unhandled exception in thread started by 

Traceback (most recent call last):

  File "server.py", line 55, in server_thread

    thread.start_new_thread(forward_client, (client_socket, server_socket))   

thread.error: can't start new thread


Damn it! so i do what all people been doing, i Google.

From below two URL, i was able to understand the situation and came up with the solution:

- http://blog.tsunanet.net/2010/10/threaderror-cant-start-new-thread.html
- http://adywicaksono.wordpress.com/2007/07/10/i-can-not-create-more-than-255-threads-on-linux-what-is-the-solutions/

Tsunanet states that memory allocation might be the issue. Sort of like where my RAM is not enough for all the threading happening. So acting l33t i follow what he (she?) did, strace:

mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)


Indeed! some memory allocation sort of thingy. but... what the hell is that? Wel.. /me dunno. There came Ady Wicaksono. From his blog, somehow i manage to understand that the formula to calculate the amount of threads that yr app can catter at any one time, is as below:

(total RAM size) / (ulimit -s size, which is the stack allocation, i think) = (number of thread)

Based on that, my RAM is 256MB, and current ulimit -s shows 10240 (10MB)
By calculation, i can only handle 25 threads! Damn it! no wonder python throws all those crappy errors..

So what i did? Simple: ulimit -s 1024 and my python app can ahdle 256 thread at one time. ok enough for me. :)

Ouh, as Tsunanet stae it, to make this permanent, ad below line in your /etc/security/limits.conf
(oh, worth to mention, im using Centos 5.5, 64-bit)

# added for more thread handling

# ref: http://blog.tsunanet.net/2010/10/threaderror-cant-start-new-thread.html



*       -       stack   1024  # 1024kB a.k.a. 1MB


Ok, hole this helps someone. Oh, and i'm on a VPS. :)

/alak

Comments

Popular posts from this blog

SAP: Segregate Different SMTP Mail Server Based on Sender Domain

SAP Client Copy Error: Errors occurred during export of container object FINB-TR-DERIVATION