 |
 |
I will analyse some memory dumps to identify possible causes. Maybe it's only related to threads ... On Aug 26, 2013, at 19:02, Ralf Buschermöhle <[log in to unmask]> wrote: > Hi, > > sorry, it seems I missed the exception in the log, after around ~ 2000 connects & disconnects I receive > > Exception in thread "SlaveMonitor:: " java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at ec.eval.SlaveConnection.buildThreads(SlaveConnection.java:161) > at ec.eval.SlaveConnection.<init>(SlaveConnection.java:82) > at ec.eval.SlaveMonitor.registerSlave(SlaveMonitor.java:220) > at ec.eval.SlaveMonitor$1.run(SlaveMonitor.java:192) > at java.lang.Thread.run(Thread.java:679) > > In the test case I started and shut down the slaves immediately (after 2s) - but I received the same exception in the log of the cluster nodes and there are usually hours between starting and stopping the slaves. > > Greetings, > > Ralf > > > P.S. Just for completion > > The others (regarding the socket(s)) are: > > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:185) > at java.net.SocketInputStream.read(SocketInputStream.java:199) > at java.io.DataInputStream.readByte(DataInputStream.java:265) > at ec.eval.SlaveConnection.readLoop(SlaveConnection.java:260) > at ec.eval.SlaveConnection$1.run(SlaveConnection.java:150) > > at "dataOut.writeByte(Slave.V_SHUTDOWN)" > > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) > at java.net.SocketOutputStream.write(SocketOutputStream.java:132) > at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) > at ec.eval.SlaveConnection.shutdown(SlaveConnection.java:99) > at ec.eval.SlaveConnection.readLoop(SlaveConnection.java:327) > at ec.eval.SlaveConnection$1.run(SlaveConnection.java:150) > > On Aug 26, 2013, at 17:20, Sean Luke <[log in to unmask]> wrote: > >> On Aug 26, 2013, at 10:27 AM, Ralf Buschermöhle wrote: >> >>> These are just the running nodes. Previously there have been a few thousand connections from a cluster (handled successfully). >> >> Some more. Try executing the following command on your BSD box to see how many sockets and files (combined) you can have open at one time: >> >> sysctl kern.maxfilesperproc >> >> On my Mac (a BSD box) I get around 10K. >> >> I wonder if ECJ isn't properly closing the sockets, and so you're hitting a socket limit by repeatedly adding and removing clients. It looks correct to me though. >> >> Sean >