events-push lease threads

Discussion:

Aaron Long

2013-11-11 15:22:44 UTC

This is our second attempt at using events-push plugin and we keep seeing
the same issue each time. Last time we thought it was because we weren't
using NIO, but now we are setup with NIO connector and using SSE protocol.

We eventually end up with a dead server and hundreds of threads like this:

"http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800 nid=0x2cc1
waiting on condition [0x00007f079b4f0000] java.lang.Thread.State:
WAITING (parking) at sun.misc.Unsafe.park(Native Method) -
parking to wait for <0x00000006372e0460> (a
java.util.concurrent.CountDownLatch$Sync) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
at
org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
at
org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)

-Aaron

Aaron Long

2013-11-11 15:25:32 UTC

Permalink

Should be leaks threads, not lease threads...autocorrect fail.

Here's a more readable stack:

"http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000 nid=0x2e1c
waiting on condition [0x00007f079359e000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x000000078c4b69a0> (a
java.util.concurrent.CountDownLatch$Sync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)

at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)

at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)

at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)

at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)

at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)

at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)

at
org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)

at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)

at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)

at
org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)

at
org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)

at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)

at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)

at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

Post by Aaron Long
This is our second attempt at using events-push plugin and we keep seeing
the same issue each time. Last time we thought it was because we weren't
using NIO, but now we are setup with NIO connector and using SSE protocol.
"http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800 nid=0x2cc1
WAITING (parking) at sun.misc.Unsafe.park(Native Method) -
parking to wait for <0x00000006372e0460> (a
java.util.concurrent.CountDownLatch$Sync) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
at
org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
at
org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
-Aaron

zyro

2013-11-11 15:51:47 UTC

Permalink

that stacktrace does look like tomcat is using nio
("http-nio-8080-exec") but atmosphere does not recognize it or cannot
use it:

...org.atmosphere.container.Tomcat7BIOSupportWithWebSocket...
...BlockingIOCometSupport...

- did you set servlet version to 3.0 everywhere (BuildConfig.groovy and
prior to 2.3.0 in application.properties as well)?
- did you make sure all filters that the requests goes through and the
meteorServlet have <async-supported>true</async-supported>?
- did you try including/excluding the atmosphere-native dependencies?

if those do not help consider using a more recent version of atmosphere
(remember then to update atmosphere.js as well (if required)).

zyro

-------- Original Message --------
Subject: [grails-user] Re: events-push lease threads
From: Aaron Long <***@gmail.com>
To: ***@grails.codehaus.org <***@grails.codehaus.org>
Date: Mon, 11 Nov 2013 10:25:32 -0500

Post by Aaron Long
Should be leaks threads, not lease threads...autocorrect fail.
"http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000
nid=0x2e1c waiting on condition [0x00007f079359e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078c4b69a0> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
at
org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
at
org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
at
org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
This is our second attempt at using events-push plugin and we keep
seeing the same issue each time. Last time we thought it was because
we weren't using NIO, but now we are setup with NIO connector and
using SSE protocol.
"http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800
nid=0x2cc1 waiting on condition [0x00007f079b4f0000]
java.lang.Thread.State: WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
at
org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
at
org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
at
org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
at
org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
at
org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
-Aaron

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Aaron Long

2013-11-11 18:30:59 UTC

Permalink

Yeah, those are all good suggestions. I noticed that Tomcat7 seems to
refuse to use the NIO adapter even though I've done everything that you
suggested.

Looks like the latest builds of events-push are against atmosphere-2.0.2
which some people claim fixes issues such as this one. I might try
switching to the snapshot build and see what happens.

Post by zyro
that stacktrace does look like tomcat is using nio
("http-nio-8080-exec") but atmosphere does not recognize it or cannot
...org.atmosphere.container.Tomcat7BIOSupportWithWebSocket...
...BlockingIOCometSupport...
- did you set servlet version to 3.0 everywhere (BuildConfig.groovy and
prior to 2.3.0 in application.properties as well)?
- did you make sure all filters that the requests goes through and the
meteorServlet have <async-supported>true</async-supported>?
- did you try including/excluding the atmosphere-native dependencies?
if those do not help consider using a more recent version of atmosphere
(remember then to update atmosphere.js as well (if required)).
zyro
-------- Original Message --------
Subject: [grails-user] Re: events-push lease threads
Date: Mon, 11 Nov 2013 10:25:32 -0500

java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)