Error message here!

Hide Error message here!

忘记密码?

Error message here!

请输入正确邮箱

Hide Error message here!

密码丢失?请输入您的电子邮件地址。您将收到一个重设密码链接。

Error message here!

返回登录

Close

从一个OutOfMemoryError 学会了分析内存泄漏问题

CanntBelieve 2019-06-19 07:19:00 阅读数:188 评论数:0 点赞数:0 收藏数:0

以前都是好好的,最近出现了 oom。

问题


开始是: java.lang.OutOfMemoryError: Java heap space
-- ::41.678 ERROR  --- [nio--exec-] c.e.p.s.c.c.core.ELDictionaryController  : 系统异常

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:) ~[spring-webmvc-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:) ~[spring-webmvc-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:) [spring-webmvc-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:) [spring-webmvc-5.1..RELEASE.jar!/:5.1..RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:) [spring-webmvc-5.1..RELEASE.jar!/:5.1..RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:) [tomcat-embed-websocket-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:) [spring-web-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:) [spring-web-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.filters.RemoteIpFilter.doFilter(RemoteIpFilter.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.filters.RemoteIpFilter.doFilter(RemoteIpFilter.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:) [druid-1.1..jar!/:1.1.]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at com.lkk.platform.system.controller.filter.CORSFilter.doFilter(CORSFilter.java:) [erdp_system_controller-2.0.-GA.jar!/:2.0.-GA]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:) [spring-web-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:) [spring-web-5.1..RELEASE.jar!/:5.1..RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:) [na:1.8.0_212]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:) [tomcat-embed-core-9.0..jar!/:9.0.]
        at java.lang.Thread.run(Thread.java:) [na:1.8.0_212]
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.jar.Manifest.read(Manifest.java:) ~[na:1.8.0_212]
        at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:) ~[na:1.8.0_212]
        at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:) ~[na:1.8.0_212]
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:) ~[na:1.8.0_212]
        at java.util.jar.JarVerifier.update(JarVerifier.java:) ~[na:1.8.0_212]
        at java.util.jar.JarInputStream.read(JarInputStream.java:) ~[na:1.8.0_212]
        at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:) ~[na:1.8.0_212]
        at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:) ~[na:1.8.0_212]
        at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:) ~[na:1.8.0_212]
        at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:) ~[na:1.8.0_212]
        at org.apache.catalina.webresources.JarWarResourceSet.getArchiveEntries(JarWarResourceSet.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.webresources.Cache.getResource(Cache.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader.findClassIgnoringNotFound(TomcatEmbeddedWebappClassLoader.java:) ~[spring-boot-2.1..RELEASE.jar!/:2.1..RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader.doLoadClass(TomcatEmbeddedWebappClassLoader.java:) ~[spring-boot-2.1..RELEASE.jar!/:2.1..RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadClass(TomcatEmbeddedWebappClassLoader.java:) ~[spring-boot-2.1..RELEASE.jar!/:2.1..RELEASE]
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:) ~[tomcat-embed-core-9.0..jar!/:9.0.]
        at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:) ~[logback-classic-1.2..jar!/:na]
        at ch.qos.logback.classic.Logger.error(Logger.java:) ~[logback-classic-1.2..jar!/:na]
View Code
512M 不够吗? 很有可能啊...
增加内存到1G 后仍然出现问题:Failed to mark a promise as failure because it has failed already: [DefaultChannelPromise@33a99639(failure: io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: GC overhead limit exceeded), io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

-- ::42.648  WARN  --- [erverWorker--] o.a.d.r.exchange.codec.ExchangeCodec     :  [DUBBO] Fail to encode response: Response [id=319633, version=2.0., status=, event=false, error=null, result=RpcResult [result=null, exception=org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded]], send bad_response info instead, cause: GC overhead limit exceeded, dubbo version: 2.7., current host: 192.168.11.183

java.lang.OutOfMemoryError: GC overhead limit exceeded

-- ::43.907 ERROR  --- [-thread-] o.a.dubbo.rpc.filter.ExceptionFilter     :  [DUBBO] Got unchecked and undeclared exception which called by 192.168.11.183. service: com.elead.platform.system.domain.service.ELCommonCodeRegulationService, method: GetCode, exception: java.lang.OutOfMemoryError: GC overhead limit exceeded, dubbo version: 2.7., current host: 192.168.11.183
View Code

 

这就奇怪了! 注意到 出现次数比较多是 com.lkk.platform.system.domain.service.ELCommonCodeRegulationService, method: GetCode,

    @Transactional(readOnly = false)
    public String GetCode(String name){
        RLock rlock = redissonManager.getRedisson().getLock(name);
        boolean getLock = false;
        try{
            getLock = rlock.tryLock(, , TimeUnit.SECONDS);
            if (getLock){
                ELCodeDef elCodeDef = findCommonCode(name);
                super.updateById(elCodeDef);
                return elCodeDef.getCode();
            }
        }catch (Exception ex){
            ex.printStackTrace();
        }finally {
            if (getLock) {
                rlock.unlock();
            }
        }
        return "";
    }

而
    @Autowired
    RedissonManager redissonManager;
 

 

分析

由此怀疑这个地方有些问题。 虽然出现了oom, 但是进程没有死, 似乎依然可以响应某些请求,于是把线程拔下来, 观察一番,发现 redisson-netty 竟然有上千个

就是这个

"redisson-netty-25-32" # prio= os_prio= tid=0x00007f7ec0187800 nid=0x3625 runnable [0x00007f7e77d6c000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:)
    - locked <0x00000000e90b27a0> (a io.netty.channel.nio.SelectedSelectionKeySet)
    - locked <0x00000000e90b27f8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000e90b2708> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:)
    at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:)
    at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
    at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:)
    at java.lang.Thread.run(Thread.java:)

 

import org.apache.commons.lang3.StringUtils;
import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;
import org.redisson.config.ReadMode;
import org.redisson.config.SentinelServersConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Component
public class RedissonManager {

    @Autowired
    RedisTemplate<String, Object> redisTemplate;

    @Value("${spring.redis.password}")
    private String redisPassword;

    @Value("${spring.redis.port}")
    private String redisPort;

    @Value("${spring.redis.host}")
    private String redisHost;

    @Value("${spring.redis.timeout}")
    private String redisTimeout;

    @Value("${spring.redis.sentinel.node}")
    private String redisSentinelNode;

    @Value("${spring.redis.sentinel.master}")
    private String redisSentinelMaster;

    @Bean
    public RedissonClient getRedisson() {
        Config config = new Config();
        if (StringUtils.isNotEmpty(redisPort)) {
            config.useSingleServer().setAddress("redis://" + redisHost + ":" + redisPort).setPassword(redisPassword);
        } else if (StringUtils.isNotEmpty(redisSentinelNode)) {
            String[] nodes = redisSentinelNode.split(",");
            List<String> newNodes = new ArrayList(nodes.length);
            Arrays.stream(nodes).forEach((index) -> newNodes.add(index.startsWith("redis://") ? index : "redis://" + index));
            SentinelServersConfig serverConfig = config.useSentinelServers()
                    .addSentinelAddress(newNodes.toArray(new String[]))
                    .setMasterName(redisSentinelMaster)
                    .setReadMode(ReadMode.SLAVE)
                    .setTimeout(Integer.valueOf(redisTimeout));
            if(StringUtils.isNotEmpty(redisPassword)){
                serverConfig.setPassword(redisPassword);
            }
        }
        return Redisson.create(config);
    }
}
View Code

太不正常了!

 

redisson 的bug 吗? redisson 的官网 的issue 搜索一番,无果。 郁闷了!

把java 的heap 拔下来,看到有些异常:

肯定不是 spring 的classloader 吧。

 

 

 

 

看到 netty 的PoolThreadCache 比较可疑啊, 还有 mybatis。

 

 

 

 

Biggest Top-Level Dominator Packages 跟之前一样的提示, 一个是netty的 PollThreadCache, 一个是netty 的epoll, 还有是redision, 还有是sun 的EPollArrayWrapper, 还有mybatis,其他 也看不出什么来啊!

 

分析只能到此为止了吗? io.netty.buffer.PoolThreadCache 是什么东东? 我不熟悉啊!  看过netty 源码, 已经全忘了!

 

还是从redision 入手吧。  咦, redision 的用法好像不太对哦!!! 改一下吧:

    @Autowired
    RedissonClient redissonClient;
==> 
 @Autowired 
 RedissonManager redissonManager; 
 
 
 RLock rlock = redissonClient.getLock(name); ==>  RLock lock = redissonManager.getRedisson().getLock(name);

RedissonManager

 

import org.apache.commons.lang3.StringUtils;
import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;
import org.redisson.config.ReadMode;
import org.redisson.config.SentinelServersConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Component
public class RedissonManager {

    @Autowired
    RedisTemplate<String, Object> redisTemplate;

    @Value("${spring.redis.password}")
    private String redisPassword;

    @Value("${spring.redis.port}")
    private String redisPort;

    @Value("${spring.redis.host}")
    private String redisHost;

    @Value("${spring.redis.timeout}")
    private String redisTimeout;

    @Value("${spring.redis.sentinel.node}")
    private String redisSentinelNode;

    @Value("${spring.redis.sentinel.master}")
    private String redisSentinelMaster;

    @Bean
    public RedissonClient getRedisson() {
        Config config = new Config();
        if (StringUtils.isNotEmpty(redisPort)) {
            config.useSingleServer().setAddress("redis://" + redisHost + ":" + redisPort).setPassword(redisPassword);
        } else if (StringUtils.isNotEmpty(redisSentinelNode)) {
            String[] nodes = redisSentinelNode.split(",");
            List<String> newNodes = new ArrayList(nodes.length);
            Arrays.stream(nodes).forEach((index) -> newNodes.add(index.startsWith("redis://") ? index : "redis://" + index));
            SentinelServersConfig serverConfig = config.useSentinelServers()
                    .addSentinelAddress(newNodes.toArray(new String[]))
                    .setMasterName(redisSentinelMaster)
                    .setReadMode(ReadMode.SLAVE)
                    .setTimeout(Integer.valueOf(redisTimeout));
            if(StringUtils.isNotEmpty(redisPassword)){
                serverConfig.setPassword(redisPassword);
            }
        }
        return Redisson.create(config);
    }
}
View Code

 

 

 

 

改了就好了!!突然自己明白了, 原来就是这个redision 用法错误导致的!!

 

 

总结

其实上面的 thread dump 和 heap dump 已经给出了比较明显的答案了!! 就是 PoolThreadCache 占用了 过多的内存, 其原因就是 PoolThreadCache 错误的创建了 太多!————  本来应该是单例的 对象, 被搞成了 prototype, 你说是不是引起了大错!!! 一个 PoolThreadCache占用内存差不多196,000byte, 921个就 是 180516000 byte 也就是 差不多 下图的180M, 一类对象就 180M, 总共才1G, 当然会不够用!!

 

其实 从错误日志也可以 分析出来一些, 在创建需要比较大的内存的对象的时候, 就会出现 oom, 因为内存确实已经不够了啊!! (这也是为什么 ELCommonCodeRegulationService 的 GetCode 方法调用的时候,出现了很多oom。 但是又不是绝对的。 因为其他 地方也可以创建大内存对象)

 

 

 

其实只要再多问几个问题就知道了答案:  为什么出现了这么多次, 占用这么多内存呢??  这个是正常的吗?

 

版权声明
本文为[CanntBelieve]所创,转载请带上原文链接,感谢
https://www.cnblogs.com/FlyAway2013/p/11051514.html