V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
jiobanma
V2EX  ›  Java

Java 程序内存不足问题求解

  •  
  •   jiobanma ·
    banmajio · 2022-10-09 11:12:56 +08:00 · 4141 次点击
    这是一个创建于 536 天前的主题,其中的信息可能已经有所发展或是发生改变。

    1.问题描述

    1. 服务忽然宕机,查看服务器发现生成了很多 hs_err_pidxxxxxx.log 这样的 log 文件。下面我会提出来文件的一些内容。
    2. 查看服务的日志文件有如下错误提示
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /neworiental/software/bootstrap/hs_err_pid17840.log
    Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006c0000000, 2863661056, 0) failed; error='Cannot allocate memory' (errno=12)
    
    1. JVM 参数
    -Xms4G -Xmx4G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/xxx/software/bootstrap/
    

    2. 问题

    1. 简单的查了查,对 jvm 不是特别的熟悉。不太清楚这个问题是 jvm 分配的内存有点小,服务内存溢出导致的,还是服务器内存资源不够了,导致服务产生异常。
    2. 我加了 HeapDumpOnOutOfMemoryError 这个参数,如果是服务运行期间内存溢出,为什么没有产生 dump 文件。
    3. 希望有了解这块内容的大佬帮忙看看,提供些思路感谢。

    3. hs_err_pidxxxxxx.log 文件

    Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)
    
    vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
    
    time: Sun Oct  9 00:28:39 2022
    elapsed time: 0 seconds (0d 0h 0m 0s)
    
    ^C
    [dontovertime@HYY-BMPP-P-sql96d40-09 bootstrap]$ tail -10000f hs_err_pid18191.log
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
    # Possible reasons:
    #   The system is out of physical RAM or swap space
    #   In 32 bit mode, the process size limit was hit
    # Possible solutions:
    #   Reduce memory load on the system
    #   Increase physical memory or swap space
    #   Check if swap backing store is full
    #   Use 64 bit Java on a 64 bit OS
    #   Decrease Java heap size (-Xmx/-Xms)
    #   Decrease number of Java threads
    #   Decrease Java thread stack sizes (-Xss)
    #   Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    #  Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
    #
    # JRE version:  (8.0_172-b11) (build )
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    
    ---------------  T H R E A D  ---------------
    
    Current thread (0x00007f6c0c00a800):  JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]
    
    Stack: [0x00007f6c15b71000,0x00007f6c15c71000],  sp=0x00007f6c15c6d2b0,  free space=1008k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0xad221a]  VMError::report_and_die()+0x2ba
    V  [libjvm.so+0x50185b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
    V  [libjvm.so+0x92fee3]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123
    V  [libjvm.so+0x930109]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    V  [libjvm.so+0x9273ca]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    V  [libjvm.so+0x99d363]  PSVirtualSpace::expand_by(unsigned long)+0x53
    V  [libjvm.so+0x98d477]  PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0xb7
    V  [libjvm.so+0x2dbb8a]  AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x39a
    V  [libjvm.so+0x9514a6]  ParallelScavengeHeap::initialize()+0x1d6
    V  [libjvm.so+0xa9a253]  Universe::initialize_heap()+0xf3
    V  [libjvm.so+0xa9a7be]  universe_init()+0x3e
    V  [libjvm.so+0x643de5]  init_globals()+0x65
    V  [libjvm.so+0xa7ef7e]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
    V  [libjvm.so+0x6d8b34]  JNI_CreateJavaVM+0x74
    C  [libagentloader.so+0x1146d]  JNI_CreateJavaVM+0x8ed
    
    ---------------  P R O C E S S  ---------------
    
    Java Threads: ( => current thread )
    
    Other Threads:
    
    =>0x00007f6c0c00a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)]
    
    VM state:not at safepoint (not fully initialized)
    
    VM Mutex/Monitor currently owned by a thread: None
    
    GC Heap History (0 events):
    No events
    
    Deoptimization events (0 events):
    No events
    
    Classes redefined (0 events):
    No events
    
    Internal exceptions (0 events):
    No events
    
    Events (0 events):
    No events
    
    
    ---------------  S Y S T E M  ---------------
    
    OS:CentOS Linux release 7.7.1908 (Core)
    
    uname:Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64
    libc:glibc 2.17 NPTL 2.17
    rlimit: STACK 8192k, CORE 0k, NPROC 63433, NOFILE 102536, AS infinity
    load average:3.00 3.58 2.35
    
    /proc/meminfo:
    MemTotal:       16260372 kB
    MemFree:          425348 kB
    MemAvailable:     367844 kB
    Buffers:               0 kB
    Cached:           235068 kB
    SwapCached:       148856 kB
    Active:         13576768 kB
    Inactive:        1854056 kB
    Active(anon):   13530096 kB
    Inactive(anon):  1771152 kB
    Active(file):      46672 kB
    Inactive(file):    82904 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:       4194300 kB
    SwapFree:         874708 kB
    Dirty:               488 kB
    Writeback:             0 kB
    AnonPages:      15048848 kB
    Mapped:            49124 kB
    Shmem:            105552 kB
    Slab:             162980 kB
    SReclaimable:      94124 kB
    SUnreclaim:        68856 kB
    KernelStack:       27168 kB
    PageTables:        65984 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:    12324484 kB
    Committed_AS:   33627812 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:      195136 kB
    VmallocChunk:   34359326716 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:   5726208 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:       87936 kB
    DirectMap2M:     5154816 kB
    DirectMap1G:    13631488 kB
    
    
    CPU:total 8 (initial active 8) (1 cores per cpu, 1 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, 3dnowpref, tsc, tscinvbit
    
    /proc/cpuinfo:
    processor	: 0
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 0
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 0
    initial apicid	: 0
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 1
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 2
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 2
    initial apicid	: 2
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 2
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 4
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 4
    initial apicid	: 4
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 3
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 6
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 6
    initial apicid	: 6
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 4
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 8
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 8
    initial apicid	: 8
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 5
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 10
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 10
    initial apicid	: 10
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 6
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 12
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 12
    initial apicid	: 12
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    processor	: 7
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 79
    model name	: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    stepping	: 1
    microcode	: 0xb00001b
    cpu MHz		: 2199.038
    cache size	: 30720 KB
    physical id	: 14
    siblings	: 1
    core id		: 0
    cpu cores	: 1
    apicid		: 14
    initial apicid	: 14
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 20
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep
    bogomips	: 4399.99
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 40 bits physical, 48 bits virtual
    power management:
    
    
    
    Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)
    
    vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
    
    time: Sun Oct  9 00:28:39 2022
    elapsed time: 0 seconds (0d 0h 0m 0s)
    
    第 1 条附言  ·  2022-10-09 14:50:23 +08:00
    好奇 log 里面最下面有一句这个:
    ```
    emory: 4k page, physical 16260372k(1798160k free), swap 4194300k(1552k free)
    ```
    这句话代表什么呢。swap 的空闲好像很小
    35 条回复    2022-10-10 18:39:12 +08:00
    ShuA1
        1
    ShuA1  
       2022-10-09 11:15:13 +08:00
    看上去就是 OOM 啊
    OctopusGO
        2
    OctopusGO  
       2022-10-09 11:21:40 +08:00
    调整一下 jvm 每个线程的初始化的时候申请的空间大小再看看
    OctopusGO
        3
    OctopusGO  
       2022-10-09 11:24:36 +08:00
    一般就是代码里面有执行慢的地方,然乎这个地方调用的多,线程太多 OOM 了
    WindWarrior
        4
    WindWarrior  
       2022-10-09 11:28:41 +08:00
    看看如果是 4GB 的服务器 -Xmx 是不能开到 4GB 的 大概开到 70%~ 80%
    facelezz
        5
    facelezz  
       2022-10-09 11:35:22 +08:00
    "Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free)"
    剩余 425M
    "Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory."
    申请 2863M

    这不是很明显 宿主机内存不够么
    larisboy
        6
    larisboy  
       2022-10-09 11:37:22 +08:00
    要不调低程序的所需内存,要不给宿主机加资源
    jiobanma
        7
    jiobanma  
    OP
       2022-10-09 11:38:22 +08:00
    @WindWarrior 服务器是 16g 的
    @facelezz 但是服务器是 16g 的 发生这个原因的时候 重启了一下 但是重启前没有查看内存占用情况,现在 free -m 看的话 内存还剩 10 个 g
    chainsR
        8
    chainsR  
       2022-10-09 11:57:36 +08:00 via iPhone
    我最近有个项目也遇到过,java 8 是么,可以试试把这些都加上去,就好了 https://www.cnblogs.com/lovelyli/p/14955585.html
    asiufasd
        9
    asiufasd  
       2022-10-09 12:03:40 +08:00   ❤️ 1
    @jiobanma
    "我加了 HeapDumpOnOutOfMemoryError 这个参数,如果是服务运行期间内存溢出,为什么没有产生 dump 文件。"
    因为可能没有发生 OutOfMemoryError ,而是操作系统级别的内存不够用了,可以检查一下是否有别的应用占用了系统的资源
    littlewing
        10
    littlewing  
       2022-10-09 12:09:43 +08:00   ❤️ 1
    HeapDumpOnOutOfMemoryError 产生 dump 文件的条件是你的程序被 JVM OOM 了,但你这个是被系统把 JVM OOM 了
    Red998
        11
    Red998  
       2022-10-09 12:14:23 +08:00
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
    # Possible reasons:
    # The system is out of physical RAM or swap space
    # In 32 bit mode, the process size limit was hit
    # Possible solutions:
    # Reduce memory load on the system
    # Increase physical memory or swap space
    # Check if swap backing store is full
    # Use 64 bit Java on a 64 bit OS
    # Decrease Java heap size (-Xmx/-Xms)
    # Decrease number of Java threads
    # Decrease Java thread stack sizes (-Xss)
    # Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    # Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
    #
    # JRE version: (8.0_172-b11) (build )
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again


    Java 运行时环境没有足够的内存来继续。
    # 本机内存分配 (mmap) 未能映射 2863661056 字节以提交保留内存。
    # 可能的原因:
    # 系统的物理 RAM 或交换空间不足
    # 在 32 位模式下,达到了进程大小限制
    # 可能的解决方案:
    # 减少系统内存负载
    # 增加物理内存或交换空间
    # 检查交换后备存储是否已满
    # 在 64 位操作系统上使用 64 位 Java
    # 减小 Java 堆大小 (-Xmx/-Xms)
    # 减少 Java 线程数
    # 减少 Java 线程堆栈大小 (-Xss)
    # 使用 -XX:ReservedCodeCacheSize= 设置更大的代码缓存
    Jooooooooo
        12
    Jooooooooo  
       2022-10-09 13:33:25 +08:00
    就是内存不够了.

    简单解决就是先把堆调大, 然后 dump 堆看看里面都是啥, 有没有可以优化的.
    jiobanma
        13
    jiobanma  
    OP
       2022-10-09 13:36:14 +08:00
    @asiufasd
    @littlewing
    明白了,之前没有遇到过服务器内存满的情况。
    @chainsR 是 java8 都加上有点离谱 线上不敢都加上去试 搞出点别的问题就 gg 了
    jiobanma
        14
    jiobanma  
    OP
       2022-10-09 13:47:58 +08:00
    @Jooooooooo 你的意思是继续加大-Xmx 吗? 现在已经是 4g 了,而且按照 9,10 楼大佬所说的,是服务器内存满了,我继续加大 jvm 的内存依然会出现这个问题的。并且现在没有产生出 dump 文件,所以才会疑惑是操作系统的问题还是代码的问题
    lmshl
        15
    lmshl  
       2022-10-09 13:57:12 +08:00
    既然已经 dump heap 了,直接丢进 eclipse memory analyzer 看原因咯
    facelezz
        16
    facelezz  
       2022-10-09 13:59:49 +08:00
    @jiobanma 基本的宿主机监控还是要有的 不过你这个显然除了 Java 还有其他进程在跑 吃掉了内存
    MineDog
        17
    MineDog  
       2022-10-09 14:05:30 +08:00   ❤️ 1
    没 dump 的原因不是都写了么,"Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again"
    wiix
        18
    wiix  
       2022-10-09 14:10:55 +08:00
    线程堆栈占的内存和 Xmx 设置的堆内存是分开的,默认每个线程的开销大概是 1M 。是不是什么地方的线程数开太大了?
    MineDog
        19
    MineDog  
       2022-10-09 14:17:21 +08:00
    还有就是用了 mmap 直接内存分配,是程序什么功能需要用到呢,这玩意也不是在堆里面分配的啊
    cloud107202
        20
    cloud107202  
       2022-10-09 14:34:57 +08:00
    程序运行在容器里么?有无设置容器内存 limit
    jiobanma
        21
    jiobanma  
    OP
       2022-10-09 14:36:30 +08:00
    @cloud107202 在服务器里的,没有在容器里跑
    @MineDog mmap 啊这 这个不懂诶,没接触过; ulimit -c unlimited 这个我试一下
    cloud107202
        22
    cloud107202  
       2022-10-09 14:59:09 +08:00
    看样子就是泄露了,可能是托管堆内也可能是线程。找机会用 jmap 主动 dump 内存,最好是做好监控的前提下,在 process memory 跳涨后去 dump, 然后扔 https://visualvm.github.io/ 里分析
    mengyxu
        23
    mengyxu  
       2022-10-09 15:26:22 +08:00
    宿主级内存不够,调小 Xms 和 Xmx 参数,如果你程序真的需要分配这么多内存的话建议宿主机少跑点进程
    urnoob
        24
    urnoob  
       2022-10-09 17:20:24 +08:00
    mmap 2G 内存,有可能是哪个库或者自己的代码加载大文件或者写文件如 log 。jvm 需要扩大 heap 不是用 mmap 。
    jvm 设的内存太大了,导致可用堆外内存不足。 如果这个应用的 jvm 确实需要那么多,那 lz 就加内存。
    剩下就是内存泄露问题。
    imzcg2
        25
    imzcg2  
       2022-10-09 18:09:47 +08:00
    Possible reasons:
    # The system is out of physical RAM or swap space
    # In 32 bit mode, the process size limit was hit

    看见这个了吗???? 你这分明是在 64 位主机上运行了 32 位的 jdk 啊! 赶紧回去在主机上运行 java --version 看是不是 32 位的 jdk,是就换 64 位
    imzcg2
        26
    imzcg2  
       2022-10-09 18:12:37 +08:00
    抱歉没看完,是 64 位的
    chainsR
        27
    chainsR  
       2022-10-09 20:35:48 +08:00 via iPhone
    @jiobanma 不过 8 的垃圾回收确实得指定好,我的是跑着跑着把内存跑满了,把这些加上去后一切正常
    LeegoYih
        28
    LeegoYih  
       2022-10-10 00:26:26 +08:00
    把初始堆内存大小和最大内存大小设置成一样的值,这样启动时内存不足报错就说明宿主机内存确实是不够,免得 JVM 后续再去申请内存时才暴露问题。

    -Xms4g
    -Xmx4g
    nothingistrue
        29
    nothingistrue  
       2022-10-10 10:11:26 +08:00
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
    # Possible reasons:
    # The system is out of physical RAM or swap space
    # In 32 bit mode, the process size limit was hit
    # Possible solutions:
    # Reduce memory load on the system
    # Increase physical memory or swap space
    # Check if swap backing store is full
    # Use 64 bit Java on a 64 bit OS
    # Decrease Java heap size (-Xmx/-Xms)
    # Decrease number of Java threads
    # Decrease Java thread stack sizes (-Xss)
    # Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    # Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700
    #
    # JRE version: (8.0_172-b11) (build )
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

    这一段已经说明错误现象,可能的原因,和大致的解决方案了。

    现象:JVM 向基本系统申请内存的时候失败了,2863661056 bytes 大概 356 MB 。
    这里申请的不是 -Xms -Xmx 指定的 JVM 堆内存,而是 JVM 需要占用的其他内存,JVM 堆内存在 JVM 启动的时候就独占了,后面不会再申请。Java 实例化后的对象(实际上是对象的成员变量),方法运行期间的变量,这俩是放到 JVM 堆内存的。但是 Java 类定义本身(类结构、类当中方法的函数定义、类的静态变量这些),是放到额外内存中的。此外 JVM 本身也会占用内存。后两个内存是动态的,现在要申请的可能就是这俩内存,然后因超出物理机或者进程内存上限而失败了。

    原因就两个:一,物理机内存,包括实际物理内存和交换区( Windows 下就是虚拟内存),都满了;二,32 位程序,单进程内存通常上限是 4G 。
    nothingistrue
        30
    nothingistrue  
       2022-10-10 10:20:38 +08:00
    上面 JVM 堆内存描述有误,应该时:启动时只占用 -Xms 指定的大小,后面随需要扩大占用,直到 -Xmx 指定的大小。
    yisier
        31
    yisier  
       2022-10-10 10:21:01 +08:00
    加个定时任务,每隔一段时间查询一下线程总数,如果线程数越来越多,那就是内存泄露了


    ThreadMXBean bean = ManagementFactory.getThreadMXBean();
    log.info("线程总数为 = " + bean.getThreadCount());
    jiobanma
        32
    jiobanma  
    OP
       2022-10-10 13:58:03 +08:00
    @nothingistrue
    @yisier
    感谢解惑
    cloud107202
        33
    cloud107202  
       2022-10-10 15:08:44 +08:00
    Java HotSpot(TM) 64-Bit Server VM warning: 已经说明是 64 位 JVM
    32bit 那个提示是在 Possible reasons 这一段下面,只是个内置的 hint 不要搞偏方向
    pierswu
        34
    pierswu  
       2022-10-10 18:38:24 +08:00
    5 楼、29 楼才是正解!(虽然 29 楼算错了 2863661056 bytes 是 1731 MB

    就是你服务器的剩余内存不够了
    pierswu
        35
    pierswu  
       2022-10-10 18:39:12 +08:00
    我也算错了 2863661056 bytes 是 2731 MB
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   5530 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 1705ms · UTC 08:58 · PVG 16:58 · LAX 01:58 · JFK 04:58
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.