# Investigating Out of Memory crashes
A large fraction of process crashes in Chromium are due to Out Of Memory (OOM)
conditions. This page is meant to help Chromium developers understand stack
traces, and investigate. Note that some of the documentation here will only be
applicable to Google Chrome, as it is specific to the way Google's crash
reporting infrastructure aggregates and reports crashes.
Some of the following also assumes that the `malloc()` implementation is
PartitionAlloc, which is as of 2022 the case on most platforms.
[TOC]
## Identifying OOM crashes
When a process crashes due to an Out Of Memory condition, this is usually
signaled by the presence of `base::internal::OnNoMemoryInternal()` on the stack.
**Google Chrome only:** crash report infrastructure tags these as "[Out of
Memory]" based on this, and other function names. The full list is determined in
the (internal) crash server's code.
Since Chromium configures its memory allocators to prefer crashing rather than
returning `nullptr`, an OOM crash can be triggered from anywhere in the code,
and most commonly from within the allocator, or higher-level functions such as
`operator new` in C++.
## Distinguishing between underlying causes
### Different causes
A process can reach an OOM condition for several reasons:
* **The OS is truly out of memory**, regardless of how much memory the *current*
process is using
* **Some limit inside the OS is reached**. For instance, on Windows, there
exists a global "commit limit", which is the amount of memory that the system
can commit. Note that it is possible to commit more memory than what is
actually in use. This may also happen on Linux systems configured with no or
limited "overcommit", though the majority of systems don't have a limit.
* **Virtual address space exhaustion**. This is most likely to happen for relatively
large allocations, on 32 bit systems, where total addressable space is
typically 2GiB (most Windows systems), 3GiB (e.g. some Windows configurations,
Linux) or 4GiB (e.g. WoW64). However, it may also happen on 64 bit systems,
either due to:
* Limited virtual addressable space in the CPU/OS. For instance most Android
ARM64 systems have only 40 bits of address space as of 2022.
* "Cage" exhaustion. This is most likely to happen with PartitionAlloc on 64
bit systems, where all allocations are grouped into a single contiguous
virtual address space "cage".
* **Sandbox per-process memory limit**. For some process types (e.g. Renderers)
and on most platforms, the sandbox enforces a maximum per-process memory
limit. Given that this limit is typically set at the OS level, it may not be
distinguishable from e.g. commit limit exhaustion.
* **Excessive allocation size**. Some allocators (notably PartitionAlloc)
purposely limit the maximum allocation size.
### Identifying the cause
In the case of PartitionAlloc, it is possible to distinguish some of these cases:
* **Virtual address space exhaustion**. This is identified by the presence of
`PartitionOutOfMemoryMappingFailure()` on the stack. It means that the
allocator was unable to find enough address space, either for its internal
memory allocation unit size, or the requested size. Since memory is *not*
committed as this step, this signals an address space issue.
* **Commit**. This is identified by the presence of
`PartitionOutOfMemoryCommitFailure()` on the stack. This signals that either
the OS or the sandbox limit has been reached.
* **Excessive allocation size**. Shown by `PartitionExcessiveAllocationSize()`
on the stack.
## What to do?
### Commit Limit Reached
The process is "truly" out of memory, or the system is. Some amount of these
crashes is expected, and the crashing location is not necessarily the
culprit. Indeed, as a rough approximation, the failing allocation is more likely
to be from a component naturally allocating a lot of memory, e.g. V8 or
rendering.
However, if there is a spike, and many stack traces come from an unusual
location (e.g. newly added code), this may signal a memory leak in the component
on the stack, or excessive temporary allocations.
Also, if `PartitionAllocDirectMap()` is on the stack, the memory allocation was
large. It may come from a large buffer, and potentially made worse by buffer
resizing. For instance, `std::vector` often double their size when out of
capacity. In which case, `reserve()`-ing the right size ahead of time may help.
### Excessive allocation size
Is the calling code expected to allocate more than 2GiB? Or it is an underflow
somewhere in the calling code?
### Virtual address space
On 32 bit systems, this is most likely to occur when overall memory usage is
high, or when the allocation size request is large. Is the calling code
allocating a very large buffer?
## Debugging
### General
On Windows, the allocation size is added into the exception record. In Google
Chrome's crash dashboard, this is shown in "Parameter[0]" of the exception
info. On other operating systems, the allocation size if put on the stack before
crashing, and thus visible in minidumps.
### PartitionAlloc and Google specific
1. Starting from a specific report, click on the bug icon to start a cloud lldb
instance
2. Locate the `PartitionRoot<true>::OutOfMemory()` frame on the stack, move to it with `f 5`
3. Locate the stack addresses by printing registers `re re`
4. Show the stack content with `x <stack_pointer> <frame pointer>`
Below is an example for a crash on x86_64:
```
( lizeb ) bt
* thread #1, stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x10c45912f)
* frame #0: 0x000000010c45912f Google Chrome Framework`base::internal::OnNoMemoryInternal(unsigned long) at memory.cc:62
frame #1: 0x000000010c459149 Google Chrome Framework`base::TerminateBecauseOutOfMemory(unsigned long) at memory.cc:69
frame #2: 0x000000010c4f39c6 Google Chrome Framework`OnNoMemory(unsigned long) at oom.cc:17
frame #3: 0x000000010d7e5794 Google Chrome Framework`WTF::PartitionsOutOfMemoryUsing2G(unsigned long) at partitions.cc:281
frame #4: 0x000000010d7e4d2c Google Chrome Framework`WTF::Partitions::HandleOutOfMemory(unsigned long) at partitions.cc:415
frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
[...]
( lizeb ) f 5
frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
( lizeb ) re re
General Purpose Registers:
rbp = 0x00007ffee7012c50
rsp = 0x00007ffee7012bf0
rip = 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) + 196 at partition_root.cc:522
21 registers were unavailable.
( lizeb ) x 0x00007ffee7012bf0 0x00007ffee7012c50
0x7ffee7012bf0: 76 61 5f 73 69 7a 65 00 00 00 00 07 00 00 00 00 va_size.........
0x7ffee7012c00: 61 6c 6c 6f 63 00 20 20 00 2d 2d 01 00 00 00 00 alloc. .--.....
0x7ffee7012c10: 63 6f 6d 6d 69 74 00 20 00 a0 9d 01 00 00 00 00 commit. ........
0x7ffee7012c20: 73 69 7a 65 00 20 20 20 00 00 20 00 00 00 00 00 size. .. .....
0x7ffee7012c30: aa aa aa aa aa aa aa aa 00 18 b0 12 01 00 00 00 ................
0x7ffee7012c40: 00 00 20 00 00 00 00 00 48 22 b0 12 01 00 00 00 .. .....H"......
```
The results here can help the PartitionAlloc team to identify issues, as
important metrics from PartitionAlloc are saved above. For instance virtual
address space usage is (in little endian) 0x70000000.
Codebase Browser
chromium