Impala 内存

Impala 内存

按照底层分类

1. JVM Memory
2. C++ TcMalloc Memory
3. code/stack/.. Untracked Memory

按照对象分类

1. Admission_control 单个Cluster/Group 整体资源
2. Jvm ...
3. BufferPool
4. ReservationTracker: BufferPool内存限制 , ExecNode 执行节点在申请内存时,会先在ReservationTracker进行逻辑上的内存社情
5. MemTracker 内存统计

Impala-内存限制

Impala-内存限制

BufferPool

BufferPool

BufferPool

Arena 别名参考LevelDB设计, BufferPool 我理解主要是进行按照不同Page大小,缓存不同PerSizeList连续空间..

Page: Logical

Buffer: Phyical

Client(PageHandle) ->(page.buffer)

Page(Pin count ) (Pinned dirty clean)

Unpinned(->splill )

Pinned: 
///   -> When pin_in_flight=false, the buffer contains the page's data and the client can
///      read and write to the buffer.
///   -> When pin_in_flight=true, the page's data is in the process of being read from
///      scratch disk into the buffer. Clients will block on the read I/O if they attempt
///      to access the buffer.
Pinned                        pinned_pages_
Unpinned - Dirty:             dirty_unpinned_pages_ 
Unpinned - Write in flight:   write_in_flight_pages_

Unpinned - Clean:  When the write to scratch has completed but the page was not evicted. The page is in a clean pages list in a BufferAllocator arena.
Unpinned - Evicted: After a clean page's buffer has been reclaimed. The page is not in any list.

复制

ReservationTracker/MemTracker

*Tracker

*Tracker

MemTracker的主要作用是跟踪内存的使用情况

ReservationTracker的主要作用上是从逻辑上控制Impala的资源使用, 代码逻辑上跟MemTracker 是大同小异的

从Process(进程)-Query(查询级别)-FragmentInstance(执行碎片级别)-ExecNode(算子级别) 以一棵树的形态进行内存管理, 在当前子节点内存不同的情况下, 会想父节点申请内存

MemTracker

MemTracker

ReservertionTracker

ReservertionTracker

内存申请过程

由BufferPool Client 申请 主要步骤为

1. client Prepare-> ReservationTracker ->MemTracker 进行资源预申请

2 client Allocate -> BufferAllocator -> (System/Arena->PerSizeList->FreeList) 做实际申请

BufferPool Allocate

BufferPool Allocate

步骤一

步骤一

代办

1 一个Impala SQL 的内存 申请过程