MergeTreeBlockSizePredictor源码解析
MergeTreeBlockSizePredictor源码解析
预计阅读时间: 11分钟
统计数据分布size，便于预测一些变量，如每行字节数，最大列每行字节数，filter比例等。方便计算出将要读取的最佳行数。
这个类目前版本看比较鸡肋，block_size_bytes，block_size_rows等变量在每次开始读取时重置为0，读取完成后又update这几个变量，看起来毫无意义。
三个在每次读取前有效参与到预测的参数：bytes_per_row_current，max_size_per_row_dynamic，filtered_rows_ratio。
这三个参数每次读取数据后，都会update对其进行更新，方便下次更好的预测。
每个列的 bytes_per_row ，在每次update时，根据本次局部的 local_bytes_per_row 和参数计算结果去更新，作为max_size_per_row_dynamic的依据。
double alpha = std::pow(1. - decay, diff_rows);
double local_bytes_per_row = static_cast<double>(diff_size) / diff_rows;
info.bytes_per_row = alpha * info.bytes_per_row + (1. - alpha) * local_bytes_per_row;
max_size_per_row_dynamic = std::max<double>(max_size_per_row_dynamic, info.bytes_per_row);bytes_per_row_current是每个预测完的 info.bytes_per_row 相加等到的，info.bytes_per_row可以看作是每列的数据，bytes_per_row_current是所有列的和
bytes_per_row_current += info.bytes_per_row;filtered_rows_ratio，根据每次读取的 current_ration 和参数计算结果更新filtered_rows_ratio
double alpha = std::pow(1. - decay, rows_was_read);
double current_ration = rows_was_filtered / std::max(1.0, static_cast<double>(rows_was_read));
filtered_rows_ratio = current_ration < filtered_rows_ratio
? current_ration
: alpha * filtered_rows_ratio + (1.0 - alpha) * current_ration;可以看到更新统计数据(data)的公式相同：
data = alpha * data + (1.0 - alpha) * current_data
那么核心在于 alpha，(1减去衰减因子) 的 (本次读取行数) 次方
double alpha = std::pow(1. - decay, diff_rows);衰减因子：
/// Aggressiveness of bytes_per_row updates. See update() implementation.
/// After n=NUM_UPDATES_TO_TARGET_WEIGHT updates v_{n} = (1 - TARGET_WEIGHT) * v_{0} + TARGET_WEIGHT * v_{target}
static constexpr double TARGET_WEIGHT = 0.5;
static constexpr size_t NUM_UPDATES_TO_TARGET_WEIGHT = 8192;
static double DECAY() { return 1. - std::pow(TARGET_WEIGHT, 1. / NUM_UPDATES_TO_TARGET_WEIGHT); } /// 0.000084609113387相关配置参数：
M(SettingUInt64, max_block_size, DEFAULT_BLOCK_SIZE, "Maximum block size for reading") \
M(SettingUInt64, preferred_block_size_bytes, 1000000, "") \
M(SettingUInt64, preferred_max_column_in_block_size_bytes, 0, "Limit on max column size in block while reading. Helps to decrease cache misses count. Should be close to L2 cache size.") \/** Which blocks by default read the data (by number of rows).
* Smaller values give better cache locality, less consumption of RAM, but more overhead to process the query.
*/
#define DEFAULT_BLOCK_SIZE 65536max_block_size ：默认最大block 行数 65536 行
preferred_block_size_bytes：最佳的block字节数 1MB
preferred_max_column_in_block_size_bytes：block中最大列最佳字节数
bytes_per_row_current，max_size_per_row_dynamic，filtered_rows_ratio
bytes_per_row_current 在设置了 preferred_block_size_bytes 参数时起作用
max_size_per_row_dynamic，filtered_rows_ratio 都是在设置了 preferred_max_column_in_block_size_bytes 参数时起作用
auto estimateNumRows = [current_preferred_block_size_bytes, current_max_block_size_rows,
&index_granularity, current_preferred_max_column_in_block_size_bytes, min_filtration_ratio](
MergeTreeReadTask & current_task, MergeTreeRangeReader & current_reader)
{
if (!current_task.size_predictor)
return static_cast<size_t>(current_max_block_size_rows);
/// Calculates number of rows will be read using preferred_block_size_bytes.
/// Can't be less than avg_index_granularity.
size_t rows_to_read = current_task.size_predictor->estimateNumRows(current_preferred_block_size_bytes); /// bytes_per_row_current 起作用
if (!rows_to_read)
return rows_to_read;
auto total_row_in_current_granule = current_reader.numRowsInCurrentGranule();
rows_to_read = std::max(total_row_in_current_granule, rows_to_read);
if (current_preferred_max_column_in_block_size_bytes) /// max_size_per_row_dynamic，filtered_rows_ratio 起作用
{
/// Calculates number of rows will be read using preferred_max_column_in_block_size_bytes.
auto rows_to_read_for_max_size_column
= current_task.size_predictor->estimateNumRowsForMaxSizeColumn(current_preferred_max_column_in_block_size_bytes);
double filtration_ratio = std::max(min_filtration_ratio, 1.0 - current_task.size_predictor->filtered_rows_ratio);
auto rows_to_read_for_max_size_column_with_filtration
= static_cast<size_t>(rows_to_read_for_max_size_column / filtration_ratio);
/// If preferred_max_column_in_block_size_bytes is used, number of rows to read can be less than current_index_granularity.
rows_to_read = std::min(rows_to_read, rows_to_read_for_max_size_column_with_filtration);
}
auto unread_rows_in_current_granule = current_reader.numPendingRowsInCurrentGranule();
if (unread_rows_in_current_granule >= rows_to_read)
return rows_to_read;
return index_granularity.countMarksForRows(current_reader.currentMark(), rows_to_read, current_reader.numReadRowsInCurrentGranule());
};End
本文分享自 MoziInnovations 微信公众号，前往查看
如有侵权，请联系 cloudcommunity@tencent.com 删除。
本文参与 腾讯云自媒体分享计划 ，欢迎热爱写作的你一起参与！