注,本次演示使用的集群为腾讯云EMR-V3.4.0,HUE版本为4.10.0。
查阅HUE代码,发现hive链接的客户端都是在本地生产的,也就是在 hiveserver2.py 中来完成初始化的。该文件在EMR集群上的路径为 /usr/local/service/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py ,其中 _prepare_hql_query 函数来完成初始化。所以只要将相关参数设置在 _prepare_hql_query 函数中就行。
def _prepare_hql_query(self, snippet, statement, session):
settings = snippet['properties'].get('settings', None)
file_resources = snippet['properties'].get('files', None)
functions = snippet['properties'].get('functions', None)
properties = session['properties'] if session else []
# Get properties from session if not defined in snippet
if not settings:
settings = next((prop['value'] for prop in properties if prop['key'] == 'settings'), None)
if not file_resources:
file_resources = next((prop['value'] for prop in properties if prop['key'] == 'files'), None)
if not functions:
functions = next((prop['value'] for prop in properties if prop['key'] == 'functions'), None)
database = snippet.get('database') or 'default'
return hql_query(
statement,
query_type=QUERY_TYPES[0],
settings=settings,
file_resources=file_resources,
functions=functions,
database=database
)
阅读 hiveserver2.py 发现除了HIVE的初始化,impala的初始化也在 _prepare_hql_query 函数中。故在给HIVE添加参数的时候,需要指定给HIVE设置,否则会导致impala无法初始化。
本次需求说明:在使用HUE的时候,用户提交的HIVE hql任务,默认执行引擎为mr;并且在指定的YANR队列中运行,避免抢占资源影响其他任务。
当前EMR集群已经配置为公平调度,并为hue用户设置了专用的队列“hue_pool”。
需要添加代码:
if snippet['type'] == 'hive':
snippet['properties']['settings'].append({"key": "hive.execution.engine", "value": "mr"})
snippet['properties']['settings'].append({"key": "mapreduce.job.queuename","value":"root.hue_pool"})
满足本次需求的改造函数完成如下:
def _prepare_hql_query(self, snippet, statement, session):
if snippet['type'] == 'hive':
snippet['properties']['settings'].append({"key": "hive.execution.engine", "value": "mr"})
snippet['properties']['settings'].append({"key": "mapreduce.job.queuename","value":"root.hue_pool"})
settings = snippet['properties'].get('settings', None)
file_resources = snippet['properties'].get('files', None)
functions = snippet['properties'].get('functions', None)
properties = session['properties'] if session else []
# Get properties from session if not defined in snippet
if not settings:
settings = next((prop['value'] for prop in properties if prop['key'] == 'settings'), None)
if not file_resources:
file_resources = next((prop['value'] for prop in properties if prop['key'] == 'files'), None)
if not functions:
functions = next((prop['value'] for prop in properties if prop['key'] == 'functions'), None)
database = snippet.get('database') or 'default'
return hql_query(
statement,
query_type=QUERY_TYPES[0],
settings=settings,
file_resources=file_resources,
functions=functions,
database=database
)
在EMR集群的master节点上进行修改,修改完成后在控制台重启HUE服务即可。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。