HBase NotServingRegionException

sparkle123

发布于 2020-09-28 15:46:06

6840

发布于 2020-09-28 15:46:06

文章被收录于专栏：大数据-Hadoop、Spark大数据-Hadoop、Spark

线上最近通过日志系统发现某BU的服务里HBase某时间段内有几十次如下这样的报错，error发生在同一个表的查询上，而业务对该表的查询仅限于点查询：

根因分析：其实是因为region split造成的抖动，trip:rt_vehicle_history_5m这张表最近一些region到10G大小开始split，相当于那region 短暂close了然后再访问它就报错。解决方案： hbase.client.retries.number 稍微调整大些，默认是3-->10, pause time 100ms~1s左右

split所需时间与表的文件数有关。

附上日志

Fri Sep 25 08:38:03 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Fri Sep 25 08:38:04 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
atorg.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

如有侵权请联系 cloudcommunity@tencent.com 删除

javascript