前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >BE节点经常挂掉:[IO_ERROR]failed to list /proc/27349/fd/: No such file or directory

BE节点经常挂掉:[IO_ERROR]failed to list /proc/27349/fd/: No such file or directory

作者头像
程裕强
发布2023-11-22 11:03:37
3320
发布2023-11-22 11:03:37
举报

最近BE节点经常挂掉

代码语言:javascript
复制
Caused by: java.lang.RuntimeException: Failed to execute internal SQL. org.apache.doris.common.UserException: errCode = 2, detailMessage = There is no scanNode Backend available.[10031: not alive] OriginStatement{originStmt='SELECT * FROM __internal_schema.column_statistics WHERE tbl_id=27273 AND idx_id=-1 AND col_id='CREATE_AID'', idx=0}
        at org.apache.doris.qe.StmtExecutor.executeInternalQuery(StmtExecutor.java:2509)
        at org.apache.doris.statistics.util.StatisticsUtil.execStatisticQuery(StatisticsUtil.java:131)
        at org.apache.doris.statistics.StatisticsRepository.loadColStats(StatisticsRepository.java:439)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.loadFromStatsTable(ColumnStatisticsCacheLoader.java:56)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.doLoad(ColumnStatisticsCacheLoader.java:38)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.doLoad(ColumnStatisticsCacheLoader.java:31)
        at org.apache.doris.statistics.StatisticsCacheLoader.lambda$asyncLoad$0(StatisticsCacheLoader.java:48)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        ... 3 more
Caused by: org.apache.doris.common.UserException: errCode = 2, detailMessage = There is no scanNode Backend available.[10031: not alive]
        at org.apache.doris.qe.SimpleScheduler.getHost(SimpleScheduler.java:147)
        at org.apache.doris.qe.Coordinator.computeFragmentHosts(Coordinator.java:1806)
        at org.apache.doris.qe.Coordinator.computeFragmentExecParams(Coordinator.java:1267)
        at org.apache.doris.qe.Coordinator.exec(Coordinator.java:573)
        at org.apache.doris.qe.StmtExecutor.executeInternalQuery(StmtExecutor.java:2505)
        ... 10 more

be.out也看不出什么有用日志,查看be.WARNING,发现了如下错误,但还不知道如何解决,先记录一下问题

[IO_ERROR]failed to list /proc/27349/fd/: (2), No such file or directory

代码语言:javascript
复制
W1121 09:36:26.929662 27477 doris_metrics.cpp:379] failed to count fd: [IO_ERROR]failed to list /proc/27349/fd/: (2), No such file or directory
0. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000b9e64c7 in /xxsys/doris-2.0.2/be/lib/doris_be
1. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000b9e4ae5 in /xxsys/doris-2.0.2/be/lib/doris_be
2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Status doris::Status::Error<true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000aecc168 in /xxsys/doris-2.0.2/be/lib/doris_be
3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::io::LocalFileSystem::list_impl(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<doris::io::FileInfo> >*, bool*) @ 0x000000000aec6eac in /xxsys/doris-2.0.2/be/lib/doris_be
4. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::io::LocalFileSystem::iterate_directory_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (doris::io::FileInfo const&)> const&) @ 0x000000000aec7fcf in /xxsys/doris-2.0.2/be/lib/doris_be
5. /root/src/doris-2.0/be/src/common/status.h:348: doris::io::LocalFileSystem::iterate_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (doris::io::FileInfo const&)> const&) @ 0x000000000aec7e4d in /xxsys/doris-2.0.2/be/lib/doris_be
6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::DorisMetrics::_update_process_fd_num() @ 0x000000000b97a65a in /xxsys/doris-2.0.2/be/lib/doris_be
7. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:368: doris::MetricRegistry::trigger_all_hooks(bool) const @ 0x000000000b9ba69f in /xxsys/doris-2.0.2/be/lib/doris_be
8. /root/src/doris-2.0/be/src/util/time.h:50: doris::Daemon::calculate_metrics_thread() @ 0x000000000ae9cc0c in /xxsys/doris-2.0.2/be/lib/doris_be
9. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba1819a in /xxsys/doris-2.0.2/be/lib/doris_be
10. start_thread @ 0x00007f2f98172aa1 in ?
11. __clone @ 0x00007f2f988f8c4d in ?
本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2023-11-21,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档