Kibana dev工具和elasticsearch-py客户端得到不同的输出
Kibana dev工具和elasticsearch-py客户端得到不同的输出
提问于 2018-08-31 02:55:18
我有一个ELK部署来收集日志。现在，我需要拉出包含一个特定字符串的所有日志。但是我得到了一个有趣的问题，我在Kibana的dev工具和elasticsearch python客户端得到了不同的输出。
以下是Kibana中的查询：
GET app_web_log-20180827/_search
{
"query": {
"bool": {
"must": [
{ "match_phrase": { "message": "Failed to call Billing API Server" }}
],
"filter": [
{ "term": { "deployment": "app_instance1" }},
{ "term": { "module": "test_module" }},
{ "range": { "@timestamp": { "gte": 1535266800000, "lt": 1535353200000 }}}
]
}
},
"size": 5
}以下是Dev工具的输出：
{
"took": 556,
"timed_out": false,
"_shards": {
"total": 175,
"successful": 175,
"skipped": 165,
"failed": 0
},
"hits": {
"total": 400,
"max_score": 34.769733,
"hits": [
{
"_index": "app_web_log-20180827",
"_type": "doc",
"_id": "FMkHeWUB_hBu7Tio4Llg",
"_score": 34.769733,
"_source": {
"beat": {
"version": "6.2.4",
"name": "app-web001",
"hostname": "app-web001"
},
"offset": 349461,
"@timestamp": "2018-08-27T01:38:03.049Z",
"source": "/apphome/app_instance1/logs/test_module.log",
"message": "2018-08-27 01:37:59,661 [http-bio-8168-exec-8] ERROR [Billing APIClientImpl] Failed to call Billing API Server. Billing API Billing server response error, tranId:c95cede3a011d97fd9f3d661eb961cb8",
"module": "test_module",
"@version": "1",
"deployment": "app_instance1"
}
},
....但是当我查询时，使用elasticsearch python客户端。它什么也没给我：
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'esserver', 'port': 9200, 'username': 'appuser', 'password': 'elastic'}])
body = {
"query": {
"bool": {
"must": [
{ "match_phrase": { "message": "Failed to call Billing API Server" }}
],
"filter": [
{ "term": { "deployment": "app_instance1" }},
{ "term": { "module": "test_module" }},
{ "range": { "@timestamp": { "gte": 1535266800000, "lt": 1535353200000 }}}
]
}
}
}
print body
page = es.search(index='app_web_log-20180827', doc_type='doc', body=body,
scroll='2m', size=100)
sid = page['_scroll_id']
scroll_size = page['hits']['total']
while (scroll_size > 0):
print "Scrolling..."
page = es.scroll(scroll_id = sid, scroll = '2m')
# Update the scroll ID
sid = page['_scroll_id']
# Get the number of results that we returned in the last scroll
scroll_size = len(page['hits']['hits'])
for m in page['hits']['hits']:
msg = m['_source']['message']
print msg我一无所获：
{'query': {'bool': {'filter': [{'term': {'deployment': 'app_instance1'}}, {'term': {'module': 'test_module'}}, {'range': {'@timestamp': {'lt': 1535353200000, 'gte': 1535266800000}}}], 'must': [{'match_phrase': {'message': 'Failed to call Billing API Server'}}]}}}
Scrolling...我想知道代码中是否有什么错误？请帮帮忙。谢谢
回答 1
Stack Overflow用户
回答已采纳
发布于 2018-08-31 06:44:43
我建议您看一下scan helper ()，它为您完成了逻辑。
我假设，由于您只在调用scroll之后而不是之前迭代页面，所以您不会处理search调用返回的命中。您还将size设置为100，因此所有命中结果很可能都在您忽略的page变量的第一个值中。
0- https://elasticsearch-py.readthedocs.io/en/master/helpers.html#scan
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/52103668
复制相关文章
相似问题