logstash迁移索引数据自动添加@version和@timestamp字段

原创

沈小翊

修改于 2023-11-23 11:07:21

3360

修改于 2023-11-23 11:07:21

文章被收录于专栏：大数据生态大数据生态

问题背景

使用Logstash迁移ES数据时发现有个索引数据无法迁移过来（其他索引正常），事先已经同步过mapping，settings，两边一致。

报错如下：

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"6251",
 :_index=>"test", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x77513d30>],
  :response=>{"index"=>{"_index"=>"test", "_type"=>"_doc", "_id"=>"6251", 
  "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"doc early 
  terminate:org.elasticsearch.index.mapper.StrictDynamicMappingException: mapping set 
  to strict, dynamic introduction of [@version] within [_doc] is not allowed"}}}}

原因：logstash迁移过程中会额外加入@version字段和@timestamp，而目标端索引动态映射参数又设置是strict无法接受不是提前在mapping中自定义的字段。

解决方法

手动过滤掉该字段

 filter{
         mutate{
            remove_field => ["@version"]
        }
}

2. 或者将索引的动态映射参数设置为true

PUT new_index/_mapping
{
  "dynamic":"strict"
}

问题复现

创建一个仅有data字段的索引

PUT old_index

PUT old_index/_doc/1
{
  "data":1
}

GET old_index

//可以看到
  "properties" : {
    "data" : {
     "type" : "long"
  }
}

Logstash管道配置

input {
  elasticsearch {
    hosts => ["source_ip:9200"]
    user => "elastic"
    password => "xxxx"
    index => "old_index"
  }
}
output {
    elasticsearch {
        hosts => ["http://target_ip:9200"]
        user => "elastic"
        password => "xxxx"
        index => "new_index"
    }
}

启动Logstash，查看new_index属性

GET new_index

//可以看到多出@version字段和@timestamp字段

      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "data" : {
          "type" : "long"
        }
      }
    }

Logstash没报错是因为dynamic参数默认为true,接受新字段

DELETE new_index

//将dynamic动态映射参数设置为strict，拒绝一切新字段
PUT new_index
{
  "mappings": {
    "dynamic":"strict",
    "properties": {
      "data":{
        "type": "integer"
      }
    }
  }
}

重新启动Logstash，发现报错，符合预期

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil,
 :_index=>"new_index", :routing=>nil}, {"@timestamp"=>2023-11-23T02:45:13.511Z, 
 "data"=>1, "@version"=>"1"}], :response=>{"index"=>{"_index"=>"new_index", 
 "_type"=>"_doc", "_id"=>"j_oR-osB-CZwbKC6NYh8", "status"=>400, "error"=>{"type"=>
 "strict_dynamic_mapping_exception", "reason"=>"mapping set to strict, dynamic 
 introduction of [@timestamp] within [_doc] is not allowed"}}}}

dynamic

dynamic	参数说明
true	新字段将添加到映射中（默认）。
runtime	新字段将作为运行时字段添加到映射中。这些字段没有索引，而是_source在查询时加载的。
false	新字段将被忽略。这些字段不会被索引或可搜索，但仍会出现在_source返回的命中字段中。这些字段不会添加到映射中，必须显式添加新字段。
strict	如果检测到新字段，则会引发异常并拒绝文档。新字段必须显式添加到映射中。