首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >flinkx同步es2hive

flinkx同步es2hive

作者头像
yiduwangkai
发布2021-11-22 10:41:18
4120
发布2021-11-22 10:41:18
举报
文章被收录于专栏:大数据进阶大数据进阶

一.elasticsearch环境准备

  1. elasticsearch 6.4.3
  2. es-head

elasticsearch的部署基于docker进行部署

首先我们来看一下目录结构

cd docker
mkdir -p es
cd es
mkdir -p conf
mkdir -p data
mkdir -p logs

下面看一下docker-compose.yml

version: '3'
services:
  elasticsearch:
    image:  elasticsearch:6.4.3
    container_name: elasticsearch
    volumes:
      - $PWD/data:/usr/share/elasticsearch/data
      - $PWD/logs:/user/share/elasticsearch/logs
    ports:
        - '9200:9200'
        - '9300:9300'
    environment:
        - discovery.type=single-node
        - http.port=9200
        - http.cors.enabled=true
        - http.cors.allow-origin=*
        - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
        - http.cors.allow-credentials=false
        - bootstrap.memory_lock=true
        - 'ES_JAVA_OPTS=-Xms512m -Xmx512m'
  es-head:
    image: tobias74/elasticsearch-head:6
    container_name: es-head
    ports:
      - '9100:9100'
    links:
      - elasticsearch  

验证

http://localhost:9200/?pretty

{
  "name" : "Yy55wm7",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "_3n-nj84QhKKpHZ71t0OKQ",
  "version" : {
    "number" : "6.4.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "fe40335",
    "build_date" : "2018-10-30T23:17:19.084789Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

flinkx支持es2hive

es2hive.json

{
  "job" : {
    "content" : [ {
      "reader": {
        "name": "esreader",
        "parameter": {
          "address": "localhost:9200",
          "index": "manage",
          "type": "user",
          "column": [
            {
              "name": "user_id",
              "type": "bigint"
            },{
              "name": "user_name",
              "type": "varchar"
            },{
              "name": "user_phone",
              "type": "varchar"
            }
          ]
        }
      },
      "writer": {
          "name" : "hivewriter",
          "parameter" : {
            "jdbcUrl" : "jdbc:hive2://localhost:10000/es",
            "username" : "wangkai",
            "password" : "wangkai",
            "fileType" : "text",
            "writeMode" : "overwrite",
            "compress" : "",
            "schema" : "es",
            "charsetName" : "UTF-8",
            "maxFileSize" : 1073741824,
            "tablesColumn" : "{\"demonstrate_users\": [{\"key\": \"user_id\",\"type\": \"BIGINT\"}, {\"key\": \"user_name\",\"type\": \"string\"}, {\"key\": \"user_phone\",\"type\": \"string\"}]}",
            "defaultFS" : "hdfs://localhost:9000"
          }
      }
    }
    ],
    "setting" : {
      "restore" : {
        "maxRowNumForCheckpoint" : 0,
        "isRestore" : false,
        "restoreColumnName" : "",
        "restoreColumnIndex" : 0
      },
      "errorLimit" : {
        "record" : 0,
        "percentage" : 0
      },
      "speed" : {
        "bytes" : 1048576,
        "channel" : 1
      }
    }
  }
}

运行命令

bin/flinkx \
  -mode local \
  -job /Users/wangkai/apps/install/flinkx/es2hive.json \
  -pluginRoot syncplugins \
  -flinkconf /Users/wangkai/apps/install/flink-1.13.1/conf  
本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
容器镜像服务
容器镜像服务(Tencent Container Registry,TCR)为您提供安全独享、高性能的容器镜像托管分发服务。您可同时在全球多个地域创建独享实例,以实现容器镜像的就近拉取,降低拉取时间,节约带宽成本。TCR 提供细颗粒度的权限管理及访问控制,保障您的数据安全。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档