维表拉链表案例_维善优案例

全栈程序员站长

发布于 2022-11-03 14:49:58

2930

发布于 2022-11-03 14:49:58

文章被收录于专栏：全栈程序员必看

一.数据准备

下表为分区增量数据表(userinfo)

创建表加载数据 用户信息

create table test.userinfo(
   userid string comment '用户编号',
   mobile string comment '手机号码',
   regdate string comment '注册日期'
)comment '用户信息'
partitioned by(dt string)
row format delimited fields terminated by ',';

拉链表(存放用户历史信息) 拉链表不是分区表:多了两个字段start_date,end_date

create table test.userhis(
   userid string comment '用户编号',
   mobile string comment '手机号码',
   regdate string comment '注册日期',
   start_date string,
   end_date string
)comment '用户信息拉链表'
row format delimited fields terminated by ',';

数据准备

001,13551111111,2020-03-01,2020-06-20
002,13561111111,2020-04-01,2020-06-20
003,13571111111,2020-05-01,2020-06-20
004,13581111111,2020-06-01,2020-06-20
002,13562222222,2020-04-01,2020-06-21
004,13582222222,2020-06-01,2020-06-21
005,13552222222,2020-06-21,2020-06-21
004,13333333333,2020-06-01,2020-06-22
005,13533333333,2020-06-21,2020-06-22
006,13733333333,2020-06-22,2020-06-22
001,13554444444,2020-03-01,2020-06-23
003,13557444444,2020-05-01,2020-06-23
005,13555555444,2020-06-21,2020-06-23
007,18674444444,2020-06-23,2020-06-23
008,13699844444,2020-06-23,2020-06-23

法一: 静态分区数据加载

/data/userinfo.dat
001,13551111111,2020-03-01
002,13561111111,2020-04-01
003,13571111111,2020-05-01
004,13581111111,2020-06-01
load data local inpath '/data/userinfo.dat' into table test.userinfo partition(dt='2020-06-20');

法二: 动态分区数据的加载:分区的值不是固定的,由输入的数据决定 创建中间表

create table test.tmp1 as 
select * from test.userinfo;

tmp1 非分区表,使用系统默认的字段分割符’\001’

alter table test.tmp1 set serdeproperties('field.delim'=',');

向中间表中加入数据

load data local inpath '/data/userinfo.dat' overwrite into table test.tmp1;

从中间表向分区表加载数据

set hive.exec.dynamic.partition.mode=nonstrict;
insert into table test.userinfo
partition(dt)
select * from test.tmp1;

将数据成功插入到了分区表中,动态加载更加能应对分区多,数据多情况

hive (test)> select * from test.userinfo;
OK
userinfo.userid	userinfo.mobile	userinfo.regdate	userinfo.dt
001	13551111111	2020-03-01	2020-06-20
002	13561111111	2020-04-01	2020-06-20
003	13571111111	2020-05-01	2020-06-20
004	13581111111	2020-06-01	2020-06-20
002	13562222222	2020-04-01	2020-06-21
004	13582222222	2020-06-01	2020-06-21
005	13552222222	2020-06-21	2020-06-21
004	13333333333	2020-06-01	2020-06-22
005	13533333333	2020-06-21	2020-06-22
006	13733333333	2020-06-22	2020-06-22
001	13554444444	2020-03-01	2020-06-23
003	13557444444	2020-05-01	2020-06-23
005	13555555444	2020-06-21	2020-06-23
007	18674444444	2020-06-23	2020-06-23
008	13699844444	2020-06-23	2020-06-23
Time taken: 0.398 seconds, Fetched: 15 row(s)

二.拉链表实现

我们的数据来源是来自test.userinfo分区表,每日更新为一个分区(包括新增+修改) 拉链表增加两个字段start_date,end_date(两字段区间表示该数据有效时间区间,否则无效) 拉链表第一天为历史数据

insert overwrite table test.userhis
select userid,mobile,regdate,dt as start_date,'9999-12-31' as end_date
from test.userinfo
where dt = '2020-06-20';

拉链表次日新增和修改数据 第二天的拉链表为今日数据+历史数据今日数据包括新增数据+历史数据修改因此此时当天拉链表=历史数据有效期整合后的数据+新增的数据即拉链表userhis(2020-06-22) <= userinfo(2020-06-22) + userhis

第一步处理新增数据

select userid,mobile,regdate,dt as start_date,'9999-12-31' as end_date
from test.userinfo
where dt = '2020-06-21';

第二步处理历史数据(变化的+未变化的)

变化的数据我们要更新有效区间,未变化的数据我们要保留(使用表的外连接)

select A.userid,A.mobile,A.regdate,A.start_date,
case when A.end_date='9999-12-31' and B.userid is not null
then date_add(B.dt,-1)
else A.end_date end as end_date 
from test.userhis A left join (select * from test.userinfo where dt = '2020-06-21') B on A.userid = B.userid;

综上2020-06-21当日的拉链表是:

insert overwrite table test.userhis
select userid,mobile,regdate,dt as start_date,'9999-12-31' as end_date
from test.userinfo
where dt = '2020-06-21'
union all
select A.userid,A.mobile,A.regdate,A.start_date,
case when A.end_date='9999-12-31' and B.userid is not null
then date_add(B.dt,-1)
else A.end_date end as end_date 
from test.userhis A left join (select * from test.userinfo where dt = '2020-06-21') B on A.userid = B.userid;

查询当前拉链表数据

hive (test)> select * from userhis cluster by userid,start_date;
userhis.userid	userhis.mobile	userhis.regdate	userhis.start_date	userhis.end_date
001	13551111111	2020-03-01	2020-06-20	9999-12-31
002	13561111111	2020-04-01	2020-06-20	2020-06-20
002	13562222222	2020-04-01	2020-06-21	9999-12-31
003	13571111111	2020-05-01	2020-06-20	9999-12-31
004	13581111111	2020-06-01	2020-06-20	2020-06-20
004	13582222222	2020-06-01	2020-06-21	9999-12-31
005	13552222222	2020-06-21	2020-06-21	9999-12-31
Time taken: 3.148 seconds, Fetched: 7 row(s)

以后每日的拉链表就是修改对应的dt信息通过以上方式我们每日拉链表可以通过脚本来更新

#!/bin/bash
if [ -n "$1" ]
then 
do_date=$1
else
do_date=`date -d "-1 day" +%F`   
fi
sql="
insert overwrite table test.userhis
select userid,mobile,regdate,dt as start_date,'9999-12-31' as end_date
from test.userinfo
where dt = '$do_date'
union all
select A.userid,A.mobile,A.regdate,A.start_date,
case when A.end_date='9999-12-31' and B.userid is not null
then date_add(B.dt,-1)
else A.end_date end as end_date 
from test.userhis A left join (select * from test.userinfo where dt = '$do_date') B on A.userid = B.userid;"
hive -e "$sql"

通过执行脚本

sh userTest.sh 2020-06-22
userhis.userid	userhis.mobile	userhis.regdate	userhis.start_date	userhis.end_date
001	13551111111	2020-03-01	2020-06-20	9999-12-31
002	13561111111	2020-04-01	2020-06-20	2020-06-20
002	13562222222	2020-04-01	2020-06-21	9999-12-31
003	13571111111	2020-05-01	2020-06-20	9999-12-31
004	13581111111	2020-06-01	2020-06-20	2020-06-20
004	13582222222	2020-06-01	2020-06-21	2020-06-21
004	13333333333	2020-06-01	2020-06-22	9999-12-31
005	13552222222	2020-06-21	2020-06-21	2020-06-21
005	13533333333	2020-06-21	2020-06-22	9999-12-31
006	13733333333	2020-06-22	2020-06-22	9999-12-31
Time taken: 3.037 seconds, Fetched: 10 row(s)
sh userTest.sh 2020-06-23
userhis.userid	userhis.mobile	userhis.regdate	userhis.start_date	userhis.end_date
001	13551111111	2020-03-01	2020-06-20	2020-06-22
001	13554444444	2020-03-01	2020-06-23	9999-12-31
002	13561111111	2020-04-01	2020-06-20	2020-06-20
002	13562222222	2020-04-01	2020-06-21	9999-12-31
003	13571111111	2020-05-01	2020-06-20	2020-06-22
003	13557444444	2020-05-01	2020-06-23	9999-12-31
004	13581111111	2020-06-01	2020-06-20	2020-06-20
004	13582222222	2020-06-01	2020-06-21	2020-06-21
004	13333333333	2020-06-01	2020-06-22	9999-12-31
005	13552222222	2020-06-21	2020-06-21	2020-06-21
005	13533333333	2020-06-21	2020-06-22	2020-06-22
005	13555555444	2020-06-21	2020-06-23	9999-12-31
006	13733333333	2020-06-22	2020-06-22	9999-12-31
007	18674444444	2020-06-23	2020-06-23	9999-12-31
008	13699844444	2020-06-23	2020-06-23	9999-12-31
Time taken: 1.442 seconds, Fetched: 15 row(s)

三.拉链表的应用

1.查看拉链表中最新的数据(2020-06-23为最新数据)

hive (test)> select * from userhis where end_date='9999-12-31';
OK
userhis.userid	userhis.mobile	userhis.regdate	userhis.start_date	userhis.end_date
002	13562222222	2020-04-01	2020-06-21	9999-12-31
004	13333333333	2020-06-01	2020-06-22	9999-12-31
006	13733333333	2020-06-22	2020-06-22	9999-12-31
001	13554444444	2020-03-01	2020-06-23	9999-12-31
003	13557444444	2020-05-01	2020-06-23	9999-12-31
005	13555555444	2020-06-21	2020-06-23	9999-12-31
007	18674444444	2020-06-23	2020-06-23	9999-12-31
008	13699844444	2020-06-23	2020-06-23	9999-12-31
Time taken: 0.517 seconds, Fetched: 8 row(s)

2.查看拉链表中给定日期的数据(也就是开始日期一定<=给定日期,结束日期>=给定日期)

例如:2020-06-22
hive (test)> select * from userhis where start_date <= '2020-06-22' and end_date >= '2020-06-22';
OK
userhis.userid	userhis.mobile	userhis.regdate	userhis.start_date	userhis.end_date
001	13551111111	2020-03-01	2020-06-20	2020-06-22
003	13571111111	2020-05-01	2020-06-20	2020-06-22
002	13562222222	2020-04-01	2020-06-21	9999-12-31
004	13333333333	2020-06-01	2020-06-22	9999-12-31
005	13533333333	2020-06-21	2020-06-22	2020-06-22
006	13733333333	2020-06-22	2020-06-22	9999-12-31
Time taken: 0.244 seconds, Fetched: 6 row(s)

四.拉链表回滚

(1)法一

由于种种原因,我们需要对拉链表进行回滚操作,将拉链表回滚到rollback_date那天的数据此时我们对拉链表中的数据进行分类

end_date<rollback_date 此类数据保持不变保留
start_date<= rollback_date <=end_date 此类数据在rollback_date那天是有效数据修改end_date为”9999-12-31″
rollback_date < start_date 该类数据在rollback_date那天属于无效数据-删除

按照以上方式对2020-06-22拉链表回滚进行sql编码

1.end_date<rollback_date,保留

select userid,mobile,regdate,start_date,end_date,'1' as tag
from test.userhis
where end_date < '2020-06-22';

2.start_date<= rollback_date <=end_date 修改end_date为”9999-12-31″

select userid,mobile,regdate,start_date,'9999-12-31' as end_date,'2' as tag
from test.userhis
where start_date <= '2020-06-22' and '2020-06-22' <= end_date;

将以上汇总的数据写入临时表tmp(拉链表)

drop table test.tmp;
create table test.tmp as
select userid,mobile,regdate,start_date,end_date,'1' as tag
from test.userhis
where end_date < '2020-06-22'
union all
select userid,mobile,regdate,start_date,'9999-12-31' as end_date,'2' as tag
from test.userhis
where start_date <= '2020-06-22' and '2020-06-22' <= end_date;

脚本执行

#!/bin/bash
if [ -n "$1" ]
then 
do_date=$1
else
do_date=`date -d "-1 day" +%F`   
fi
sql="
drop table test.tmp;
create table test.tmp as
select userid,mobile,regdate,start_date,end_date
from test.userhis
where end_date < '$do_date'
union all
select userid,mobile,regdate,start_date,'9999-12-31' as end_date
from test.userhis
where start_date <= '$do_date' and '$do_date' <= end_date;
select * from test.tmp;"
hive -e "$sql"

我们这里以临时表来测试数据正确性,实际可以替换真实拉链表插入更新

(2)法二(简单处理方式)

保存一段时间增量数据(userinfo),定期对拉链表做备份(10天或者一个月);如需回滚,直接在备份的拉链表上向后跑增量数据(类似向后更新拉链表)

发布者：全栈程序员栈长，转载请注明出处：https://javaforall.cn/181097.html原文链接：https://javaforall.cn

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2022年10月17日，如有侵权请联系 cloudcommunity@tencent.com 删除

https

网络安全

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体同步曝光计划，欢迎热爱写作的你一起参与！

https

网络安全

登录后参与评论

0 条评论

热度