最近一个群友的boss让研究hbase,让hbase的入库速度达到5w+/s,这可愁死了,4台个人电脑组成的集群,多线程入库调了好久,速度也才1w左右,都没有达到理想的那种速度,然后就想到了这种方式,但是网上多是用mapreduce来实现入库,而现在的需求是实时入库,不生成文件了,所以就只能自己用代码实现了,但是网上查了很多资料都没有查到,最后在一个网友的指引下,看了源码,最后找到了生成Hfile的方式,实现了之后,发现单线程入库速度才达到1w4左右,和之前的多线程的全速差不多了,百思不得其解之时,调整了一下代码把列的Byte.toBytes(cols)这个方法调整出来只做一次,速度立马就到3w了,提升非常明显,这是我的电脑上的速度,估计在它的集群上能更快一点吧,下面把代码和大家分享一下。
1 String tableName = "taglog";
2 byte[] family = Bytes.toBytes("logs");
3 //配置文件设置
4 Configuration conf = HBaseConfiguration.create();
5 conf.set("hbase.master", "192.168.1.133:60000");
6 conf.set("hbase.zookeeper.quorum", "192.168.1.135");
7 //conf.set("zookeeper.znode.parent", "/hbase");
8 conf.set("hbase.metrics.showTableName", "false");
9 //conf.set("io.compression.codecs", "org.apache.hadoop.io.compress.SnappyCodec");
10
11 String outputdir = "hdfs://hadoop.Master:8020/user/SEA/hfiles/";
12 Path dir = new Path(outputdir);
13 Path familydir = new Path(outputdir, Bytes.toString(family));
14 FileSystem fs = familydir.getFileSystem(conf);
15 BloomType bloomType = BloomType.NONE;
16 final HFileDataBlockEncoder encoder = NoOpDataBlockEncoder.INSTANCE;
17 int blockSize = 64000;
18 Configuration tempConf = new Configuration(conf);
19 tempConf.set("hbase.metrics.showTableName", "false");
20 tempConf.setFloat(HConstants.HFILE_BLOCK_CACHE_SIZE_KEY, 1.0f);
21 //实例化HFile的Writer,StoreFile实际上只是HFile的轻量级的封装
22 StoreFile.Writer writer = new StoreFile.WriterBuilder(conf, new CacheConfig(tempConf),
23 fs, blockSize)
24 .withOutputDir(familydir)
25 .withCompression(Compression.Algorithm.NONE)
26 .withBloomType(bloomType).withComparator(KeyValue.COMPARATOR)
27 .withDataBlockEncoder(encoder).build();
28 long start = System.currentTimeMillis();
29
30 DecimalFormat df = new DecimalFormat("0000000");
31
32
33
34 KeyValue kv1 = null;
35 KeyValue kv2 = null;
36 KeyValue kv3 = null;
37 KeyValue kv4 = null;
38 KeyValue kv5 = null;
39 KeyValue kv6 = null;
40 KeyValue kv7 = null;
41 KeyValue kv8 = null;
42
43 //这个是耗时操作,只进行一次
44 byte[] cn = Bytes.toBytes("cn");
45 byte[] dt = Bytes.toBytes("dt");
46 byte[] ic = Bytes.toBytes("ic");
47 byte[] ifs = Bytes.toBytes("if");
48 byte[] ip = Bytes.toBytes("ip");
49 byte[] le = Bytes.toBytes("le");
50 byte[] mn = Bytes.toBytes("mn");
51 byte[] pi = Bytes.toBytes("pi");
52
53 int maxLength = 3000000;
54 for(int i=0;i<maxLength;i++){
55 String currentTime = ""+System.currentTimeMillis() + df.format(i);
56 long current = System.currentTimeMillis();
57 //rowkey和列都要按照字典序的方式顺序写入,否则会报错的
58 kv1 = new KeyValue(Bytes.toBytes(currentTime),
59 family, cn,current,KeyValue.Type.Put,Bytes.toBytes("3"));
60
61 kv2 = new KeyValue(Bytes.toBytes(currentTime),
62 family, dt,current,KeyValue.Type.Put,Bytes.toBytes("6"));
63
64 kv3 = new KeyValue(Bytes.toBytes(currentTime),
65 family, ic,current,KeyValue.Type.Put,Bytes.toBytes("8"));
66
67 kv4 = new KeyValue(Bytes.toBytes(currentTime),
68 family, ifs,current,KeyValue.Type.Put,Bytes.toBytes("7"));
69
70 kv5 = new KeyValue(Bytes.toBytes(currentTime),
71 family, ip,current,KeyValue.Type.Put,Bytes.toBytes("4"));
72
73 kv6 = new KeyValue(Bytes.toBytes(currentTime),
74 family, le,current,KeyValue.Type.Put,Bytes.toBytes("2"));
75
76 kv7 = new KeyValue(Bytes.toBytes(currentTime),
77 family, mn,current,KeyValue.Type.Put,Bytes.toBytes("5"));
78
79 kv8 = new KeyValue(Bytes.toBytes(currentTime),
80 family,pi,current,KeyValue.Type.Put,Bytes.toBytes("1"));
81
82 writer.append(kv1);
83 writer.append(kv2);
84 writer.append(kv3);
85 writer.append(kv4);
86 writer.append(kv5);
87 writer.append(kv6);
88 writer.append(kv7);
89 writer.append(kv8);
90 }
91
92
93 writer.close();
94
95 //把生成的HFile导入到hbase当中
96 HTable table = new HTable(conf,tableName);
97 LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf);
98 loader.doBulkLoad(dir, table);
最后再附上查看hfile的方式,查询正确的hfile和自己生成的hfile,方便查找问题。
hbase org.apache.hadoop.hbase.io.hfile.HFile -p -f hdfs://hadoop.Master:8020/user/SEA/hfiles/logs/51aa97b2a25446f89d5c870af92c9fc1