sqlldr性能调优测试(r3第5天)

接着续上次提到的sqlldr的性能问题,加载一个表数据400多万条记录,竟然用了6个多小时。最后大家争论不休的时候,我发现应该是网络的问题。 http://blog.itpub.net/23718752/viewspace-1182534/ 今天客户IT的同事把网络做了调整,他们就想看看到底改进有多大。 下面是测试的一些记录。

àoriginal logs for issue table, loading around 6 hours. 
Total logical records skipped:          0 
Total logical records read:       4096786 
Total logical records rejected:         0 
Total logical records discarded:        0 
Run began on Wed Jun 11 08:52:55 2014 
Run ended on Wed Jun 11 14:57:40 2014 
Elapsed time was:     06:04:44.05 
CPU time was:         00:00:38.18 
àafter network enhanced, loading time has reduced to around 4 mins. 
Total logical records skipped:          0 
Total logical records read:       4096786 
Total logical records rejected:         0 
Total logical records discarded:        0 
Run began on Mon Jun 16 12:58:01 2014 
Run ended on Mon Jun 16 13:02:15 2014 
Elapsed time was:     00:04:13.28 
CPU time was:         00:00:30.00 
àconsidering impact from index, I loaded data again, keep table structure,index exactly the same. 
Total logical records skipped:          0 
Total logical records read:       4096786 
Total logical records rejected:         0 
Total logical records discarded:        0 
Run began on Mon Jun 16 13:09:34 2014 
Run ended on Mon Jun 16 13:16:37 2014 
Elapsed time was:     00:07:02.61 
CPU time was:         00:00:31.23 
àtune sqlldr command with below options, keep table structure,index exactly the same. 
parallel=true bindsize=7500000 readsize=7500000 streamsize=7500000 rows=50000


Total logical records skipped:          0

Total logical records read:       4096786 
Total logical records rejected:         0 
Total logical records discarded:        0 
Run began on Mon Jun 16 13:21:13 2014 
Run ended on Mon Jun 16 13:24:19 2014 
Elapsed time was:     00:03:05.75 
CPU time was:         00:00:41.27

所以综上所述,在排查了cpu,io,缓存,网络的因素后,还可以使用一些选项来优化sqlldr,在某种程度上得到提升。

原文发布于微信公众号 - 杨建荣的学习笔记(jianrong-notes)

原文发表时间:2014-06-16

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏Golang语言社区

在GO中编写一个简单的shell

In this post, we will write a minimalistic shell for UNIX(-like) operating syste...

1045
来自专栏WindCoder

Rails Service Objects: A Comprehensive Guide

Ruby on Rails ships with everything you need to prototype your application quick...

311
来自专栏pangguoming

Node.js 开发模式(设计模式)

Asynchronous code & Synchronous code As we have seen in an earlier post (here), ...

2857
来自专栏zaking's

RFC2616-HTTP1.1-Header Field Definitions(头字段规定部分—单词注释版)

992
来自专栏函数式编程语言及工具

PICE(2):JDBCStreaming - gRPC-JDBC Service

   在一个akka-cluster环境里,从数据调用的角度上,JDBC数据库与集群中其它节点是脱离的。这是因为JDBC数据库不是分布式的,不具备节点位置透明化...

680
来自专栏ml

hdu-----(3746)Cyclic Nacklace(kmp)

Cyclic Nacklace Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 32768/32...

2524
来自专栏大数据学习笔记

Spark2.x学习笔记:16、Spark Streaming入门实例NetworkWordCount

16、 Spark Streaming入门实例NetworkWordCount 16.1 源码解析 在Spark源码的spark-2.1.0-bin-hadoo...

2088
来自专栏ml

HDUOJ-----1085Holding Bin-Laden Captive!

Holding Bin-Laden Captive! Time Limit: 2000/1000 MS (Java/Others)    Memory Limi...

26611
来自专栏WindCoder

Declarative Programming: Is It A Real Thing?

由于合作方希望能以英文形式发布,故以后top的译文看时间而定,没时间就不再尝试翻译(而且本来水平也不咋地),仅保留原文于此。本次是一篇关于声明式编程的讨论文章,...

301
来自专栏pangguoming

NodeJS 各websocket框架性能分析

For a current project at WhoScored, I needed to learn JavaScript, Node.js and We...

3785

扫描关注云+社区