blocks|key|149722|text|因为当您尝试从Cassandra中导出一个包含数百万行的表时，使用COPY将非常具有挑战性，所以我所做的是创建一个简单的工具来从cassandra表中逐块(分页)获取数据块，并将其导出到CSV。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|149723|使用来自datastax的java库查看我的example+solution。|offset|length|149724|entityMap|0|LINK|mutability|MUTABLE|url|https://moshimon.wordpress.com/2015/01/19/export-data-from-cassandra-to-csv/^0|0|M|G|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@$D|P|E|Q|1|R]]|A|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|A|$]]]|G|$H|$5|I|J|K|A|$L|M]]]]

Because using COPY will be quite challenging when you are trying to export a table with millions of rows from Cassandra, So what I have done is to create simple tool to get the data chunk by chunk (paginated) from cassandra table and export it to CSV.

Look at my <a href="https://moshimon.wordpress.com/2015/01/19/export-data-from-cassandra-to-csv/" rel="nofollow">example solution</a> using java library from datastax.

blocks|key|2947178|text|受@user1859675答案的启发，下面是我们如何使用Spark从Cassandra导出数据|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2947179|val+cassandraHostNode+=+"10.xxx.xxx.x5,10.xxx.xxx.x6,10.xxx.xxx.x7";
val+spark+=+org.apache.spark.sql.SparkSession
++++++++++++++++++++++++++++++++++++.builder
++++++++++++++++++++++++++++++++++++.config("spark.cassandra.connection.host",++cassandraHostNode)
++++++++++++++++++++++++++++++++++++.appName("Awesome+Spark+App")
++++++++++++++++++++++++++++++++++++.master("local[*]")
++++++++++++++++++++++++++++++++++++.getOrCreate()

val+dataSet+=+spark.read.format("org.apache.spark.sql.cassandra")
++++++++++++++++++++++++.options(Map("table"+->+"xxxxxxx",+"keyspace"+->+"xxxxxxx"))
++++++++++++++++++++++++.load()

val+targetfilepath+=+"/opt/report_values/"
dataSet.write.format("csv").save(targetfilepath)++//+Spark+2.x|code-block|syntax|javascript|2947180|您需要在类路径中使用"spark-cassandra-connector“才能正常工作。|offset|length|style|CODE|2947181|我使用的版本如下|2947182|++++<groupId>com.datastax.spark</groupId>
++++<artifactId>spark-cassandra-connector_2.11</artifactId>
++++<version>2.3.2</version>|2947183|entityMap^0|0|0|B|P|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|U|8|@$I|V|J|W|K|L]]|9|@]|A|$]]|$1|M|3|N|5|6|7|X|8|@]|9|@]|A|$]]|$1|O|3|P|5|D|7|Y|8|@]|9|@]|A|$E|F]]|$1|Q|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|R|$]]

Inspired by @user1859675 's answer, Here is how we can export data from Cassandra using Spark

<pre><code>val cassandraHostNode = "10.xxx.xxx.x5,10.xxx.xxx.x6,10.xxx.xxx.x7";
val spark = org.apache.spark.sql.SparkSession
 .builder
 .config("spark.cassandra.connection.host", cassandraHostNode)
 .appName("Awesome Spark App")
 .master("local[*]")
 .getOrCreate()

val dataSet = spark.read.format("org.apache.spark.sql.cassandra")
 .options(Map("table" -&gt; "xxxxxxx", "keyspace" -&gt; "xxxxxxx"))
 .load()

val targetfilepath = "/opt/report_values/"
dataSet.write.format("csv").save(targetfilepath) // Spark 2.x
</code></pre>

You will need "<code>spark-cassandra-connector</code>" in your classpath for this to work. 
The version I am using is below 

<pre><code> &lt;groupId&gt;com.datastax.spark&lt;/groupId&gt;
 &lt;artifactId&gt;spark-cassandra-connector_2.11&lt;/artifactId&gt;
 &lt;version&gt;2.3.2&lt;/version&gt;
</code></pre>

blocks|key|152477|text|针对2020的更新:+DataStax提供了一个名为DSBulk的特殊工具，用于从Cassandra+(从Cassandra+2.1开始)和DSE+(从DSE4.7/4.8开始)加载和卸载数据。在最简单的情况下，命令行如下所示：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|152478|dsbulk+unload+-k+keyspace+-t+table+-url+path_to_unload|code-block|syntax|javascript|152479|DSBulk针对加载/卸载操作进行了大量优化，并有许多选项，包括从压缩文件导入/导出，提供自定义查询等。|152480|有一系列关于DSBulk的博客文章，可以提供更多信息和示例：1，2，3，4，5，6|offset|length|152481|entityMap|0|LINK|mutability|MUTABLE|url|https://www.datastax.com/blog/2019/03/datastax-bulk-loader-introduction-and-loading|1|https://www.datastax.com/blog/2019/04/datastax-bulk-loader-more-loading|2|https://www.datastax.com/blog/2019/04/datastax-bulk-loader-common-settings|3|https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading|4|https://www.datastax.com/blog/2019/07/datastax-bulk-loader-counting|5|https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations^0|0|0|0|U|1|0|W|1|1|Y|1|2|10|1|3|12|1|4|14|1|5|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|15|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|16|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|17|8|@]|9|@$K|18|L|19|1|1A]|$K|1B|L|1C|1|1D]|$K|1E|L|1F|1|1G]|$K|1H|L|1I|1|1J]|$K|1K|L|1L|1|1M]|$K|1N|L|1O|1|1P]]|A|$]]|$1|M|3|-4|5|6|7|1Q|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|T]]|U|$5|P|Q|R|A|$S|V]]|W|$5|P|Q|R|A|$S|X]]|Y|$5|P|Q|R|A|$S|Z]]|10|$5|P|Q|R|A|$S|11]]|12|$5|P|Q|R|A|$S|13]]]]

Update for 2020th: DataStax provides a special tool called DSBulk for loading and unloading of data from Cassandra (starting with Cassandra 2.1), and DSE (starting with DSE 4.7/4.8). In simplest case, the command line looks as following:

<pre><code>dsbulk unload -k keyspace -t table -url path_to_unload
</code></pre>

DSBulk is heavily optimized for loading/unloading operations, and has a lot of options, including import/export from/to compressed files, providing the custom queries, etc.

There is a series of blog posts about DSBulk, that could provide more information &amp; examples: <a href="https://www.datastax.com/blog/2019/03/datastax-bulk-loader-introduction-and-loading" rel="nofollow noreferrer">1</a>, <a href="https://www.datastax.com/blog/2019/04/datastax-bulk-loader-more-loading" rel="nofollow noreferrer">2</a>, <a href="https://www.datastax.com/blog/2019/04/datastax-bulk-loader-common-settings" rel="nofollow noreferrer">3</a>, <a href="https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading" rel="nofollow noreferrer">4</a>, <a href="https://www.datastax.com/blog/2019/07/datastax-bulk-loader-counting" rel="nofollow noreferrer">5</a>, <a href="https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations" rel="nofollow noreferrer">6</a>

I'm using Cassandra 2.0.9 for store quite big amounts of data, let's say 100Gb, in one column family. I would like to export this data to CSV in fast way. I tried:

<ul>
<li>sstable2json - it produces quite big json files which are hard to parse - because tool puts data in one row and uses complicated schema (ex. 300Mb Data file = ~2Gb json), it takes a lot of time to dump and Cassandra likes to change source file names according its internal mechanism</li>
<li>COPY - causes timeouts on quite fast EC2 instances for big number of records</li>
<li>CAPTURE - like above, causes timeouts</li>
<li>reads with pagination - I used timeuuid for it, but it returns about 1,5k records per second</li>
</ul>

I use Amazon Ec2 instance with fast storage, 15 Gb of RAM and 4 cores

Is there any better option for export gigabytes of data from Cassandra to CSV?

Export large amount of data from Cassandra to CSV

我使用Cassandra 2.0.9在一个列系列中存储相当大的数据量，比如说100‘s。我想以快速的方式将此数据导出到CSV。我试过了：sstable2json -它生成了相当大的json文件，很难解析-因为工具将数据放在一行中，并使用复杂的模式(例如。300Mb数据文件=~2 2Gb)，转储需要很长时间，而且Cass...

问将大量数据从Cassandra导出到CSV
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将大量数据从Cassandra导出到CSVEN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将大量数据从Cassandra导出到CSV
EN