我一直在寻找各种来源,但对这个新手来说还不清楚。如何将云存储中的数据(CSV文件)从AppEngine应用程序加载到云数据存储?我确实有一个现有的方法,它可以下载文件,然后将每一行作为一个事务加载。几百万行需要几个小时,所以这似乎不是最好的方法,并且一直在寻找更有效的方法。感谢您的指导。
编辑这段代码,因为我已经切换到尝试使用远程URL,从该URL将JSON数据从GAE加载到Datastore。代码不工作,尽管我还不知道原因:
<?php
require 'vendor/autoload.php';
use Google\Auth\ApplicationDefaultCredentials;
use Google\Cloud\Datastore\DatastoreClient;
/**
* Create a new product with a given SKU.
*
* @param DatastoreClient $datastore
* @param $sku
* @param $product
* @return Google\Cloud\Datastore\Entity
*/
function add_product(DatastoreClient $datastore, $sku, $product)
{
$productKey = $datastore->key('SKU', $sku);
$product = $datastore->entity(
$productKey,
[
'created' => new DateTime(),
'name' => strtolower($product)
]);
$datastore->upsert($product);
return $product;
}
/*
Load Cloud DataStore Kind from remote URL
@param $projectId
@param $url
*/
function load_datastore($projectId, $url) {
// Create Datastore client
$datastore = new DatastoreClient(['projectId' => $projectId]);
// Enable `allow_url_fopen` to allow reading file from URL
ini_set("allow_url_fopen", 1);
// Read the products listing and load to Cloud Datastore.
// Use batches of 20 for a transaction
$json = json_decode(file_get_contents($url), true);
$count = 1;
foreach($json as $sku_key => $product_val) {
if ($count == 1) {
$transaction = $datastore->transaction();
}
add_product($datastore, $sku_key, $product_val);
if ($count == 20) {
$transaction->commit();
$count = 0;
} catch (Exception $err) {
echo 'Caught exception: ', $err->getMessage(), "\n";
$transaction->rollback();
}
$count++;
}
}
try
{
$projectId = 'development';
$url = 'https://raw.githubusercontent.com/BestBuyAPIs/open-data-set/master/products.json';
load_datastore($projectId, $url);
} catch (Exception $err) {
echo 'Caught exception: ', $err->getMessage(), "\n";
$transaction->rollback();
}
?>
发布于 2018-06-26 23:26:05
这个问题类似于Import CSV into google cloud datastore和Google Cloud Datastore: Bulk Importing w Node.js。
简单的答案是,您可以使用Apache光束或Cloud Dataflow将CSV数据导入到Cloud Datastore中。
https://stackoverflow.com/questions/50897699
复制相似问题