我有一个特定的用例,我找不到一个例子,到目前为止,试图把不同的例子拼凑在一起对我来说效果不大。
我在亚马逊网络服务的Lambda中使用了一个nodeJS函数
我有一个缓冲区形式的压缩文件(只需使用getObject从S3读取)。压缩包永远不会超过10mb,所以在内存中进行压缩是可以的,这样可以避免使用流,也希望避免使用本地tmp。
每个压缩包都有一些文件和文件夹,我想要得到一个名为"src/“的文件夹中的所有文件。这对于所有的拉链都是一样的。这些文件不需要解压缩,我想把它们直接放到一个没有"src“子文件夹的新zip中-所以只有新zip根目录下的"src”中的文件和文件夹,压缩过的,src/中的任何文件夹都应该保留它们的层次结构。
这样做的最终结果应该是输出zip的缓冲区。
任何帮助都将不胜感激。
发布于 2021-01-21 23:10:59
我可能已经自己解决了,但我仍然对这方面的任何改进持开放态度,因为它感觉有点粗糙和低效,因为它是解压缩然后再压缩的。
const yauzl = require("yauzl");
const yazl = require("yazl");
const uploadStream = ({ Bucket, Key }) => {
const s3 = new AWS.S3();
const pass = new stream.PassThrough();
return {
writeStream: pass,
promise: s3.upload({ Bucket, Key, Body: pass }).promise(),
};
}
// retrieve the artifact
return s3.getObject({Bucket: bucket, Key: key}).promise().then(res => {
// unzip the artifact
return new Promise(function(resolve, reject){
// new zip that will contain only src
const newzip = new yazl.ZipFile();
// read the zip from buffer (entire zip, this cannot be streamed)
yauzl.fromBuffer(res.Body, {lazyEntries: true}, (err, zip) => {
if(err) {
console.log("Error accessing artifact: ",err);
return reject("Error accessing artifact");
}
// read each item in the zip
zip.readEntry();
zip.on("entry", function(entry){
// we only want files in the src dir, skip others
if(entry.fileName.substr(0,3) !== "src") return zip.readEntry();
// extract file
zip.openReadStream(entry, {decompress: entry.isCompressed() ? true : null}, function (err, readStream) {
if(err){
zip.close();
console.log("Failed to read file in artifact: ", err);
return reject("Failed to read file in artifact");
}
// collect data into buffer
let buffer = null;
readStream.on('data', function(d) {
if(!buffer){
buffer = d;
}else{
buffer = Buffer.concat([buffer, d]);
}
});
// file data collection completed
readStream.on('end', function () {
// add it to the new zip (without the src dir in the path)
newzip.addBuffer(buffer,entry.fileName.substr(4));
// continue to next entry
zip.readEntry();
});
// fail on error
readStream.on('error', function (err) {
zip.close();
console.log("Failed to extract file from artifact: ", err);
return reject("Failed to extract file from artifact");
});
});
});
// all items processed
zip.on("end", function(){
console.log("Completed extracting all files");
// all files added
newzip.end();
// store on s3
const { writeStream, promise } = uploadStream({Bucket: bucket, Key: key+"Deploy"});
newzip.outputStream.pipe(writeStream).on("close", function(){
resolve({result:true,artifact:key+"Deploy"});
});
});
});
}).catch(err => {
console.log("Unzip artifact error: ",err);
return Promise.reject("Could not unzip artifact");
});
}).catch(err => {
console.log("Retrieve artifact error: ",err);
return Promise.reject("Could not retrieve artifact");
});
https://stackoverflow.com/questions/65829425
复制相似问题