处理几十兆甚至上百兆的 JSON 文件时，很多 Node.js 项目会直接 `fs.readFileSync` 读进内存再 `JSON.parse`，本地小文件没问题，一到生产环境跑批处理就容易遇到 `JavaScript heap out of memory`，或者把容器内存顶到上限被 OOM Kill。这个场景在日志分析、数据导入、第三方接口导出对账里很常见。根本原因是 `JSON.parse` 需要先拿到完整字符串，构建出完整对象树才能返回，内存占用大致是文件体积的几倍。这篇文章从实际代码出发，讲清楚怎么用流式解析器逐条消费 JSON 数组元素，怎么在 `highWaterMark`、背压和异步写入之间取舍，以及哪些情况适合流式、哪些情况老老实实分批读更省事。

先看一个容易踩坑的写法：

```js
const fs = require('fs');

const raw = fs.readFileSync('./orders.json', 'utf8');
const orders = JSON.parse(raw);

for (const order of orders) {
 await saveToDb(order);
}
```

文件 200MB 时，`raw` 字符串大约 200MB（UTF-8 下 ASCII 内容接近 1:1），`JSON.parse` 出来的对象数组加上字符串键值，堆内存峰值轻松到 600MB 以上，V8 默认老生代空间在 64 位下通常也就 1.4GB～2GB，批量任务并发几个就撑不住了。

如果 JSON 顶层是一个数组，可以用 `stream-json` 这类流式解析库，按 token 消费，不必等整个文件读完：

```js
const fs = require('fs');
const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');

async function importOrders(filePath) {
 const pipeline = chain([
 fs.createReadStream(filePath, { highWaterMark: 64 * 1024 }),
 parser(),
 streamArray(),
 ]);

 for await (const { value: order } of pipeline) {
 await saveToDb(order);
 }
}
```

`streamArray` 会在解析到数组每个元素结束时抛出一个 `{ key, value }`，内存里同时只保留当前元素和解析器状态。注意 `for await` 在这里会等待 `saveToDb` 完成才继续拉下一个元素，天然形成背压，不会因为数据库慢而把解析结果堆在内存里。如果写入侧是 `http.request` 或 `fs.createWriteStream`，也要用 `await once(stream, 'drain')` 或者 `pipeline` 处理背压，否则解析快了照样会把可写流缓冲撑大。

如果顶层不是数组，而是 `{ "data": [...] }` 这种结构，`stream-json` 可以用 `pick` 配合 `streamArray` 指定路径：

```js
const { pick } = require('stream-json/filters/Pick');

const pipeline = chain([
 fs.createReadStream(filePath),
 parser(),
 pick({ filter: 'data' }),
 streamArray(),
]);
```

还有一个实际会遇到的细节：JSON 数组元素之间可能夹杂换行和空格，流式解析器本身能处理，但如果你自己按行切分再 `JSON.parse`，就要确保对象不会跨行。NDJSON（每行一个完整 JSON 对象）的文件更适合按行读：

```js
const readline = require('readline');

async function importNdjson(filePath) {
 const rl = readline.createInterface({
 input: fs.createReadStream(filePath),
 crlfDelay: Infinity,
 });

 for await (const line of rl) {
 if (!line.trim()) continue;
 const order = JSON.parse(line);
 await saveToDb(order);
 }
}
```

这种方式代码量最少，但只适用于每行都是合法 JSON 的文件格式。数据提供方如果给的是标准 JSON 数组，还是得用流式解析器或者先做一次格式转换。

反过来，如果文件只有几兆，或者你确实需要随机访问整个对象图，流式解析反而增加复杂度。这时候更实际的做法是限制文件大小、用 `worker_threads` 把解析放到独立线程并设置 `resourceLimits`，避免主线程被阻塞，同时让 OOM 只影响 worker：

```js
const { Worker } = require('worker_threads');

function parseInWorker(filePath) {
 return new Promise((resolve, reject) => {
 const worker = new Worker('./parse-worker.js', {
 workerData: { filePath },
 resourceLimits: { maxOldGenerationSizeMb: 512 },
 });
 worker.once('message', resolve);
 worker.once('error', reject);
 });
}
```

流式解析不是万能药，它把内存问题换成了代码复杂度和错误处理问题：解析中途遇到非法 JSON 时，前面已经处理的数据要不要回滚，需要业务上想清楚。批量导入通常配合幂等写入（唯一键 upsert）或者先写临时表再切换，避免解析到一半失败留下脏数据。

 推荐查看英文单词：https://www.tianqi92.com/danci/13055.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13056.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13057.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13058.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13059.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13060.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13061.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13062.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13063.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13064.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13065.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13066.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13067.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13068.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13069.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13070.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13071.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13072.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13073.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13074.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13075.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13076.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13077.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13078.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13079.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13080.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13081.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13082.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13083.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13084.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13085.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13086.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13087.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13088.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13089.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13090.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13091.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13092.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13093.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13094.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13095.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13096.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13097.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13098.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13099.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13100.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13101.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13102.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13103.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13104.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13105.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13106.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13107.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13108.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13109.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13110.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13111.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13112.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13113.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13114.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13115.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13116.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13117.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13118.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13119.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13120.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13121.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13122.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13123.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13124.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13125.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13126.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13127.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13128.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13129.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13130.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13131.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13132.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13133.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13134.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13135.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13136.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13137.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13138.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13139.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13140.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13141.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13142.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13143.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13144.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13145.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13146.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13147.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13148.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13149.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13150.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13151.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13152.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13153.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13154.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13155.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13156.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13157.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13158.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13159.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13160.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13161.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13162.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13163.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13164.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13165.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13166.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13167.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13168.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13169.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13170.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13171.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13172.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13173.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13174.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13175.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13176.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13177.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13178.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13179.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13180.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13181.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13182.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13183.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13184.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13185.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13186.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13187.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13188.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13189.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13190.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13191.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13192.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13193.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13194.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13195.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13196.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13197.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13198.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13199.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13200.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13201.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13202.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13203.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13204.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13205.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13206.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13207.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13208.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13209.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13210.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13211.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13212.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13213.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13214.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13215.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13216.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13217.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13218.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13219.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13220.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13221.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13222.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13223.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13224.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13225.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13226.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13227.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13228.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13229.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13230.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13231.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13232.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13233.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13234.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13235.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13236.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13237.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13238.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13239.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13240.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13241.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13242.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13243.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13244.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13245.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13246.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13247.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13248.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13249.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13250.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13251.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13252.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13253.html
 推荐查看英文单词：https://www.tianqi92.com/danci/13254.html

处理几十兆甚至上百兆的 JSON 文件时，很多 Node.js 项目会直接 fs.readFileSync 读进内存再 JSON.parse，本地小文件没问题，一到生产环境跑批处理就容易遇到 JavaScript heap out of memory，或者把容器内存顶到上限被 OOM Kill。这个场景在日志分析、数据导入、第三方接口导出对账里很常见。根本原因是 JSON.parse 需要先拿到完整字符串，构建出完整对象树才能返回，内存占用大致是文件体积的几倍。这篇文章从实际代码出发，讲清楚怎么用流式解析器逐条消费 JSON 数组元素，怎么在 highWaterMark、背压和异步写入之间取舍，以及哪些情况适合流式、哪些情况老老实实分批读更省事。

Node.js 中流式处理大 JSON 文件：避免内存爆掉的实际做法

处理几十兆甚至上百兆的 JSON 文件时，很多 Node.js 项目会直接 fs.readFileSync 读进内存再 JSON.parse，本地小文件没问题，一到生产环境跑批处理就容易遇到 JavaScript heap out of memory，或者把容器内存顶到上限被 OOM Kill。这个场景在日志分析、数据导入、第三方接口导出对账里很常见。根本原因是 JSON.parse 需要先拿到完

编程语言

后端

开发工具

处理超大JSON文件时，用fs.readFileSync配合JSON.parse会导致JavaScript堆内存溢出和容器OOM Kill。本文介绍流式解析方案：利用stream-json库逐条消费数组元素，通过背压控制内存，支持顶层非数组结构，并对比NDJSON按行读取及worker_threads方案，帮助你在批处理场景中稳定处理百兆级数据。

日志分析

代码解释

数据库

容器

JSON

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

EdgeOne AI 安全实战专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

处理超大JSON文件时，用fs.readFileSync配合JSON.parse会导致JavaScript堆内存溢出和容器OOM Kill。本文介绍流式解析方案：利用stream-json库逐条消费数组元素，通过背压控制内存，支持顶层非数组结构，并对比NDJSON按行读取及worker_threads方案，帮助你在批处...

Node.js 中流式处理大 JSON 文件：避免内存爆掉的实际做法-腾讯云开发者社区-腾讯云

Node.js 中流式处理大 JSON 文件：避免内存爆掉的实际做法

Node.js 中流式处理大 JSON 文件：避免内存爆掉的实际做法

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐