记入一次爬虫之旅

用户4792657

发布于 2022-07-06 14:12:39

64200

代码可运行

文章被收录于专栏：山海亦有归期山海亦有归期

运行总次数：0

代码可运行

Plain text or HTML

import fetch from 'node-fetch';

const response = await fetch('https://github.com/');
const body = await response.text();

console.log(body);
复制代码

Streams

The "Node.js way" is to use streams when possible. You can pipe res.body to another stream. This example uses stream.pipeline to attach stream error handlers and wait for the download to complete.

import {createWriteStream} from 'node:fs';
import {pipeline} from 'node:stream';
import {promisify} from 'node:util'
import fetch from 'node-fetch';

const streamPipeline = promisify(pipeline);

const response = await fetch('https://github.githubassets.com/images/modules/logos_page/Octocat.png');

if (!response.ok) throw new Error(`unexpected response ${response.statusText}`);

await streamPipeline(response.body, createWriteStream('./octocat.png'));
复制代码

import fetch from 'node-fetch';
import cheerio from 'cheerio';
import {createWriteStream} from 'node:fs';
import {pipeline} from 'node:stream';
import {promisify} from 'node:util'

const response = await fetch('https://mmzztt.com/beauty/page/10');
const body = await response.text();
const $ = cheerio.load(body);
const downloadImg = async (url, index) => {
  const streamPipeline = promisify(pipeline);
  const date = new Date().getFullYear()+'-'+ (new Date().getMonth())+'-'+(new 
 Date().getDate()+1)
  const response = await fetch(url);

  if (!response.ok) throw new Error(`unexpected response ${response.statusText}`);

  await streamPipeline(response.body, createWriteStream(`./img/${date}-${index}.png`));
  console.log(`已爬取第${index}张`)
}
$('.uk-article .uk-inline .u-thumb-f img').each(function (i) {
  downloadImg($(this).attr('data-src'), i)
})

复制代码

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2022-02-16，如有侵权请联系 cloudcommunity@tencent.com 删除

download