刮动图数据
刮动图数据
提问于 2019-09-12 19:50:14
我想刮的价格鹿特丹- in 380在07/04/2019年( $380.50)在动态图表从上述网站。
如果我想将数据存储到本地数据库中，我不确定是否“漂亮汤”是最好的方法。
from bs4 import BeautifulSoup
import requests
import pymongo
# URL of page to be scraped
url = 'https://shipandbunker.com/prices/emea/nwe/nl-rtm-rotterdam#IFO380'
# Retrieve page with the requests module
response = requests.get(url)
# Create BeautifulSoup object; parse with 'lxml'
soup = BeautifulSoup(response.text, 'lxml')回答 3
Stack Overflow用户
回答已采纳
发布于 2019-09-12 21:28:52
此页面使用JavaScript获取所有数据并创建图形。
JavaScript使用带有url https://shipandbunker.com/a/.json的POST请求来获取JSON格式的数据，这可以很容易地转换成https://shipandbunker.com/a/.json的字典，并且不需要BeautifulSoup来抓取HTML。
import requests
import datetime
day = datetime.date(2019, 7, 4)
payload = {
'api-method': 'pricesForAllSeriesGet',
'resource': 'MarketPriceGraph_Block',
'mc0': 'NL RTM',
'mc1': 'AV G20',
}
url = 'https://shipandbunker.com/a/.json'
r = requests.post(url, data=payload)
#print(r.content)
data = r.json()
for number, value in data['api']['NL RTM']['data']['prices']['IFO380']['dayprice']:
# convert day number to date object
timestamp = data['api']['NL RTM']['data']['day_list']['IFO380'][str(number)]
date = datetime.date.fromtimestamp(timestamp/1000)
if date == day:
print(day, value)
break它显示
2019-07-04 380.5Stack Overflow用户
发布于 2019-09-12 20:12:31
您可以使用Scrapy刮除您想要的每一个站点。
下面是一种简单的爬行器：
网址：IFO380
spiders.py
import scrapy
class ShippingSpider(scrapy.Spider):
name = 'shipping_spider'
start_urls = [
'https://shipandbunker.com/prices/emea/nwe/nl-rtm-rotterdam#_IFO380',
]
def parse(self, response):
xpath = '//*[@id="block_284"]/div/div/div/table/tbody/tr[2]/td[1]/text()'
rotterdam = response.xpath(xpath).extract()
print(rotterdam)
# output: ['315.00']建立和管理蜘蛛真的很容易，这里是文档。
Stack Overflow用户
发布于 2019-09-12 21:44:32
下面是一种在json中搜索从API调用中动态返回的特定日期的方法，您可以在网络选项卡中找到。实现与另一个答案略有不同。
import requests
from datetime import datetime, date
import calendar
def get_timestamp(date_var):
return calendar.timegm(date_var.timetuple()) * 1000
data = [
('api-method', 'pricesForAllSeriesGet'),
('resource', 'MarketPriceGraph_Block'),
('mc0', 'NL RTM'),
('mc1', 'AV G20')]
r = requests.post('https://shipandbunker.com/a/.json', data=data).json()
date_var = get_timestamp(date(2019, 7, 4))
d = r['api']['NL RTM']['data']['day_list']['IFO380']
keys = list(d.keys())
prices = r['api']['NL RTM']['data']['prices']['IFO380']['dayprice']
found = [i for i in range(len(d.keys())) if d[keys[i]] == date_var][0]
print(prices[found][1])页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/57913629
复制相关文章
相似问题
腾讯云开发者
