网站使用cookie时用curl下载
网站使用cookie时用curl下载
提问于 2022-11-18 13:41:41
我可以从下面的URL下载数据( json响应)，在Windows上通过curl下载一段时间。
curl.exe https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA
或
curl.exe https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA&exchange=
但是最近它停止了工作，反而得到了一个错误html，上面写着
“启用JavaScript和cookies继续”
有什么办法能让这件事再起作用吗？
提前谢谢。
回答 2
Stack Overflow用户
发布于 2022-11-19 04:04:25
我花了几个小时在这个问题上，因为我对Cloudflare 403紫禁风的反应很感兴趣。这已经成为卷发的一个大问题。现在也是你的问题了。
我基本上得到了两种不同的反应。一个来自Win 10 curl (cmd)使用3个不同的请求。
PHP只得到一个403禁止的HTTP状态响应。
这可能会影响到我的curl证书，或者没有它们。
我相信Win 10卷发是与边缘相同的CA设置。
Cloudflare经常拒绝卷发请求。Cloudflare配置文件浏览器TLS握手。如果握手与用户代理不匹配，则不会将请求传递给主机。相反，它返回一个403紫禁城。
如果您搜索"Fix Cloudflare 403禁忌“，就不会缺少信息。
我相信你的问题是因为云彩。
我不认为这与JavaScript有关。
我在边缘和火狐上禁用了JavaScript，并且仍然得到了JSON。
响应头：
HTTP/2 200 OK
date: Sat, 19 Nov 2022 01:18:10 GMT
content-type: text/plain; charset=utf-8
content-encoding: gzip
vary: Accept-Encoding
request-context: appId=cid-v1:30c6c72c-1099-4721-b73f-c69b90d8ae95
strict-transport-security: max-age=2592000; includeSubDomains; preload
cf-cache-status: DYNAMIC
set-cookie: __cf_bm=4i3UCcOc3ABOk_qatlAieY0OkXfECjDOJG_G_Nm.ex0-1668820690-0-Acph3yaZ3B2kSsPbygIqBJubZSuDb+OUkgmIMDycJ4JyRBJq8gv4QCJEirVC9o6WedMi/p6xrc0z+xdBS5J4s4c=; path=/; expires=Sat, 19-Nov-22 01:48:10 GMT; domain=.theocc.com; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 76c51c3f3a952209-MIA
X-Firefox-Spdy: h2响应体中的JSON：
{"Content":[{"Date":"2022-11-18T00:00:00Z","Results":[{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":17,"StrikePriceFrac":500,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":0,"PutOpenInt":5950,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"500"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":20,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":0,"PutOpenInt":1553,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":22,"StrikePriceFrac":500,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":25,"PutOpenInt":3145,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"500"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":25,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":115,"PutOpenInt":2714,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":30,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":263,"PutOpenInt":4645,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":32,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":21,"PutOpenInt":396,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":33,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":44,"PutOpenInt":409,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":34,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":28,"PutOpenInt":145,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":35,"StrikePriceFrac":0,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":271,"PutOpenInt":3760,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"000"},{"ProdSymbolId":"AA ","ExprDate":1668751200000000,"StrikePriceIntg":35,"StrikePriceFrac":500,"CallTickSymbolId":"C","PutTickSymbolId":"P","CallOpenInt":30,"PutOpenInt":101,"PosLimit":25000000,"SecuId":"AA ","CusiId":null,"Name":"Alcoa Corporation ","OnnGrp":"EU","StrPriceFrac":null,"StrikePriceFracStr":"500"},{"ProdSymbolId":"AA 一个Win 10卷曲URL必须以双引号和
有时，必须转义&查询。\&。
我在Win 10 cmd窗口中使用转义的\&尝试了这些请求
curl "https://www.theocc.com/mdapi/series-search?symbol_type=U\&symbol=AA"
curl "https://www.theocc.com/mdapi/series-search?symbol_type=U\&symbol=AA" -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: ARRAffinity=50cf07c4923930f3ead4232e638b6657fb662613b8f01df70fb38ac353b8e3d6; ARRAffinitySameSite=50cf07c4923930f3ead4232e638b6657fb662613b8f01df70fb38ac353b8e3d6' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1'
curl "https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA" ^
-H "authority: www.theocc.com" ^
-H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
-H "accept-language: en-US,en;q=0.9" ^
-H "cache-control: max-age=0" ^
-H "cookie: ARRAffinity=50cf07c4923930f3ead4232e638b6657fb662613b8f01df70fb38ac353b8e3d6; ARRAffinitySameSite=50cf07c4923930f3ead4232e638b6657fb662613b8f01df70fb38ac353b8e3d6; __cf_bm=FS_Oi_E3bPH7QBQUW.XzcyQDP6PFkL8RHPbA0qbdj8I-1668824712-0-AfSfUDI6XgJiQRrzEHCihbO2DP1VUL1Y9fkmzArjKYqZ356bWDu0eSuUlxkJoAc+N7nVyOBw4itGMsGRSuYf6oc=" ^
-H "sec-fetch-dest: document" ^
-H "sec-fetch-mode: navigate" ^
-H "sec-fetch-site: none" ^
-H "sec-fetch-user: ?1" ^
-H "upgrade-insecure-requests: 1" ^
-H "user-agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident/6.0; IEMobile/10.0; ARM; Touch; NOKIA; Lumia 920) UCBrowser/10.1.0.563 Mobile" ^
--compressed来自Firefox网络的第二个请求凸轮作为curl复制(POSIX)
第三个请求从边缘网络工具复制为curl (cmd)。但是，在我删除所有^cr/lf之前，它是行不通的。
胜利10卷曲的反应，对所有三个请求相同。
下面的回应来自Cloudflare。
根据响应头，我不认为这些请求是传递给主机theocc.com的。
响应标头
HTTP/1.1 403 Forbidden
Date: Sat, 19 Nov 2022 04:17:29 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
CF-Chl-Bypass: 1
Referrer-Policy: same-origin
Permissions-Policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
X-Frame-Options: SAMEORIGIN
Set-Cookie: __cf_bm=jNH8U9rGVke8A3M9NVoGyM_54P4yqbD6GUdBex2U9as-1668831449-0-Aajlm5as+JsCCdzuStWeD6FNiZE6Q7BqSSGV+axg1GOcbz3CREOYW+gy0pq0iJpVVZ5HRo2RQA3AQaaFBjNnIsk=; path=/; expires=Sat, 19-Nov-22 04:47:29 GMT; domain=.theocc.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
CF-RAY: 76c622f05c20d9d1-MIA响应HTML
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Just a moment...</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<meta name="robots" content="noindex,nofollow">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
</head>
<body class="no-js">
<div class="main-wrapper" role="main">
<div class="main-content">
<h1 class="zone-name-title h1">
<img class="heading-favicon" src="/favicon.ico"
onerror="this.onerror=null;this.parentNode.removeChild(this)">
www.theocc.com
</h1>
<h2 class="h2" id="challenge-running">
Checking if the site connection is secure
</h2>
<noscript>
<div id="challenge-error-title">
<div class="h2">
<span class="icon-wrapper">
<div class="heading-icon warning-icon"></div>
</span>
<span id="challenge-error-text">
Enable JavaScript and cookies to continue
</span>
</div>
</div>
</noscript>
<div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=76c5e617baf0221a')"></div>
<div id="challenge-body-text" class="core-msg spacer">
www.theocc.com needs to review the security of your connection before proceeding.
</div>
<form id="challenge-form" action="/mdapi/series-search?symbol_type=U\&symbol=AA&__cf_chl_f_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0" method="POST" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="md" value="C_LKeu2P5NrY0aR4BxvKBGqNiMKJ1potyy02pUTruns-1668828957-0-ASvIMrC-30PITC58EAtdhq56Jx7fME4EOUOUE-FMk5FBMP_d7Pfr7XFI_Qr6SAVlPAQCGA9_b9beL_agWaV1I44NJRLSb5IkRQ3E6icwIe8Eq7QVMQWYJ-KlvST_JI7znFj_MeqOMA50UlrgSYomFz8TuMTnjbbiiYo9FziI0gyhbYedZ0BtSiUakd9eMeV54maAR1UqybWW52Lj1qpMUZtAes6YaVvVNcjVLtBtjt5Dbspqa8XsoRwpk11VO5GnlLoz4uSqh-laMjyd6zOas7YCD7Qzs05AGIBGKLuPZ3xeyq43ZS0zcGLkxsxwgbF1IWdBz8br6tVuO6YFvY5qSsEeXvymMXbCltsj8tqKDb-XoCbfyFF_Gb33LwFmxYudOqrvZ-2miy-BbZAGrSRGe7_nNDfKf94f_ZRrpSu25VvNhvni46u1AUa4v4wy7O6ujeT9JHpwem_0Y2RUw5QJywl8ZXSFf9yGkYq9us8Dwbvkhcis7ovToZv3eP2Gf0MKjgSIj0twbMVIguUCrluO5vVhdzWu1HJfaZI0htq0Rn8ZWD9g0XF5amnaWs80iXocIIsJKPEhHDuh5jpNOH0suaXfZm7waukxfDxRUURM6Kg4BEEaJrNH0ZAsLLvaDfaQ-XCgr_s2VFRdLEQHMEScQ76P271i_MfhA23Xv0uSlLwPRjcnrrYhhfAOoYXMxUqMBjntLp52kLxAgBlV7fD4l8xe-JNpKYWaqi5IXQKLn87mG3XV4SCeeysfvhT26n4CHWXlgroGtv6AwcGtIR6Hkvg">
<input type="hidden" name="r" value="NEE5A4XvbVzMlgGmfv94hrNQy8HlFwlN9MvsAnrBqEc-1668828957-0-AU6LelP5fDUpZVzOa9xe8l/70P6WeCebWpPrNZ5DR1Chl+2RZwXwTYc2PhlpBrSEc527YLgU+biTV2VNZunN7E3l1/kL6wiimvLxzx46KIUC5IXDcXgDJ3uXX5P3eO/x5OE2RSxI4N07jV1pfmsKwsYf0hJtJzCraW+rZt7/bVZVBmvFrYOSBTqI5jmZ+Lsf4gF+kLRBLKoheLKgvvblO8GLixR6SYt56sv8Hp9yS/DWmDoPC3yx8Z7DwvRFkpuvLdOCgqcTGNHGDfKjWQOwNFA1yB0Sc/6+8qD4mQgSPTMlPYdwiMgfERL9aHQy2A8/GQztGizyp+nAceOiUhFripvog+p3MWYdVNzMD4odVVakRdp/wva0LVzt8rtj1G5FBh6wVA94kCxlDRO6M7pYKQjz0c7ejdYUnFtQYFn4pR9B7H3eZNc2vIIJdTQpyxHNcIbjBWSTn5Bv9aZX/JrXZQE8bxK1PKD9wQtSy2k60cuotYOGmrwjNS4Jgk4bt3SaytN0t02mdvOZ6ZuLK9soRGqIN9iJvZUcU3TCvC+pZdcm9x7DXTC9LCSjzkQWQygXwOAfqrf6floZcdmlJIzr6S2XgPN3OAy6Mz9hkc08Vu6qtAYjI2j1igUjLFZw07MIeluznsZCyLipxf/2LWpqMlotab3q8yT9W9WY8480IBsErIOmJ3c+q2aptTpJM+Cuayb9896SOl9niJP5m9UuwYdiyExE+DieDGR2SIuAuzFxU2XuEgC1mgJmvcpkh+jTlcvE/6rHbQBtZdY5xyMUSJI0u0G187QdjEYlGkTX7nL7t+EKnK/bJCz0qFI/JWEBwY/j3OVAL/0DzI9H3d+r6GnxqZZCUEwfBsl/3kLf6PdKzrtP+otybkpG+RvBd3lfuNx06hRVxmtfVAR3R3WfOCo9o8XOMORG1uvGPggT0DBBVJE8TKv0tbIyE3drvyRjJ4KqLxpPwYB2D/7rwDssEVlZhXf0AAu26R8ajqKWcjy/17tyX4tK8ERknmutokwP/hB4b2VEarMK4XgEapf7alPJvY7KB4NQ694DepNXMnhUncUTynw1T1H+T7LTGaSFn9gcZFcNOPBWzIMElyUjfrB6xJEwG+PcgitK9IWKZkHanvSeTTbPdkaCt6tEhk0LlR0IVPUCLhxUOKeo/1hLWah6oZU6LtF+lHVJ0CF/pNK+YnhREmbUaESoeiRs4Ill2CjS+69hl5CAuLtbCXdAoyZUHyBh3wrJBA+u8EiqSZdKlXfxR85Y8jOO7td2XqS1tVDCLc4VODyXsjQ2CP2GKTThzlySl4s4aYlefAYb5/DDaFBnM1AKHM2TX/Sl90Pyzs7wAfx/UG9V1PW+qNBbhizXJDyxYFBdRjK/dR0Mm4Diau5bv+fbyfa1uWW6wywvl6+HwrCbMi7T2fdWzGUwhmyDZzqTlUBHB4ZR8eU0puljriB3uEeH9w11JK/2qQCviYRkvqfCm32lDPel4PIIAJlUb4qdFPyg0b7Faazh/UjyyS+hK23EM1HVUCVX+QY/aYj9dnzSIBxcy0UQDwmK8TpfcIPXD9iyYzrCQVCaa+e5OtSP5ZJwrEX53f5vn/zFJWo0q4xUTMR5+mmrG+QTVbfpRlRcbz3tI5qIQGUpSlg/MKSUNqernPf9b49FsHQM2NvssW/pA3XiJZQrVL2L8Xq/Ed0VbCIufQdOSvf1nQntUnGzKMbwordtTmsGA8z4Pogv5C7AoQcpPPbSNgiQL5D1Nvlz6Y1IlnG5ovU51a69ptkyX/CY7bFugJ66djMp9vFF2xrIVp0r1arEPvYzpS1DcVj2WQTdaWtXvj8OEAykHuBu1tD3UgtmAdEPE1aftX36eOhfveVgOUXhc47ObpQS54Vbl5vXPbIfdi3pnSEKyZY3CUtd8pNb7OhqqRw5UeRTN47JDhJITP5J4iLz+GiPK7t2ORqDsTnm6T5s3/tksp/pa8/v+ooA9hjSmkEssBgIShffOAUEwyIXJesXUoNx83pJyoHYcXoZI0ZfHsNNRoFn+9F+TckR0U//xKdeNbkCbyXjJrB7yTOvKGolqx0LuAkx/SKW1isTCAdhs3CvbQNxb/GxlWi8D1PzUwVJeQ==">
</form>
</div>
</div>
<script>
(function(){
window._cf_chl_opt={
cvId: '2',
cType: 'managed',
cNounce: '32555',
cRay: '76c5e617baf0221a',
cHash: '7592d66e2323e80',
cUPMDTk: "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0",
cFPWv: 'b',
cTTimeMs: '1000',
cTplV: 4,
cTplB: 'cf',
cRq: {
ru: 'aHR0cHM6Ly93d3cudGhlb2NjLmNvbS9tZGFwaS9zZXJpZXMtc2VhcmNoP3N5bWJvbF90eXBlPVVcJnN5bWJvbD1BQQ==',
ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwNy4wLjAuMCBTYWZhcmkvNTM3LjM2IEVkZy8xMDcuMC4xNDE4LjQy',
rm: 'R0VU',
d: 'HOWi54+55wh9FM6AuZDX9ANcT34bxF2qiuKEraFDkFFxVWfBHI0H0RQrt4iKgIqp/rTy8AJZBTvoHTeHVUHEU8cHb4I9k9KfbQsqkWEYZbs82TTZ7KS7RE4eFUiXiDSp7Y0QjRU9BL8II412SDZ45C5FFFg8HXvVVScSCJoDt+ARF1Ew3Jaw7unS6jp/2QqADMAF7HjZDag23KMOyCqhfXHrFkopkGie3S8ReM8KDMpt/WINtw/ZIztXsz/1gZvG9JYWW+V8qsmBzlVrIHcoD70dm38ABF5ki4OZ34mXVsMrah0cRovMSqbMnbHDiOGr3lRnmAhcALHm8fzgAekQ1s95t7GfNd9mz+Yextk5Ifx/XiwdGfksP67UBE+svAbNUF/XtVzhCNqQjt6T025f+DXfYNRCVK5i6ZpkHcZzndk+x2uQO6lpzqa5w2yrPr7SMlbD+b5vE5XZvNkHTchXBd74S0N68QjUIdxdrS679dCxfrv06qsk57teE5USCsYaxQNhj5n53vDx2FRTWlIJfo2vqHlzyXK2zFAry5MIEjmfYq9FYdciS5ngRM3AxTWA36so3+7syHH4uk3SZb5JQ579YnsmXEiA1ZGzN1h8Vpb9b0g2utYK6QYeRdyN7QZphZZ1Vzt/mMYIA96uakzHfNn3357jDIrZ3kQQDX7NXNo=',
t: 'MTY2ODgyODk1Ny40MDQwMDA=',
m: '/aGrMgSdZO6ogXdUQEEOQsDgNn+UAxlig04+B+a9YQY=',
i1: 'wffO0nID3av5tavW3mXdtA==',
i2: 'eIr4tQZjzWSBg3EFfM1UhA==',
zh: 'rjV/vZpjqak/inbeUs7nHF3bR1cMk2CD6n2tHAx8WJg=',
uh: 'p19SMC98WhDwvsQz+Kqkj/EMDYzdbqDnaATK9trmhK0=',
hh: 'rTcMqa1eOqviz82idTtNQqeGJGdhBGQmQRzkYbDEzA4=',
}
};
var trkjs = document.createElement('img');
trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=76c5e617baf0221a');
trkjs.setAttribute('style', 'display: none');
document.body.appendChild(trkjs);
var cpo = document.createElement('script');
cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/managed/v1?ray=76c5e617baf0221a';
window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
if (window.history && window.history.replaceState) {
var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
history.replaceState(null, null, "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_rt_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0" + window._cf_chl_opt.cOgUHash);
cpo.onload = function() {
history.replaceState(null, null, ogU);
};
}
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
</script>
<div class="footer" role="contentinfo">
<div class="footer-inner">
<div class="clearfix diagnostic-wrapper">
<div class="ray-id">Ray ID: <code>76c5e617baf0221a</code></div>
</div>
<div class="text-center">Performance & security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
</div>
</div>
</body>
</html>我还使用了PHP的curl。
$headers = array();
//$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0';
$headers[] = 'User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0';
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8';
$headers[] = 'Accept-Language: en-US,en;q=0.5';
$headers[] = 'Accept-Encoding: gzip, deflate, br';
$headers[] = 'DNT: 1';
$headers[] = 'Connection: keep-alive';
$headers[] = 'Upgrade-Insecure-Requests: 1';
$headers[] = 'Sec-Fetch-Dest: document';
$headers[] = 'Sec-Fetch-Mode: navigate';
$headers[] = 'Sec-Fetch-Site: none';
$headers[] = 'Sec-Fetch-User: ?1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
echo "Response: $response\n";
echo "\nheader out: \n" . curl_getinfo($ch,CURLINFO_HEADER_OUT);
echo "curl Error: \n" . curl_error($ch);
echo "\n\ncurl_get_info:\n";
var_export(curl_getinfo($ch));上面的PHP代码没有返回响应头或响应主体，只返回403禁止的HTTP状态
Response:
header out:
GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
curl Error:
The requested URL returned error: 403
curl_get_info:
array (
'url' => 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA',
'content_type' => 'text/html; charset=UTF-8',
'http_code' => 403,
'header_size' => 954,
'request_size' => 470,
'filetime' => -1,
'ssl_verify_result' => 20,
'redirect_count' => 0,
'total_time' => 0.082475,
'namelookup_time' => 0.000513,
'connect_time' => 0.016717,
'pretransfer_time' => 0.039101,
'size_upload' => 0.0,
'size_download' => 0.0,
'speed_download' => 0.0,
'speed_upload' => 0.0,
'download_content_length' => -1.0,
'upload_content_length' => 0.0,
'starttransfer_time' => 0.0824,
'redirect_time' => 0.0,
'redirect_url' => '',
'primary_ip' => '2606:4700:90:0:3d7d:1d0b:b681:7270',
'certinfo' =>
array (
),
'primary_port' => 443,
'local_ip' => '2604:4500:0:8d:ec4:7aff:fe76:281a',
'local_port' => 46564,
'request_header' => 'GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
',
)我不知道这是否有帮助，但这个URL工作在Win 10 cmd中。
并在PHP中工作。
注意:转义的\&没有在这个URL上工作。
这不是在Cloudflare上托管的。
>curl "https://marketdata.theocc.com/series-search?symbolType=U&symbol=AA"响应：
Series/contract Strike Open Interest
ProductSymbol year Month Day Integer Dec C/P Call Put Position Limit
AA 2022 11 18 17 500 C P 0 5950 25000000
AA 2022 11 18 20 000 C P 0 1553 25000000
AA 2022 11 18 22 500 C P 0 3145 25000000
AA 2022 11 18 25 000 C P 0 2714 25000000
AA 2022 11 18 30 000 C P 0 4645 25000000
AA 2022 11 18 32 000 C P 0 396 25000000
AA 2022 11 18 33 000 C P 0 409 25000000
AA 2022 11 18 34 000 C P 0 145 25000000
AA 2022 11 18 35 000 C P 0 3760 25000000
AA 2022 11 18 35 500 C P 0 101 25000000
AA 2022 11 18 36 000 C P 0 211 25000000
AA 2022 11 18 36 500 C P 0 163 25000000Stack Overflow用户
发布于 2022-11-18 13:47:40
要回答你问题的标题：
curl -b cookies -c cookies <URL>但是如果javascript是强制性的，它就会失败。Curl没有javascript功能
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/74490577
复制相关文章
相似问题
腾讯云开发者
