python scrapy 模拟登录(最基础)

l=ItemLoader(item=xxxItem(),response=response) l.add_xpath('title','//xxx',MapCompose(str.strip,str.title)) MapCompose(float) #turn to float l.add_value('title',response.url) l.load_item() start_URL=[i.strip() for i in open('xxx').readlines()] 1、scrapy startproject loginscrapy cd loginscrapy scrapy genspider -t basic loginspider example 2、设置setting.py:添加 USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5' 3、vi spider/loginspider

-- coding: utf-8 --

import scrapy from scrapy import Request,FormRequest

class BasicloginSpider(scrapy.Spider): name = 'basiclogin' allowed_domains = ['example.webscraping.com']

start_urls = ['http://example.webscrping.com/']

def start_requests(self):
    return [Request("http://example.webscraping.com/places/default/user/login",callback=self.login,meta={"cookiejar":1})]
def login(self, response):
    data={"email":"liushuo@webscraping.com","password":"12345678"}
    return [FormRequest.from_response(response,meta={"cookiejar":response.meta["cookiejar"]},callback=self.parse)]
def parse(self,response):
    print("logined")
    print(response.url)

登录页面只有用户名和密码，没有验证码，且里面有form元素，可以通过formid来指定用哪个form