V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
xiaoheijw
V2EX  ›  Python

scarpy 爬虫存入数据库报错 'YysItem' object is not callable 执行 pipelines 程序时报错。

  •  
  •   xiaoheijw · 2017-07-28 15:47:49 +08:00 · 2409 次点击
    这是一个创建于 2710 天前的主题,其中的信息可能已经有所发展或是发生改变。

    请大神帮忙解惑 爬虫主代码

    #coding:utf-8
    
    import scrapy,json
    from yys.items import YysItem
    from scrapy.selector import  Selector #选择器
    from .get_urls import get_url
    
    class Yyspider(scrapy.Spider):
        name='yys'
        allow_domain=['comp-sync.webapp.163.com']
        start_urls=get_url()
    
        def parse(self, response):
            items=[]
            item=YysItem()
            jsonresponse = json.loads(response.body_as_unicode())
            # print (jsonresponse)
            try:
                for j in jsonresponse['data']:
                    item['id']=j['req_id']
                    item['time']=j['get_time']
                    item['whi']=j['prop_info']['from']
                    item['level']=j['prop_info']['prop_name'].split("式神")[0]
                    item['name']=j['prop_info']['prop_name'].split("式神")[-1]
                    item['nick']=j['user_info']['nick']
                    item['server']=j['user_info']['server']
                    item['uid']=j['user_info']['uid']
                    yield item
            except Exception as e:
                print (e)
    
    
    

    pipelines 程序

    # -*- coding: utf-8 -*-
    
    # Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
    import pymysql
    from yys import settings
    from yys.items import YysItem #数据库结构
    class YysPipeline(object):
        def __init__(self):
            self.connect = pymysql.connect(
                host=settings.MYSQL_HOST,
                db=settings.MYSQL_DBNAME,
                user=settings.MYSQL_USER,
                passwd=settings.MYSQL_PASSWD,
                port=settings.MYSQL_PORT,
                charset='utf8',
                use_unicode=True)
            self.cursor = self.connect.cursor()
    
        def process_item(self, item, spider):
            if item.__class__ == YysItem:
                try:
                    print ('执行 sql')
                    # print (item)
                    insert_sql= "insert into yys values(%s,%s,%s,%s,%s,%s,%s,%d)"
                    print (insert_sql)
                    print (item['id'])
                    self.cursor.execute(insert_sql,(item['id'],item['time'],item['whi'],item['level'],item['name'],item['nick'],item('server'),item['uid']))
                    self.connect.commit()
                except Exception as e:
                    print (e)
                return item
    
    

    item.py

    import scrapy
    
    
    class YysItem(scrapy.Item):
        id=scrapy.Field()
        time=scrapy.Field()
        whi=scrapy.Field()
        level=scrapy.Field()
        name=scrapy.Field()
        nick=scrapy.Field()
        server=scrapy.Field()
        uid=scrapy.Field()
    
    
    knightdf
        1
    knightdf  
       2017-07-28 16:04:32 +08:00
    item('server')
    mark06
        2
    mark06  
       2017-07-28 16:09:06 +08:00
    楼上正解
    xiaoheijw
        3
    xiaoheijw  
    OP
       2017-07-28 16:33:45 +08:00
    @knightdf 智障了,丢人,感谢。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1610 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 25ms · UTC 16:50 · PVG 00:50 · LAX 08:50 · JFK 11:50
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.