python抓取京东联盟优惠券_[爬虫]使用python抓取京东全站数据（商品 店铺 分类 评论）...

网上抓取京东数据的文章，现在要么无法抓取数据，要么只能抓取部分数据，本文将介绍如何抓取京东全站数据，包括商品信息、店铺信息，评论信息，分类信息等。

-------------------------------------------------------------------------------

一、环境OS：win10

python：3.5

scrapy：1.3.2

pymongo：3.2

pycharm

环境搭建，自行百度

二、数据库说明

1. 产品分类

京东大概有1183个分类，这是除去了一些虚拟产品(话费、彩票、车票等)的分类，可以到如下网页查看：

/allSort.aspx

我们也是从这个网址开始抓取。由于这些分类里面也有属于频道的页面，也就是说，这个分类里面也有很多子分类，需要做一些特殊处理才可以拿到所有分类，具体方法，下文再说。

name #分类名称

url #分类url

_id #分类id

2. 产品

url #产品url

_id #产品id

category #产品分类

reallyPrice #产品价格

originalPrice #原价

description #产品描述

shopId #shop id

venderId #vender id

commentCount #评价总数

goodComment #好评数

generalComment #中评数

poolComment #差评数

favourableDesc1 #优惠描述1

favourableDesc2 #优惠描述2

3. 评论

_id #评论id

productId #产品id

guid

content #评论内容

creationTime #评论时间

isTop

referenceId

referenceName

referenceType

referenceTypeId

firstCategory

secondCategory

thirdCategory

replyCount #回复次数

score #分数

status

title

usefulVoteCount #被标记的有用评论数

uselessVoteCount #被标记的无用评论数

userImage

userImageUrl

userLevelId

userProvince

viewCount

orderId #订单id

isReplyGrade

nickname #评论人的名称

userClient

mergeOrderStatus

discussionId

productColor

productSize

imageCount #评论中图片的数量

integral

userImgFlag

anonymousFlag

userLevelName

plusAvailable

recommend

userLevelColor

userClientShow

isMobile #是否移动端评论

days

afterDays #追加评论数

4. 店铺

_id #店铺名称

name #店铺名称

url1 #店铺url1

url2 #店铺url2

shopId #shop id

venderId #vender id

5. 评论总结

_id

goodRateShow #好评率

poorRateShow #差评率

poorCountStr #差评数字符串

averageScore #平均分

generalCountStr #中评数字符串

showCount

showCountStr

goodCount #好评数

generalRate #中评率

generalCount #中评数

skuId

goodCountStr #好评数字符串

poorRate #差评率

afterCount #追评数

goodRateStyle

poorCount

skuIds

poorRateStyle

generalRateStyle

commentCountStr

commentCount

productId #产品id

afterCountStr

goodRate

generalRateShow

jwotestProduct

maxPage

score

soType

imageListCount

三、抓取说明

1. 抓取分类

代码如下：

def parse_category(self, response):

"""获取分类页"""

selector = Selector(response)

try:

texts = selector.xpath('//div[@class="category-item m"]/div[@class="mc"]/div[@class="items"]/dl/dd/a').extract()

for text in texts:

items = re.findall(r'(.*?)', text)

for item in items:

if item[0].split('.')[0][2:] in key_word:

if item[0].split('.')[0][2:] != 'list':

yield Request(url='https:' + item[0], callback=self.parse_category)

else:

categoriesItem = CategoriesItem()

categoriesItem['name'] = item[1]

categoriesItem['url'] = 'https:' + item[0]

categoriesItem['_id'] = item[0].split('=')[1].split('&')[0]

yield categoriesItem

yield Request(url='https:' + item[0], callback=self.parse_list)

except Exception as e:

print('error:', e)

如前文所说，有些类别里面包含有很多子类别，所以对于这样的url，需要再次进行类别抓取：

if item[0].split('.')[0][2:] != 'list':

yield Request(url='https:' + item[0], callback=self.parse_category)

2. 抓取产品

访问每个类别的url就可以获取得到产品列表，找到产品的URL，进入详情页面抓取产品的详情：

def parse_list(self, response):

"""分别获得商品的地址和下一页地址"""

meta = dict()

meta['category'] = response.url.split('=')[1].split('&')[0]

selector = Selector(response)

texts = selector.xpath('//*[@id="plist"]/ul/li/div/div[@class="p-img"]/a').extract()

for text in texts:

items = re.findall(r'', text)

yield Request(url='https:' + items[0], callback=self.parse_product, meta=meta)

产品的基本信息在详情页面基本可以获取，但是有些信息，比如：价格、优惠政策等信息，是需要动态获取的。

先来看价格信息，访问的URL格式为：

/prices/mgets?skuIds=J_(product_id)

这个url最后括号里面的信息就是产品的id，需要动态获取，代码如下：

response = requests.get(url=price_url + product_id)

price_json = response.json()

productsItem['reallyPrice'] = price_json[0]['p']

productsItem['originalPrice'] = price_json[0]['m']

获取得到的都是json格式，比较好解析。

再来看优惠信息，优惠信息分为两种：优惠券和满减描述：

所以需要抓取这两种信息，都是动态加载，代码如下：

# 优惠

res_url = favourable_url % (product_id, shop_id, vender_id, category.replace(',', '%2c'))

# print(res_url)

response = requests.get(res_url)

fav_data = response.json()

if fav_data['skuCoupon']:

desc1 = []

for item in fav_data['skuCoupon']:

start_time = item['beginTime']

end_time = item['endTime']

time_dec = item['timeDesc']

fav_price = item['quota']

fav_count = item['discount']

fav_time = item['addDays']

desc1.append(u'有效期%s至%s,满%s减%s' % (start_time, end_time, fav_price, fav_count))

productsItem['favourableDesc1'] = ';'.join(desc1)

if fav_data['prom'] and fav_data['prom']['pickOneTag']:

desc2 = []

for item in fav_data['prom']['pickOneTag']:

desc2.append(item['content'])

productsItem['favourableDesc1'] = ';'.join(desc2)

3. 抓取店铺信息

在每个产品的详情页面都可以直接找到店铺id和vender id：

ids = re.findall(r"venderId:(.*?),\s.*?shopId:'(.*?)'", response.text)

if not ids:

ids = re.findall(r"venderId:(.*?),\s.*?shopId:(.*?),", response.text)

vender_id = ids[0][0]

shop_id = ids[0][1]

店铺的名称比较难取，有多种不同页面，店铺标题也在不同地方，而且自营产品，在详情页面也可以店铺名称，代码如下：

try:

name = response.xpath('//ul[@class="parameter2 p-parameter-list"]/li/a//text()').extract()[0]

except:

try:

name = response.xpath('//div[@class="name"]/a//text()').extract()[0].strip()

except:

try:

name = response.xpath('//div[@class="shopName"]/strong/span/a//text()').extract()[0].strip()

except:

try:

name = response.xpath('//div[@class="seller-infor"]/a//text()').extract()[0].strip()

except:

name = u'京东自营'

4. 抓取评论

评论的信息也是动态加载，返回的格式也是json，访问url格式为：

/comment/productPageComments.action?productId=(product_id)&score=0&sortType=5&page=%s&pageSize=10

只需要产品的ID即可。

获取评论信息代码如下：

"""获取商品comment"""

try:

data = json.loads(response.text)

except Exception as e:

print('get comment failed:', e)

return None

product_id = response.meta['product_id']

commentSummaryItem = CommentSummaryItem()

commentSummary = data.get('productCommentSummary')

commentSummaryItem['goodRateShow'] = commentSummary.get('goodRateShow')

commentSummaryItem['poorRateShow'] = commentSummary.get('poorRateShow')

commentSummaryItem['poorCountStr'] = commentSummary.get('poorCountStr')

commentSummaryItem['averageScore'] = commentSummary.get('averageScore')

commentSummaryItem['generalCountStr'] = commentSummary.get('generalCountStr')

commentSummaryItem['showCount'] = commentSummary.get('showCount')

commentSummaryItem['showCountStr'] = commentSummary.get('showCountStr')

commentSummaryItem['goodCount'] = commentSummary.get('goodCount')

commentSummaryItem['generalRate'] = commentSummary.get('generalRate')

commentSummaryItem['generalCount'] = commentSummary.get('generalCount')

commentSummaryItem['skuId'] = commentSummary.get('skuId')

commentSummaryItem['goodCountStr'] = commentSummary.get('goodCountStr')

commentSummaryItem['poorRate'] = commentSummary.get('poorRate')

commentSummaryItem['afterCount'] = commentSummary.get('afterCount')

commentSummaryItem['goodRateStyle'] = commentSummary.get('goodRateStyle')

commentSummaryItem['poorCount'] = commentSummary.get('poorCount')

commentSummaryItem['skuIds'] = commentSummary.get('skuIds')

commentSummaryItem['poorRateStyle'] = commentSummary.get('poorRateStyle')

commentSummaryItem['generalRateStyle'] = commentSummary.get('generalRateStyle')

commentSummaryItem['commentCountStr'] = commentSummary.get('commentCountStr')

commentSummaryItem['commentCount'] = commentSummary.get('commentCount')

commentSummaryItem['productId'] = commentSummary.get('productId') # 同ProductsItem的id相同

commentSummaryItem['_id'] = commentSummary.get('productId')

commentSummaryItem['afterCountStr'] = commentSummary.get('afterCountStr')

commentSummaryItem['goodRate'] = commentSummary.get('goodRate')

commentSummaryItem['generalRateShow'] = commentSummary.get('generalRateShow')

commentSummaryItem['jwotestProduct'] = data.get('jwotestProduct')

commentSummaryItem['maxPage'] = data.get('maxPage')

commentSummaryItem['score'] = data.get('score')

commentSummaryItem['soType'] = data.get('soType')

commentSummaryItem['imageListCount'] = data.get('imageListCount')

yield commentSummaryItem

for hotComment in data['hotCommentTagStatistics']:

hotCommentTagItem = HotCommentTagItem()

hotCommentTagItem['_id'] = hotComment.get('id')

hotCommentTagItem['name'] = hotComment.get('name')

hotCommentTagItem['status'] = hotComment.get('status')

hotCommentTagItem['rid'] = hotComment.get('rid')

hotCommentTagItem['productId'] = hotComment.get('productId')

hotCommentTagItem['count'] = hotComment.get('count')

hotCommentTagItem['created'] = hotComment.get('created')

hotCommentTagItem['modified'] = hotComment.get('modified')

hotCommentTagItem['type'] = hotComment.get('type')

hotCommentTagItem['canBeFiltered'] = hotComment.get('canBeFiltered')

yield hotCommentTagItem

for comment_item in data['comments']:

comment = CommentItem()

comment['_id'] = comment_item.get('id')

comment['productId'] = product_id

comment['guid'] = comment_item.get('guid')

comment['content'] = comment_item.get('content')

comment['creationTime'] = comment_item.get('creationTime')

comment['isTop'] = comment_item.get('isTop')

comment['referenceId'] = comment_item.get('referenceId')

comment['referenceName'] = comment_item.get('referenceName')

comment['referenceType'] = comment_item.get('referenceType')

comment['referenceTypeId'] = comment_item.get('referenceTypeId')

comment['firstCategory'] = comment_item.get('firstCategory')

comment['secondCategory'] = comment_item.get('secondCategory')

comment['thirdCategory'] = comment_item.get('thirdCategory')

comment['replyCount'] = comment_item.get('replyCount')

comment['score'] = comment_item.get('score')

comment['status'] = comment_item.get('status')

comment['title'] = comment_item.get('title')

comment['usefulVoteCount'] = comment_item.get('usefulVoteCount')

comment['uselessVoteCount'] = comment_item.get('uselessVoteCount')

comment['userImage'] = 'http://' + comment_item.get('userImage')

comment['userImageUrl'] = 'http://' + comment_item.get('userImageUrl')

comment['userLevelId'] = comment_item.get('userLevelId')

comment['userProvince'] = comment_item.get('userProvince')

comment['viewCount'] = comment_item.get('viewCount')

comment['orderId'] = comment_item.get('orderId')

comment['isReplyGrade'] = comment_item.get('isReplyGrade')

comment['nickname'] = comment_item.get('nickname')

comment['userClient'] = comment_item.get('userClient')

comment['mergeOrderStatus'] = comment_item.get('mergeOrderStatus')

comment['discussionId'] = comment_item.get('discussionId')

comment['productColor'] = comment_item.get('productColor')

comment['productSize'] = comment_item.get('productSize')

comment['imageCount'] = comment_item.get('imageCount')

comment['integral'] = comment_item.get('integral')

comment['userImgFlag'] = comment_item.get('userImgFlag')

comment['anonymousFlag'] = comment_item.get('anonymousFlag')

comment['userLevelName'] = comment_item.get('userLevelName')

comment['plusAvailable'] = comment_item.get('plusAvailable')

comment['recommend'] = comment_item.get('recommend')

comment['userLevelColor'] = comment_item.get('userLevelColor')

comment['userClientShow'] = comment_item.get('userClientShow')

comment['isMobile'] = comment_item.get('isMobile')

comment['days'] = comment_item.get('days')

comment['afterDays'] = comment_item.get('afterDays')

yield comment

if 'images' in comment_item:

for image in comment_item['images']:

commentImageItem = CommentImageItem()

commentImageItem['_id'] = image.get('id')

commentImageItem['associateId'] = image.get('associateId') # 和CommentItem的discussionId相同

commentImageItem['productId'] = image.get('productId') # 不是ProductsItem的id，这个值为0

commentImageItem['imgUrl'] = 'http:' + image.get('imgUrl')

commentImageItem['available'] = image.get('available')

commentImageItem['pin'] = image.get('pin')

commentImageItem['dealt'] = image.get('dealt')

commentImageItem['imgTitle'] = image.get('imgTitle')

commentImageItem['isMain'] = image.get('isMain')

yield commentImageItem

# next page

for i in range(1, int(data['maxPage'])):

url = comment_url % (product_id, str(i))

meta = dict()

meta['product_id'] = product_id

yield Request(url=url, callback=self.parse_comments2, meta=meta)

5. 抓取过程

基本代码已经在文中贴出，写的比较乱，欢迎大家一起讨论。

如果觉得《python抓取京东联盟优惠券_[爬虫]使用python抓取京东全站数据（商品 店铺 分类 评论）...》对你有帮助，请点赞、收藏，并留下你的观点哦！