Scrapy settings

Before starting Scrapy, is recommended that you modify the settings and limit the speed at which the data is accessed, so as not to create a DOS attack. For doing this task, we need to configure settings.py with the DOWNLOAD_DELAY property:

# Scrapy settings for scrapy project
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
#
BOT_NAME = 'hacker_news'
SPIDER_MODULES = ['hacker_news.spiders']
NEWSPIDER_MODULE = 'hacker_news.spiders'
# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
DOWNLOAD_DELAY = 60
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset