Now, you may execute your recipe by going to the folder, RequestsThrottler-0.1.0/requests_throttler.
$ cd RequestsThrottler-0.1.0/requests_throttler
Make sure to give the full addresses as the command-line argument. For example, provide http://cnn.com. Not cnn.com. Otherwise, you will receive an error message similar to the one shown following:
requests.exceptions.MissingSchema: Invalid URL 'cnn.com': No schema supplied. Perhaps you meant http://cnn.com?
Run the recipe using the following command:
$ python 20_10_requests_throttling.py --address="http://cnn.com" ^[[1~[Thread=MainThread - 2017-07-29 19:34:08,897 - INFO]: Starting base throttler 'base-throttler'... [Thread=MainThread - 2017-07-29 19:34:08,897 - INFO]: Submitting request to base throttler (url: http://cnn.com)... [Thread=Thread-1 - 2017-07-29 19:34:08,897 - INFO]: Starting main loop... [Thread=MainThread - 2017-07-29 19:34:08,900 - INFO]: Submitting request to base throttler (url: http://cnn.com)... [Thread=Thread-1 - 2017-07-29 19:34:08,900 - INFO]: Sending request (url: http://cnn.com/)... [Thread=MainThread - 2017-07-29 19:34:08,900 - INFO]: Submitting request to base throttler (url: http://cnn.com)... [Thread=MainThread - 2017-07-29 19:34:08,901 - INFO]: Submitting request to base throttler (url: http://cnn.com)... [Thread=MainThread - 2017-07-29 19:34:08,901 - INFO]: Submitting request to base throttler (url: http://cnn.com)... ............. [Thread=Thread-1 - 2017-07-29 19:34:09,184 - INFO]: Request sent! (url: http://cnn.com/) [Thread=Thread-1 - 2017-07-29 19:34:09,184 - INFO]: Sending request (url: http://cnn.com/)... <Response [200]> [Thread=Thread-1 - 2017-07-29 19:34:09,343 - INFO]: Request sent! (url: http://www.cnn.com/) <Response [200]> [Thread=Thread-1 - 2017-07-29 19:34:10,401 - INFO]: Sending request (url: http://cnn.com/)... [Thread=Thread-1 - 2017-07-29 19:34:10,545 - INFO]: Request sent! (url: http://www.cnn.com/) <Response [200]> ....... Success: 10, Failures: 0 [Thread=Thread-1 - 2017-07-29 19:34:22,412 - INFO]: Exited from main loop.
The following screenshot shows the output of our recipe:
Throttled Requests to a Web Server
Listing 11.10 shows our recipe, that sends requests to the website provided as the command-line input, in a throttled manner:
#!/usr/bin/env python # Python Network Programming Cookbook, Second Edition
-- Chapter - 11 # This program is optimized for Python 2.7.12. # It may run on any other version with/without
modifications. import argparse import requests from throttler.base_throttler import BaseThrottler def main(address): # Throttle the requests with the BaseThrottler, delaying 1.5s. bt = BaseThrottler(name='base-throttler', delay=1.5) # Visit the address provided by the user. Complete URL only. r = requests.Request(method='GET', url=address) # 10 requests. reqs = [r for i in range(0, 10)] # Submit the requests with the required throttling. with bt: throttled_requests = bt.submit(reqs) # Print the response for each of the requests. for r in throttled_requests: print (r.response) # Final status of the requests. print ("Success: {s}, Failures: {f}".format(s=bt.successes,
f=bt.failures)) if __name__ == '__main__': parser = argparse.ArgumentParser(description=
'Requests Throttling') parser.add_argument('--address', action="store",
dest="address", default='http://www.google.com') given_args = parser.parse_args() address = given_args.address main (address)
This recipe performs the accounting action, by making sure each request are sent only after a certain delay. Here we use BaseThrottler, this ensures that each request is started with a 1.5 second delay in between.