urljoin

urljoin(base_url_string,relative_url_string)

Returns a URL string u, obtained by joining relative_url_string, which may be relative, with base_url_string. The joining procedure that urljoin performs to obtain its result u may be summarized as follows:

• When either of the argument strings is empty, u is the other argument.

• When relative_url_string explicitly specifies a scheme that is different from that of base_url_string, u is relative_url_string. Otherwise, u’s scheme is that of base_url_string.

• When the scheme does not allow relative URLs (e.g., mailto) or relative_url_string explicitly specifies a location (even when it is the same as the location of base_url_string), all other components of u are those of relative_url_string. Otherwise, u’s location is that of base_url_string.

• u’s path is obtained by joining the paths of base_url_string and relative_url_string according to standard syntax for absolute and relative URL paths, as per RFC 1808. For example:

from urllib import parse as urlparse  # in v3
# in v2 it would be: import urlparse
urlparse.urljoin(
  'http://somehost.com/some/path/here','../other/path')
# Result is: 'http://somehost.com/some/other/path'

urlsplit

urlsplit(url_string,default_scheme='',allow_fragments=True)

Analyzes url_string and returns a tuple (actually an instance of SplitResult, which you can treat as a tuple or use with named attributes) with five string items: scheme, netloc, path, query, and fragment. default_scheme is the first item when the url_string lacks an explicit scheme. When allow_fragments is False, the tuple’s last item is always '', whether or not url_string has a fragment. Items corresponding to missing parts are ''. For example:

urlparse.urlsplit(
'http://www.python.org:80/faq.cgi?src=fie')
# Result is: 
# ('http','www.python.org:80','/faq.cgi','src=fie','')

urlunsplit

urlunsplit(url_tuple)

url_tuple is any iterable with exactly five items, all strings. Any return value from a urlsplit call is an acceptable argument for urlunsplit. urlunsplit returns a URL string with the given components and the needed separators, but with no redundant separators (e.g., there is no # in the result when the fragment, url_tuple’s last item, is ''). For example:

urlparse.urlunsplit((
'http','www.python.org:80','/faq.cgi','src=fie',''))
# Result is: 'http://www.python.org:80/faq.cgi?src=fie'

urlunsplit(urlsplit(x)) returns a normalized form of URL string x, which is not necessarily equal to x because x need not be normalized. For example:

urlparse.urlunsplit(
urlparse.urlsplit('http://a.com/path/a?'))
# Result is: 'http://a.com/path/a'

In this case, the normalization ensures that redundant separators, such as the trailing ? in the argument to urlsplit, are not present in the result.

quote

quote(str,safe='/')

Returns a copy of str where special characters are changed into Internet-standard quoted form %xx. Does not quote alphanumeric characters, spaces, any of the characters _,.- (underscore, comma, period, hyphen), nor any of the characters in string safe. For example:

print(urllib.quote('zip&zap'))
# prints: zip%26zap

quote_plus

quote_plus(str, safe='/')

Like quote, but also changes spaces into plus signs.

unquote

unquote(str)

Returns a copy of str where each quoted form %xx is changed into the corresponding character. For example:

print(urllib.unquote('zip%26zap'))
# prints: zip&zap

unquote_plus

unquote_plus(str)

Like unquote, but also changes plus signs into spaces.

urlcleanup

urlcleanup()

Clears the cache of function urlretrieve, covered below.

urlencode

urlencode(query,doseq=False)

Returns a string with the URL-encoded form of query. query can be either a sequence of (name, value) pairs, or a mapping, in which case the resulting string encodes the mapping’s (key, value) pairs. For example:

urllib.urlencode([('ans',42),('key','val')])
# result is: 'ans=42&key=val'
urllib.urlencode({'ans':42, 'key':'val'})
# result is: 'key=val&ans=42'

The order of items in a dictionary is arbitrary. Should you need the URL-encoded form to have key/value pairs in a specific order, use a sequence as the query argument, as in the first call in this snippet.

When doseq is true, any value in query that is a sequence and is not a string is encoded as separate parameters, one per item in value. For example:

urllib.urlencode([('K',('x','y','z'))], True)
# result is: 'K=x&K=y&K=z'

When doseq is false (the default), each value is encoded as the quote_plus of its string form given by built-in str, regardless of whether the value is a sequence:

urllib.urlencode([('K',('x','y','z'))], False)
# result is: 'K=%28%27x%27%2C+%27y%27%2C+%27z%27%29'

prompt_user_passwd

f.prompt_user_passwd(host,realm)

Returns a pair (user,password) that urllib is to use in order to authenticate access to host in the security realm. The default implementation in class FancyURLopener prompts the user for this data in interactive text mode. Your subclass can override this method in order to interact with the user via a GUI or to fetch authentication data from persistent storage.

version

f.version

The string that f uses to identify itself to the server—for example, via the User-Agent header in the HTTP protocol. You can override this attribute by subclassing or rebind it directly on an instance of FancyURLopener.

build_opener

build_opener(*handlers)

Creates and returns an instance of the class OpenerDirector (covered in “The OpenerDirector class”) with the given handlers. Each handler can be a subclass of the class BaseHandler, instantiable without arguments, or an instance of such a subclass, however instantiated. build_opener adds instances of various handler classes provided by the module urllib2 in front of the handlers you specify, in order to handle proxies; unknown schemes; the http, file, and https schemes; HTTP errors; and HTTP redirects. However, if you have instances or subclasses of said classes in handlers, this indicates that you want to override these defaults.

install_opener

install_opener(opener)

Installs opener as the opener for further calls to urlopen. opener can be an instance of the class OpenerDirector, such as the result of a call to the function build_opener, or any signature-compatible object.

urlopen

urlopen(url,data=None)

Almost identical to the urlopen function in the module urllib. However, you customize behavior via the opener and handler classes of urllib2 (covered in “The OpenerDirector class” and “Handler classes”) rather than via the class FancyURLopener as in the module urllib. The argument url can be a URL string, like for the urlopen function in the module urllib. Alternatively, url can be an instance of the class Request, covered in the next section.

Request

class Request(urlstring,data=None,headers={})

urlstring is the URL that this instance of the Request class embodies. For example, when there are no data and headers, calling:

urllib2.urlopen(urllib2.Request(urlstring))

is just like calling:

urllib2.urlopen(urlstring)

When data is not None, the Request constructor implicitly calls on the new instance r its method r.add_data(data). headers must be a mapping of header names to header values. The Request constructor executes the equivalent of the loop:

for k,v in headers.items(): r.add_header(k,v)

The Request constructor also accepts optional parameters allowing fine-grained control of HTTP cookie behavior, but such advanced functionality is rarely necessary—the class’s default handling of cookies is generally sufficient. For fine-grained, client-side control of cookies, see the online docs; we do not cover the cookielib module of the standard library in this book.

An instance r of the class Request supplies the following methods:

add_data

r.add_data(data)

Sets data as r’s data. Calling urlopen(r) then becomes like calling urlopen(r,data)—that is, it requires r’s scheme to be http and uses a POST request with a body of data, which must be a URL-encoded string.

Despite its name, the method add_data does not necessarily add the data. If r already had data, set in r’s constructor or by previous calls to r.add_data, then the latest call to r.add_data replaces the previous value of r’s data with the new given one. In particular, r.add_data(None) removes r’s previous data, if any.

add_header

r.add_header(key,value)

Adds a header with the given key and value to r’s headers. If r’s scheme is http, r’s headers are sent as part of the request. When you add more than one header with the same key, later additions overwrite previous ones, so out of all headers with one given key, only the one given last matters.

add_unredirec-ted_header

r.add_unredirected_header(key,value)

Like add_header, except that the header is added only for the first request, and is not used if the requesting procedure meets and follows any further HTTP redirection.

get_data

r.get_data()

Returns the data of r, either None or a URL-encoded string.

get_full_url

r.get_full_url()

Returns the URL of r, as given in the constructor for r.

get_host

r.get_host()

Returns the host component of r’s URL.

get_method

r.get_method()

Returns the HTTP method of r, either of the strings 'GET' or 'POST'.

get_selector

r.get_selector()

Returns the selector components of r’s URL (path and all following components).

get_type

r.get_type()

Returns the scheme component of r’s URL (i.e., the protocol).

has_data

r.has_data()

Like r.get_data() is not None.

has_header

r.has_header(key)

Returns True if r has a header with the given key; otherwise, returns False.

set_proxy

r.set_proxy(host,scheme)

Sets r to use a proxy at the given host and scheme for accessing r’s URL.

add_password

x.add_password(realm,URLs,user,password)

Records in x the pair (user,password) as the credentials in the given realm for URLs given by URLs. realm is a string that names an authentication realm; when realm is None, it matches any realm not specifically recorded. URLs is a URL string or a sequence of URL strings. A URL u is deemed applicable for these credentials if there is an item u1 of URLs such that the location components of u and u1 are equal, and the path component of u1 is a prefix of that of u. Other components (scheme, query, fragment) don’t affect applicability for authentication purposes.

The following example shows how to use urllib2 with basic HTTP authentication:

import urllib2

x = urllib2.HTTPPasswordMgrWithDefaultRealm( )
x.add_password(None, 'http://myhost.com/', 
'auser', 'apassword')
auth = urlib2.HTTPBasicAuthHandler(x)
opener = urllib2.build_opener(auth)
urllib2.install_opener(opener)

flob = urllib2.urlopen('http://myhost.com/index.html')
for line in flob.readlines(): print line,