5.3. HTTP Messages

This modules defines objects that represent the values of HTTP messages and message headers and a special-purpose parser for parsing them from strings of octets.

5.3.1. Messages

class pyslet.http.messages.Request(**kwargs)

Bases: pyslet.http.messages.Message

method = None

the http method, always upper case, e.g., ‘POST’

request_uri = None

the request uri as it appears in the start line

response = None

the associated response

send_start()

Returns the start-line for this message

send_transferlength()

Adds request-specific processing for transfer-length

Request messages that must not have a message body are automatically detected and will raise an exception if they have a non-None body.

Request messages that may have a message body but have a transfer-length of 0 bytes will have a Content-Length header of 0 added if necessary

get_start()

Returns the start line

is_idempotent()

Returns True if this is an idempotent request

extract_authority()

Extracts the authority from the request

If the request_uri is an absolute URL then it is updated to contain the absolute path only and the Host header is updated with the authority information (host[:port]) extracted from it, otherwise the Host header is read for the authority information. If there is no authority information in the request None is returned.

If the url contains user information it raises NotImplementedError

get_accept()

Returns an AcceptList instance or None if no “Accept” header is present.

set_accept(accept_value)

Sets the “Accept” header, replacing any existing value.

accept_value

A AcceptList instance or a string that one can be parsed from.

get_accept_charset()

Returns an AcceptCharsetList instance or None if no “Accept-Charset” header is present.

set_accept_charset(accept_value)

Sets the “Accept-Charset” header, replacing any existing value.

accept_value

A AcceptCharsetList instance or a string that one can be parsed from.

get_accept_encoding()

Returns an AcceptEncodingList instance or None if no “Accept-Encoding” header is present.

set_accept_encoding(accept_value)

Sets the “Accept-Encoding” header, replacing any existing value.

accept_value

A AcceptEncodingList instance or a string that one can be parsed from.

get_cookie()

Reads the ‘Cookie’ header(s)

Returns a dictionary of cookies. If there are multiple values for a cookie the dictionary value is a set, otherwise it is a string.

set_cookie(cookie_list)

Set a “Set-Cookie” header

cookie_list

a list of cookies such as would be returned by pyslet.http.cookie.CookieStore.search().

If cookie list is None the Cookie header is removed.

class pyslet.http.messages.Response(request=None, **kwargs)

Bases: pyslet.http.messages.Message

REASON = {200: 'OK', 201: 'Created', 202: 'Accepted', 203: 'Non-Authoritative Information', 204: 'No Content', 205: 'Reset Content', 206: 'Partial Content', 400: 'Bad Request', 401: 'Unauthorized', 402: 'Payment Required', 403: 'Forbidden', 404: 'Not Found', 405: 'Method Not Allowed', 406: 'Not Acceptable', 407: 'Proxy Authentication Required', 408: 'Request Time-out', 409: 'Conflict', 410: 'Gone', 411: 'Length Required', 412: 'Precondition Failed', 413: 'Request Entity Too Large', 414: 'Request-URI Too Large', 415: 'Unsupported Media Type', 416: 'Requested range not satisfiable', 417: 'Expectation Failed', 100: 'Continue', 101: 'Switching Protocols', 300: 'Multiple Choices', 301: 'Moved Permanently', 302: 'Found', 303: 'See Other', 304: 'Not Modified', 305: 'Use Proxy', 307: 'Temporary Redirect', 500: 'Internal Server Error', 501: 'Not Implemented', 502: 'Bad Gateway', 503: 'Service Unavailable', 504: 'Gateway Time-out', 505: 'HTTP Version not supported'}

A dictionary mapping status code integers to their default message defined by RFC2616

send_start()

Returns the start-line for this message

get_accept_ranges()

Returns an AcceptRanges instance or None if no “Accept-Ranges” header is present.

set_accept_ranges(accept_value)

Sets the “Accept-Ranges” header, replacing any existing value.

accept_value

A AcceptRanges instance or a string that one can be parsed from.

get_age()

Returns an integer or None if no “Age” header is present.

set_age(age)

Sets the “Age” header, replacing any existing value.

age

an integer or long value or None to remove the header

get_etag()

Returns a EntityTag instance parsed from the ETag header or None if no “ETag” header is present.

set_etag(etag)

Sets the “ETag” header, replacing any existing value.

etag

a EntityTag instance or None to remove any ETag header.

get_location()

Returns a pyslet.rfc2396.URI instance created from the Location header.

If no Location header was present None is returned.

set_location(location)

Sets the Location header

location:

a pyslet.rfc2396.URI instance or a string from which one can be parsed. If None, the Location header is removed.

get_www_authenticate()

Returns a list of Challenge instances.

If there are no challenges an empty list is returned.

set_www_authenticate(challenges)

Sets the “WWW-Authenticate” header, replacing any exsiting value.

challenges

a list of Challenge instances

get_set_cookie()

Reads all ‘Set-Cookie’ headers

Returns a list of Cookie instances

set_set_cookie(cookie, replace=False)

Set a “Set-Cookie” header

cookie

a Cookie instance

replace=True

Remove all existing cookies from the response

replace=False

Add this cookie to the existing cookies in the response (default value)

If called multiple times the header value will become a list of cookie values. No folding together is performed.

If cookie is None all Set-Cookie headers are removed, implying replace mode.

class pyslet.http.messages.Message(entity_body=None, protocol=<pyslet.http.params.HTTPVersion object>, send_stream=None, recv_stream=None)

Bases: pyslet.pep8.PEP8Compatibility, object

An abstract class to represent an HTTP message.

The methods of this class are thread safe, using a lock to protect all access to internal structures.

The generic syntax of a message involves a start line, followed by a number of message headers and an optional message body.

entity_body

The optional entity_body parameter is a byte string containing the entity body, a file like object or object derived from io.RawIOBase. There are restrictions on the use of non-seekable streams, in particular the absence of a working seek may affect redirects and retries.

There is a subtle difference between passing None, meaning no entity body and an empty string ‘’. The difference is that an empty string will generate a Content-Length header indicating a zero length message body when the message is sent, whereas None will not. Some message types are not allowed to have an entity body (e.g., a GET request) and these messages must not have a message body (even a zero length one) or an error will be raised.

File-like objects do not generate a Content-Length header automatically as there is no way to determine their size when sending, however, if a Content-Length header is set explicitly then it will be used to constrain the amount of data read from the entity_body.

GENERAL_HEADERS = {'transfer-encoding': 'Transfer-Encoding', 'connection': 'Connection', 'upgrade': 'Upgrade', 'pragma': 'Pragma', 'cache-control': 'Cache-Control', 'date': 'Date', 'warning': 'Warning', 'via': 'Via', 'trailer': 'Trailer'}

a mapping from lower case header name to preferred case name

MAX_READAHEAD = 131072

A constant used to control the maximum read-ahead on an entity body’s stream. Entity bodies of undetermined length that exceed this size cannot be sent in requests to HTTP/1.0 server.

lock = None

the lock used to protect multi-threaded access

got_headers = None

boolean indicating that all headers have been received

keep_alive = None

by default we’ll keep the connection alive

set_protocol(version)

Sets the protocol

version

An params.HTTPVersion instance or a string that can be parsed for one.

clear_keep_alive()

Clears the keep_alive flag on this message

The flag always starts set to True and cannot be set once cleared.

start_sending(protocol=<pyslet.http.params.HTTPVersion object>)

Starts sending this message

protocol

The protocol supported by the target of the message, defaults to HTTP/1.1 but can be overridden when the recipient only supports HTTP/1.0. This has the effect of suppressing some features.

The message is sent using the send_ family of methods.

send_start()

Returns the start-line for this message

send_header()

Returns a data string ready to send to the server

send_transferlength()

Calculates the transfer length of the message

It will read the Transfer-Encoding or Content-Length headers to determine the length.

If the length of the entity body is known, this method will verify that it matches the Content-Length or set that header’s value accordingly.

If the length of the entity body is not known, this method will set a Transfer-Encoding header.

abort_sending()

Aborts sending the message body

Called after start_sending, this method attempts to abort the sending the message body and returns the (approximate) number of bytes that will be returned by future calls to send_body. (Ignoring chunk boundaries.)

Messages that are already complete will return 0.

Messages that are using chunked transfer encoding can be aborted and will return 0 indicating that the next chunk returned by send_body() will be the trailing chunk.

Messages that are not using chunked transfer encoding cannot be aborted and will return the number of bytes remaining or -1 if this cannot be determined (the latter case is only possible when the message body will be terminated by a connection close and so only applies to responses).

This method has a very special use case. In cases where a server rejects a request before reading the entire message body the client may attempt to abort the sending of the body without closing the connection. The only way to do this is to truncate a body being sent with chunked encoding. You might wonder why a client would go to such lengths to keep the connection open. The answer is NTLM which authenticates a connection, so a large POST that gets an early 401 response must be retried on the same connection. This can only be done if the message boundaries are well defined. There’s a good discussion of the issue at https://curl.haxx.se/mail/lib-2004-08/0002.html

send_body()

Returns (part of) the message body

Returns an empty string when there is no more data to send.

Returns None if the message is read blocked.

start_receiving()

Starts receiving this message

The message is received using the recv_mode() and recv() methods.

RECV_HEADERS = -3

recv_mode constant for a set of header lines terminated by CRLF, followed by a blank line.

RECV_LINE = -2

recv_mode constant for a single CRLF terminated line

RECV_ALL = -1

recv_mode constant for unlimited data read

recv_mode()

Indicates the type of data expected during recv

The result is interpreted as follows, using the recv_mode constants defined above:

RECV_HEADERS

this message is expecting a set of headers, terminated by a blank line. The next call to recv must be with a list of binary CRLF terminated strings the last of which must the string CRLF only.

RECV_LINE

this message is expecting a single terminated line. The next call to recv must be with a binary string representing a single terminated line.

integer > 0

the minimum number of bytes we are waiting for when data is expected. The next call to recv must be with a binary string of up to but not exceeding integer number of bytes

0

we are currently write-blocked but may need more data, the next call to recv must pass None to give the message time to write out existing buffered data.

RECV_ALL

we want to read until the connection closes, the next call to recv must be with a binary string. The string can be of any length but an empty string signals the end of the data.

None

the message is not currently in receiving mode, calling recv will raise an error.

recv_start(start_line)

Receives the start-line

Implemented differently for requests and responses.

handle_headers()

Hook for processing the message headers

This method is called after all headers have been received but before the message body (if any) is received. Derived classes should always call this implementation first (using super) to ensure basic validation is performed on the message before the body is received.

The default implementation sets got_headers to True.

recv_transferlength()

Called to calculate the transfer length when receiving

The values of transferlength and transferchunked are set by this method. The default implementation checks for a Transfer-Encoding header and then a Content-Length header in that order.

If it finds neither then behaviour is determined by the derived classes Request and Response which wrap this implementation.

RFC2616:

If a Transfer-Encoding header field is present and has any value other than “identity”, then the transfer-length is defined by use of the “chunked” transfer-coding, unless the message is terminated by closing the connection

This is a bit weird, if I have a non-identity value which fails to mention ‘chunked’ then it seems like I can’t imply chunked encoding until the connection closes. In practice, when we handle this case we assume chunked is not being used and read until connection close.

handle_message()

Hook for processing the message

This method is called after the entire message has been received, including any chunk trailer.

get_headerlist()

Returns all header names

The list is alphabetically sorted and lower-cased.

has_header(field_name)

True if this message has a header with field_name

get_header(field_name, list_mode=False)

Returns the header with field_name as a string.

list_mode=False

In this mode, get_header always returns a single binary string, this isn’t always what you want as it automatically ‘folds’ multiple headers with the same name into a string using “, ” as a separator.

list_mode=True

In this mode, get_header always returns a list of binary strings.

If there is no header with field_name then None is returned in both modes.

set_header(field_name, field_value, append_mode=False)

Sets the header with field_name to the string field_value.

If field_value is None then the header is removed (if present).

If a header already exists with field_name then the behaviour is determined by append_mode:

append_mode==True

field_value is joined to the existing value using “, ” as a separator.

append_mode==False (Default)

field_value replaces the existing value.

get_allow()

Returns an Allow instance or None if no “Allow” header is present.

set_allow(allowed)

Sets the “Allow” header, replacing any existing value.

allowed

A Allow instance or a string that one can be parsed from.

If allowed is None any existing Allow header is removed.

get_authorization()

Returns a Credentials instance.

If there are no credentials None returned.

set_authorization(credentials)

Sets the “Authorization” header

credentials

a Credentials instance

get_cache_control()

Returns an CacheControl instance or None if no “Cache-Control” header is present.

set_cache_control(cc)

Sets the “Cache-Control” header, replacing any existing value.

cc

A CacheControl instance or a string that one can be parsed from.

If cc is None any existing Cache-Control header is removed.

get_connection()

Returns a set of connection tokens from the Connection header

If no Connection header was present an empty set is returned. All tokens are returned as lower case.

set_connection(connection_tokens)

Set the Connection tokens from an iterable set of connection_tokens

If the list is empty any existing header is removed.

get_content_encoding()

Returns a list of lower-cased content-coding tokens from the Content-Encoding header

If no Content-Encoding header was present an empty list is returned.

Content-codings are always listed in the order they have been applied.

set_content_encoding(content_codings)

Sets the Content-Encoding header from a an iterable list of content-coding tokens. If the list is empty any existing header is removed.

get_content_language()

Returns a list of LanguageTag instances from the Content-Language header

If no Content-Language header was present an empty list is returned.

set_content_language(lang_list)

Sets the Content-Language header from a an iterable list of LanguageTag instances.

get_content_length()

Returns the integer size of the entity from the Content-Length header

If no Content-Length header was present None is returned.

set_content_length(length)

Sets the Content-Length header from an integer or removes it if length is None.

get_content_location()

Returns a pyslet.rfc2396.URI instance created from the Content-Location header.

If no Content-Location header was present None is returned.

set_content_location(location)

Sets the Content-Location header from location, a pyslet.rfc2396.URI instance or removes it if location is None.

get_content_md5()

Returns a 16-byte binary string read from the Content-MD5 header or None if no Content-MD5 header was present.

The result is suitable for comparing directly with the output of the Python’s MD5 digest method.

set_content_md5(digest)

Sets the Content-MD5 header from a 16-byte binary string returned by Python’s MD5 digest method or similar. If digest is None any existing Content-MD5 header is removed.

get_content_range()

Returns a ContentRange instance parsed from the Content-Range header.

If no Content-Range header was present None is returned.

set_content_range(range)

Sets the Content-Range header from range, a ContentRange instance or removes it if range is None.

get_content_type()

Returns a MediaType instance parsed from the Content-Type header.

If no Content-Type header was present None is returned.

set_content_type(mtype=None)

Sets the Content-Type header from mtype, a MediaType instance, or removes it if mtype is None.

get_date()

Returns the value of the Date header.

The return value is a params.FullDate instance. If no Date header was present None is returned.

set_date(date=None)

Sets the value of the Date header

date

a params.FullDate instance or None to remove the Date header.

To set the date header to the current date use:

set_date(params.FullDate.from_now_utc())

get_last_modified()

Returns the value of the Last-Modified header

The result is a params.FullDate instance. If no Last-Modified header was present None is returned.

set_last_modified(date=None)

Sets the value of the Last-Modified header field

date

a FullDate instance or None to remove the header

To set the Last-Modified header to the current date use:

set_last_modified(params.FullDate.from_now_utc())

get_transfer_encoding()

Returns a list of params.TransferEncoding

If no TransferEncoding header is present None is returned.

set_transfer_encoding(field_value)

Set the Transfer-Encoding header

field_value

A list of params.TransferEncoding instances or a string from which one can be parsed. If None then the header is removed.

set_upgrade(protocols)

Sets the “Upgrade” header, replacing any existing value.

protocols

An iterable list of params.ProductToken instances.

In addition to setting the upgrade header this method ensures that “upgrade” is present in the Connection header.

5.3.2. General Header Types

class pyslet.http.messages.CacheControl(*args)

Bases: pyslet.http.params.Parameter

Represents the value of a Cache-Control general header.

Instances are immutable, they are constructed from a list of arguments which must not be empty. Arguments are treated as follows:

string

a simple directive with no parmeter

2-tuple of string and non-tuple

a directive with a simple parameter

2-tuple of string and tuple

a directive with a quoted list-style parameter

Instances behave like read-only lists implementing len, indexing and iteration in the usual way. Instances also support basic key lookup of directive names by implementing __contains__ and __getitem__ (which returns None for defined directives with no parameter and raises KeyError for undefined directives). Instances are not truly dictionary like.

classmethod from_str(source)

Create a Cache-Control value from a source string.

5.3.3. Request Header Types

class pyslet.http.messages.AcceptList(*args)

Bases: object

Represents the value of an Accept header

The built-in str function can be used to format instances according to the grammar defined in the specification.

Instances are immutable, they are constructed from one or more AcceptItem instances. There are no comparison methods.

Instances behave like read-only lists implementing len, indexing and iteration in the usual way.

select_type(mtype_list)

Returns the best match from mtype_list, a list of media-types

In the event of a tie, the first item in mtype_list is returned.

classmethod from_str(source)

Create an AcceptList from a source string.

class pyslet.http.messages.MediaRange(type='*', subtype='*', parameters={})

Bases: pyslet.http.params.MediaType

Represents an HTTP media-range.

Quoting from the specification:

“Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence.”

We override the base class ordering so that MediaRange instances sort according to these rules. The following media ranges would be sorted in the order shown:

1. image/png
2. image/*
3. text/plain;charset=utf-8
4. text/plain
5. text/*
6. */*

If we have two rules with identical precedence then we sort them alphabetically by type; sub-type and ultimately alphabetically by parameters

classmethod from_str(source)

Creates a media-rannge from a source string.

Unlike the parent media-type we ignore all spaces.

match_media_type(mtype)

Tests whether a media-type matches this range.

mtype

A MediaType instance to be compared to this range.

The matching algorithm takes in to consideration wild-cards so that */* matches all types, image/* matches any image type and so on.

If a media-range contains parameters then each of these must be matched exactly in the media-type being tested. Parameter names are treated case-insensitively and any additional parameters in the media type are ignored. As a result:

• text/plain does not match the range text/plain;charset=utf-8
• application/myapp;charset=utf-8;option=on does match the range application/myapp;option=on

class pyslet.http.messages.AcceptItem(range=MediaType('*', '*', {}), qvalue=1.0, extensions={})

Bases: pyslet.http.params.SortableParameter

Represents a single item in an Accept header

Accept items are sorted by their media ranges. Equal media ranges sort by descending qvalue, for example:

text/plain;q=0.75 < text/plain;q=0.5

Extension parameters are ignored in all comparisons.

range = None

the MediaRange instance that is acceptable

q = None

the q-value (defaults to 1.0)

classmethod from_str(source)

Creates a single AcceptItem instance from a source string.

class pyslet.http.messages.AcceptCharsetItem(token='*', qvalue=1.0)

Bases: pyslet.http.messages.AcceptToken

Represents a single item in an Accept-Charset header

class pyslet.http.messages.AcceptCharsetList(*args)

Bases: pyslet.http.messages.AcceptTokenList

Represents an Accept-Charset header

ItemClass

alias of AcceptCharsetItem

select_token(token_list)

Overridden to provide default handling of iso-8859-1

class pyslet.http.messages.AcceptEncodingItem(token='*', qvalue=1.0)

Bases: pyslet.http.messages.AcceptToken

Represents a single item in an Accept-Encoding header

class pyslet.http.messages.AcceptEncodingList(*args)

Bases: pyslet.http.messages.AcceptTokenList

Represents an Accept-Encoding header

ItemClass

alias of AcceptEncodingItem

select_token(token_list)

Overridden to provide default handling of identity

class pyslet.http.messages.AcceptLanguageItem(token='*', qvalue=1.0)

Bases: pyslet.http.messages.AcceptToken

Represents a single item in an Accept-Language header.

class pyslet.http.messages.AcceptLanguageList(*args)

Bases: pyslet.http.messages.AcceptTokenList

Represents an Accept-Language header

ItemClass

the class used to create items in this token list

alias of AcceptLanguageItem

select_token(token_list)

Remapped to select_language()

class pyslet.http.messages.AcceptToken(token='*', qvalue=1.0)

Bases: pyslet.http.params.SortableParameter

Represents a single item in a token-based Accept-* header

AcceptToken items are sorted by their token, with wild cards sorting behind specified tokens. Equal values sort by descending qvalue, for example:

iso-8859-2;q=0.75 < iso-8859-2;q=0.5

token = None

the token that is acceptable or “*” for any token

q = None

the q-value (defaults to 1.0)

classmethod from_str(source)

Creates a single AcceptToken instance from a source string.

class pyslet.http.messages.AcceptTokenList(*args)

Bases: pyslet.http.params.Parameter

Represents the value of a token-based Accept-* header

Instances are immutable, they are constructed from one or more AcceptToken instances. There are no comparison methods.

Instances behave like read-only lists implementing len, indexing and iteration in the usual way.

ItemClass

the class used to create new items in this list

alias of AcceptToken

select_token(token_list)

Returns the best match from token_list, a list of tokens.

In the event of a tie, the first item in token_list is returned.

classmethod from_str(source)

Create an AcceptTokenList from a source string.

5.3.4. Response Header Types

class pyslet.http.messages.AcceptRanges(*args)

Bases: pyslet.http.params.SortableParameter

Represents the value of an Accept-Ranges response header.

Instances are immutable, they are constructed from a list of string arguments. If the argument list is empty then a value of “none” is assumed.

Instances behave like read-only lists implementing len, indexing and iteration in the usual way. Comparison methods are provided.

classmethod from_str(source)

Create an AcceptRanges value from a source string.

5.3.5. Entity Header Types

class pyslet.http.messages.Allow(*args)

Bases: pyslet.http.params.SortableParameter

Represents the value of an Allow entity header.

Instances are immutable, they are constructed from a list of string arguments which may be empty.

Instances behave like read-only lists implementing len, indexing and iteration in the usual way. Comparison methods are provided.

classmethod from_str(source)

Create an Allow value from a source string.

is_allowed(method)

Tests if method is allowed by this value.

class pyslet.http.messages.ContentRange(first_byte=None, last_byte=None, total_len=None)

Bases: object

Represents a single content range

first_byte

Specifies the first byte of the range

last_byte

Specifies the last byte of the range

total_len

Specifies the total length of the entity

With no arguments an invalid range representing an unsatisfied range request from an entity of unknown length is created.

If first_byte is specified on construction last_byte must also be specified or TypeError is raised.

The built-in str function can be used to format instances according to the grammar defined in the specification.

Instances are immutable.

first_byte = None

first byte in the range

last_byte = None

last byte in the range

classmethod from_str(source)

Creates a single ContentRange instance from a source string.

is_valid()

Returns True if this range is valid, False otherwise.

A valid range is any non-empty byte range wholly within the entity described by the total length. Unsatisfied content ranges are treated as invalid.

5.3.6. Parsing Header Values

In most cases header values will be parsed automatically when reading them from messages. For completeness a header parser is exposed to enable you to parse these values from more complex strings.

class pyslet.http.messages.HeaderParser(source, ignore_sp=True)

Bases: pyslet.http.params.ParameterParser

A special parser for parsing HTTP headers from TEXT

In keeping with RFC2616 all parsing is done on binary strings. See base class for more information.

require_media_range()

Parses a MediaRange instance.

Raises BadSyntax if no media-type was found.

require_accept_item()

Parses a AcceptItem instance

Raises BadSyntax if no item was found.

require_accept_list()

Parses a AcceptList instance

Raises BadSyntax if no valid items were found.

require_accept_token(cls=<class 'pyslet.http.messages.AcceptToken'>)

Parses a single AcceptToken instance

Raises BadSyntax if no item was found.

cls

An optional sub-class of AcceptToken to create instead.

require_accept_token_list(cls=<class 'pyslet.http.messages.AcceptTokenList'>)

Parses a list of token-based accept items

Returns a AcceptTokenList instance. If no tokens were found then an empty list is returned.

cls

An optional sub-class of AcceptTokenList to create instead.

require_contentrange()

Parses a ContentRange instance.

require_product_token_list()

Parses a list of product tokens

Returns a list of params.ProductToken instances. If no tokens were found then an empty list is returned.

5.3.7. Exceptions

class pyslet.http.messages.HTTPException

Bases: exceptions.Exception

Class for all HTTP message-related errors.