HTTP Engine

From ESM Wiki
Jump to: navigation, search

Contents


Introduction

HTTP engine is a generic engine that encapsulates different methods for making HTTP queries. HTTP engine node is <http>:

<mppd>
    <engines>
        <http id="my_http_engine">
            ...
        </http>
    </engines>
</mppd>

Common options

Common options are specified under engine node and affect all methods of an engine that depend on corresponding options. In the following XML all common options are specified with example values. Their meaning and properties are specified as comments under corresponding nodes. Additional information about options may follow after XML in subsections.

<mppd>
    <engines>
        <http id="my_http_engine">
 
            <!-- Specifies: default URI template. -->
            <!-- Value: template -->
            <!-- Default: n/a -->
            <uri>https://host.name.com:443/path/to/resource/${uri_encode $recipient}-configs.xml</uri>

            <!-- Specifies: default method token that will be used in request lines. -->
            <!-- Value: string -->
            <!-- Default: n/a -->
            <method>GET</method>
 
            <!-- Specifies: default options for timeouts. -->
            <timeout>

                <!-- Specifies: default DNS resolving timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method (usually 60000 (60 sec). -->
                <resolve>60000</resolve>

                <!-- Specifies: default connecting timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <connect>60000</connect>

                <!-- Specifies: default SSL handshaking timeout timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <handshake>60000</handshake>

                <!-- Specifies: default reading timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <read>60000</read>

                <!-- Specifies: default writing timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <write>60000</write>
 
            </timeout>

            <!-- Specifies: SSL configuration options. Some options must be specified if any engine method is expected to use SSL. -->
            <ssl_context>

                <!-- Specifies: supported SSL method. -->
                <!-- Value: sslv2 | sslv2_client | sslv2_server | sslv3 | sslv3_client | sslv3_server      -->
                <!--        | tlsv1 | tlsv1_client | tlsv1_server | sslv23 | sslv23_client | sslv23_server -->
                <!-- Default: sslv23 -->
                <method>sslv23</method>

                <!-- Specifies: directory containing certificate authority files to be used for performing verification. -->
                <!-- Value: path to a directory. For a required directory structure see 'man SSL_CTX_load_verify_locations'. -->
                <!-- Default: n/a -->
                <add_verify_path>/usr/local/MPP/ssl/ca</add_verify_path>

                <!-- Specifies: certification authority file for performing verification. -->
                <!-- Value: path to a file. For a required file format see 'man SSL_CTX_load_verify_locations'.-->
                <!-- Default: n/a -->
                <load_verify_file>/usr/share/ssl/cert.pem</load_verify_file>

                <!-- Specifies: context options. -->
                <!-- Value: default_workarounds, single_dh_use, no_sslv2, no_sslv3, no_tlsv1 -->
                <!-- Default: default_workarounds -->
                <set_options>default_workarounds</set_options>

                <!-- Specifies: peer verification mode. -->
                <!-- Value: verify_none, verify_peer, verify_fail_if_no_peer_cert, verify_client_once -->
                <!-- Default: verify_peer -->
                <set_verify_mode>verify_peer</set_verify_mode>

                <!-- Specifies: certificate chain from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_certificate_chain_file>/usr/local/MPP/ssl/certificate_chain</use_certificate_chain_file>

                <!-- Specifies: certificate from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_certificate_file>/usr/local/MPP/ssl/certificate.pem</use_certificate_file>

                <!-- Specifies: certificate from a file format -->
                <!-- Value: pem | asn1-->
                <!-- Default: pem -->
                <use_certificate_file_format>pem</use_certificate_file_format>

                <!-- Specifies: private key from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_private_key_file>/usr/local/MPP/ssl/private_key.pem</use_private_key_file>

                <!-- Specifies: private key from a file format. -->
                <!-- Value: pem | asn1 -->
                <!-- Default: pem -->
                <use_private_key_file_format>pem</use_private_key_file_format>

                <!-- Specifies: RSA private key from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_rsa_private_key_file>/usr/local/MPP/ssl/rsa_private_key.pem</use_rsa_private_key_file>

                <!-- Specifies: RSA private key from a file format. -->
                <!-- Value: pem | asn1 -->
                <!-- Default: pem -->
                <use_rsa_private_key_file_format>pem</use_rsa_private_key_file_format>

                <!-- Specifies: file to obtain the temporary Diffie-Hellman parameters. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_tmp_dh_file>/usr/local/MPP/ssl/tmp_dh_file</use_tmp_dh_file>

            </ssl_context>
 
        </http>
    </engines>
</mppd>

SSL Context for engine

If engine method is expected to use SSL (for example if URI template of query method can evaluate to https://... URI) the <ssl_context> options must be specified. Minimum requirement is to specifies either <add_verify_path> or <load_verify_file> or both which will add trusted certificate authorities to verify peer certificates. If this is not done then SSL error will occure because peer certificate will not be accepted. One may use verify file that comes with OpenSSL installation and usually located at /usr/share/ssl/cert.pem.

Query method

Query method performs HTTP query to a remote HTTP server. It does the following:

  • evaluates URI template and parses obtained URI;
  • establishes connection to a remote host specified by URI;
  • evaluates entity template;
  • evaluates headers template;
  • adds Content-Length and Host headers if they are not present yet;
  • evaluates request line template;
  • evaluates request template;
  • sends obtained request to a remote host;
  • reads and parses response from a remote host;
  • exports parse results as engine results for them to be used in processing pipeline;
  • provides embeded mechanisms for extracting data from XML response entities;

Query method is defined with <query> node under <http> engine node:

<mppd>
    <engines>
        <http id="my_http_engine">
            <query id="my_query">
                ...
            </query>
        </http>
    </engines>
</mppd>

Options of query method

In the following XML all options of query method are specified with example values. Their meaning and properties are specified as comments under corresponding nodes. Additional information about options may follow after XML in subsections.

<mppd>
    <engines>
        <http id="my_http_engine">
            <query id="my_query">

                <!-- Specifies: URI template. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Default: inherited from common options or otherwise n/a -->
                <!-- Available action macros: $action.method -->
                <uri>https://host.name.com:443/path/to/resource/${uri_encode $recipient}-configs.xml</uri>

                <!-- Specifies: method token that will be used in request lines. -->
                <!-- Value: string -->
                <!-- Default: inherited from common options or otherwise POST -->
                <method>GET</method>
 
                <!-- Specifies: options for timeouts. -->
                <timeout>

                    <!-- Specifies: default DNS resolving timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <resolve>60000</resolve>

                    <!-- Specifies: default connecting timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: depends on method (usually 60000 (60 sec). -->
                    <connect>60000</connect>

                    <!-- Specifies: default SSL handshaking timeout timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <handshake>60000</handshake>

                    <!-- Specifies: default reading timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <read>60000</read>

                    <!-- Specifies: default writing timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <write>60000</write>
 
                </timeout>

                <!-- Specifies: HTTP request entity template. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Default: if absent not entity will be sent. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query                                     -->
                <entity>$data</entity>

                <!-- Specifies: HTTP request headers template. -->
                <!-- Value: template. Before template parsing: option value is split on lines,   -->
                <!--        empty lines and whitespace at the beginnig of lines are removed,     -->
                <!--        line-breaks are replaced with CRLF's. Spaces are literal.            -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity                     -->
                <!-- Default: only default Content-Length and Host headers will be sent. -->
                <headers>
                    X-My-Header: header body
                    X-My-Another-Header: another header body
                </headers>

                <!-- Specifies: HTTP request line template. Default value should suit for most cases. -->
                <!-- Value: template. Before template parsing: CRLF is added to the end. Spaces are literal -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!-- Default: $action.method $action.uri HTTP/1.1 -->
                <request_line>$action.method $action.uri HTTP/1.1</request_line>

                <!-- Specifies: HTTP request template. Default value should suit for most cases. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!--                          $action.request_line                                  -->
                <!-- Default: $action.request_line $action.headers $crlf $action.entity -->
                <request>$action.request_line $action.headers $crlf $action.entity</request>

                <!-- Specifies: maximum size of response. -->
                <!-- Value: template for unsigned integer that specifies number of bytes. Spaces are removed. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!--                          $action.request_line, $action.request                 -->
                <!-- Default: 1048576 -->
                <response_max_size>1048576</response_max_size>

                <!-- Specifies: condition at which response isn't expected to have entity. -->
                <!--            Default value should suit for most cases.                  -->
                <!--            Not applicable for HTTP/1.0 responses.                     -->
                <!-- Value: condition. -->
                <!-- Available action macros:                                                            -->
                <!--     $action.uri.host, $action.uri.post, $action.uri.path,                           -->
                <!--     $action.uri.host, $action.uri.post, $action.uri.path,                           -->
                <!--     $action.uri.query, $action.entity, $action.headers                              -->
                <!--     $action.request_line, $action.request, $action.response_max_size,               -->
                <!--     $action.response.version, $action.response.code, $action.response.code.category -->
                <!--     $action.response.reason, $action.response.headers, ${http_header ...}           -->
                <!-- Default:                                         -->
                <!--         $action.method                  $EQ HEAD -->
                <!--     $OR $action.response.code.category  $EQ 1    -->
                <!--     $OR $action.response.code           $EQ 204  -->
                <!--     $OR $action.response.code           $EQ 304  --> 
                <response_without_entity>
                        $action.method                  $EQ HEAD
                    $OR $action.response.code.category  $EQ 1
                    $OR $action.response.code           $EQ 204
                    $OR $action.response.code           $EQ 304
                </response_without_entity>
  
        </http>
    </engines>
</mppd>

Macros of query method

The following engine specific macros are available for query method:

MacroProperties
$action.method Description: HTTP method token. This is simply the value of <method> option.
Scope: <uri>, <entity>, <headers>, <request_line>, <request>, <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.uri
$action.uri.scheme
$action.uri.host
$action.uri.port
$action.uri.path
$action.uri.query
Description: Request URI and its components. Evaluated and parsed <uri> template option.
Scope: <entity>, <headers>, <request_line>, <request>, <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.entity Description: HTTP request entity. Evaluated and parsed <entity> template option. Empty if no entity is specified.
Scope: <headers>, <request_line>, <request>, <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.headers Description: HTTP request headers. Evaluated and parsed <headers> template option possibly prepended with default Content-Length and Host headers.
Scope: <request_line>, <request>, <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.request_line Description: HTTP request-line. Evaluated and parsed <request_line>.
Scope: <request>, <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.request Description: Final HTTP request that is sent to remote host. Evaluated and parsed <request> template option.
Scope: <response_max_size>, <response_without_entity>, <result>
Arguments: none
$action.response_max_size Description: Maximum allowed HTTP response size (including response-line, response headers, and response entity). Evaluated and parsed <response_max_size> template option.
Scope: <result>
Arguments: none
$action.response Description: HTTP response received from a remote host.
Scope: <result>
Arguments: none
$action.response.version Description: HTTP response version string. It includes "HTTP/" previx for example: HTTP/1.1
Scope: <response_without_entity>, <result>
Arguments: none
$action.response.code Description: HTTP response code.
Scope: <response_without_entity>, <result>
Arguments: none
$action.response.code.category Description: HTTP response code category. First character (digit) of response code.
Scope: <response_without_entity>, <result>
Arguments: none
$action.response.reason Description: HTTP response reason.
Scope: <response_without_entity>, <result>
Arguments: none
$action.response.headers Description: HTTP response headers.
Scope: <response_without_entity>, <result>
Arguments: none
$action.response.entity Description: HTTP response entity. Will be empty of response doesn't contain entity.
Scope: <result>
Arguments: none
$http_header Description: Value of a specific HTTP response header. Leading and trailing spaces are removed from a value.
Scope: <response_without_entity>, <result>
Arguments: 1. Header name (case-insensitive).
$xml_xpath Description: Parses response entity as XML file, searches for XML nodes using specified XPath expression and extracts string value (all text and cdata content concatenated in one string) from first found node. If no node is found then macro will be evaluated to empty string. Example of simplest XPath expression is "/mppd/engines/http/method" which fetches the value of option under specified path.
Scope: <result>
Arguments: 1. XPath expression.
$xml_xpath_count
$xml_xpath_count_distinct
Description: Parses response entity as XML file, searches for XML nodes using specified XPath expression and returns count of found nodes. Distinct version of the macro counts only nodes with distinct string values.
Scope: <result>
Arguments: 1. XPath expression.
$xml_xpath_make_list
$xml_xpath_make_list_distinct
Description: Parses response entity as XML file, searches for XML nodes using specified XPath expression and makes textual list from nodes' string values using specified list separator, list entry prefix and suffix and doing replacement of specified sequences in list entry (escaping).
Scope: <result>
Arguments: 1. XPath expression.
2. List separator (optional).
3. List entry prefix (optional).
4. List entry suffix (optional).
*. Starting from 5-th argument begins list (optional) of pairs replacement-replacer like for $text_find_replace_all.

For example the following expression:
${xml_xpath_make_list //*[local-name()='Id'] ,$sp ' ' ' \\' \\ \\\\}

Extracts from XML string values of all nodes with name 'Id' and produces a list like this one:

'0038000000jGpR0AAK', '0039000000PGKR0AAK', '0048000000gGpR0AOK'
$counter8
$counter16
$counter32
$counter64
Description: 8-, 16-, 32- and 64-bit counters respectively. Each counters starts from 1 and increments at each evaluation of a macro. A counter wraps around to zero when it reaches maximum value and then starts incrementing again. A counter is not persistent - it is destroyed when MPP stops. Main purpose of counters is to be used in generation of unique ID's.
Scope: any
Arguments: none

Implicit results of query method

Query method doesn't export any action result implicitly. Required action results should be exported explicitly with <result> options.

See also

Personal tools