HTTP Engine

From ESM Wiki

Introduction

HTTP engine is a generic engine that encapsulates different methods for making HTTP queries. HTTP engine node is <http>:

<mppd>
    <engines>
        <http id="my_http_engine">
            ...
        </http>
    </engines>
</mppd>

Common options

Common options are specified under engine node and affect all methods of an engine that depend on corresponding options. In the following XML all common options are specified with example values. Their meaning and properties are specified as comments under corresponding nodes. Additional information about options may follow after XML in subsections.

<mppd>
    <engines>
        <http id="my_http_engine">
 
            <!-- Specifies: default URI template. -->
            <!-- Value: template -->
            <!-- Default: n/a -->
            <uri>https://host.name.com:443/path/to/resource/${uri_encode $recipient}-configs.xml</uri>

            <!-- Specifies: default method token that will be used in request lines. -->
            <!-- Value: string -->
            <!-- Default: n/a -->
            <method>GET</method>
 
            <!-- Specifies: default options for timeouts. -->
            <timeout>

                <!-- Specifies: default DNS resolving timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method (usually 60000 (60 sec). -->
                <resolve>60000</resolve>

                <!-- Specifies: default connecting timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <connect>60000</connect>

                <!-- Specifies: default SSL handshaking timeout timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <handshake>60000</handshake>

                <!-- Specifies: default reading timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <read>60000</read>

                <!-- Specifies: default writing timeout. -->
                <!-- Value: unsigned integer (milliseconds)-->
                <!-- Default: depends on method, usually 60000 (60 sec). -->
                <write>60000</write>
 
            </timeout>

            <!-- Specifies: SSL configuration options. Some options must be specified if any engine method is expected to use SSL. -->
            <ssl_context>

                <!-- Specifies: supported SSL method. -->
                <!-- Value: sslv2 | sslv2_client | sslv2_server | sslv3 | sslv3_client | sslv3_server      -->
                <!--        | tlsv1 | tlsv1_client | tlsv1_server | sslv23 | sslv23_client | sslv23_server -->
                <!-- Default: sslv23 -->
                <method>sslv23</method>

                <!-- Specifies: directory containing certificate authority files to be used for performing verification. -->
                <!-- Value: path to a directory. For a required directory structure see 'man SSL_CTX_load_verify_locations'. -->
                <!-- Default: n/a -->
                <add_verify_path>/usr/local/MPP/ssl/ca</add_verify_path>

                <!-- Specifies: certification authority file for performing verification. -->
                <!-- Value: path to a file. For a required file format see 'man SSL_CTX_load_verify_locations'.-->
                <!-- Default: n/a -->
                <load_verify_file>/usr/share/ssl/cert.pem</load_verify_file>

                <!-- Specifies: context options. -->
                <!-- Value: default_workarounds, single_dh_use, no_sslv2, no_sslv3, no_tlsv1 -->
                <!-- Default: default_workarounds -->
                <set_options>default_workarounds</set_options>

                <!-- Specifies: peer verification mode. -->
                <!-- Value: verify_none, verify_peer, verify_fail_if_no_peer_cert, verify_client_once -->
                <!-- Default: verify_peer -->
                <set_verify_mode>verify_peer</set_verify_mode>

                <!-- Specifies: certificate chain from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_certificate_chain_file>/usr/local/MPP/ssl/certificate_chain</use_certificate_chain_file>

                <!-- Specifies: certificate from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_certificate_file>/usr/local/MPP/ssl/certificate.pem</use_certificate_file>

                <!-- Specifies: certificate from a file format -->
                <!-- Value: pem | asn1-->
                <!-- Default: pem -->
                <use_certificate_file_format>pem</use_certificate_file_format>

                <!-- Specifies: private key from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_private_key_file>/usr/local/MPP/ssl/private_key.pem</use_private_key_file>

                <!-- Specifies: private key from a file format. -->
                <!-- Value: pem | asn1 -->
                <!-- Default: pem -->
                <use_private_key_file_format>pem</use_private_key_file_format>

                <!-- Specifies: RSA private key from a file. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_rsa_private_key_file>/usr/local/MPP/ssl/rsa_private_key.pem</use_rsa_private_key_file>

                <!-- Specifies: RSA private key from a file format. -->
                <!-- Value: pem | asn1 -->
                <!-- Default: pem -->
                <use_rsa_private_key_file_format>pem</use_rsa_private_key_file_format>

                <!-- Specifies: file to obtain the temporary Diffie-Hellman parameters. -->
                <!-- Value: path to a file. -->
                <!-- Default: n/a -->
                <use_tmp_dh_file>/usr/local/MPP/ssl/tmp_dh_file</use_tmp_dh_file>

            </ssl_context>
 
        </http>
    </engines>
</mppd>

SSL Context for engine

If engine method is expected to use SSL (for example if URI template of query method can evaluate to https://... URI) the <ssl_context> options must be specified. Minimum requirement is to specifies either <add_verify_path> or <load_verify_file> or both which will add trusted certificate authorities to verify peer certificates. If this is not done then SSL error will occure because peer certificate will not be accepted. One may use verify file that comes with OpenSSL installation and usually located at /usr/share/ssl/cert.pem.

Query method

Query method performs HTTP query to a remote HTTP server. It does the following:

• evaluates URI template and parses obtained URI;
• establishes connection to a remote host specified by URI;
• evaluates entity template;
• evaluates headers template;
• adds Content-Length and Host headers if they are not present yet;
• evaluates request line template;
• evaluates request template;
• sends obtained request to a remote host;
• reads and parses response from a remote host;
• exports parse results as engine results for them to be used in processing pipeline;
• provides embeded mechanisms for extracting data from XML response entities;

Query method is defined with <query> node under <http> engine node:

<mppd>
    <engines>
        <http id="my_http_engine">
            <query id="my_query">
                ...
            </query>
        </http>
    </engines>
</mppd>

Options of query method

In the following XML all options of query method are specified with example values. Their meaning and properties are specified as comments under corresponding nodes. Additional information about options may follow after XML in subsections.

<mppd>
    <engines>
        <http id="my_http_engine">
            <query id="my_query">

                <!-- Specifies: URI template. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Default: inherited from common options or otherwise n/a -->
                <!-- Available action macros: $action.method -->
                <uri>https://host.name.com:443/path/to/resource/${uri_encode $recipient}-configs.xml</uri>

                <!-- Specifies: method token that will be used in request lines. -->
                <!-- Value: string -->
                <!-- Default: inherited from common options or otherwise POST -->
                <method>GET</method>
 
                <!-- Specifies: options for timeouts. -->
                <timeout>

                    <!-- Specifies: default DNS resolving timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <resolve>60000</resolve>

                    <!-- Specifies: default connecting timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: depends on method (usually 60000 (60 sec). -->
                    <connect>60000</connect>

                    <!-- Specifies: default SSL handshaking timeout timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <handshake>60000</handshake>

                    <!-- Specifies: default reading timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <read>60000</read>

                    <!-- Specifies: default writing timeout. -->
                    <!-- Value: unsigned integer (milliseconds)-->
                    <!-- Default: inherited from common options or otherwise 60000 (60 sec). -->
                    <write>60000</write>
 
                </timeout>

                <!-- Specifies: HTTP request entity template. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Default: if absent not entity will be sent. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query                                     -->
                <entity>$data</entity>

                <!-- Specifies: HTTP request headers template. -->
                <!-- Value: template. Before template parsing: option value is split on lines,   -->
                <!--        empty lines and whitespace at the beginnig of lines are removed,     -->
                <!--        line-breaks are replaced with CRLF's. Spaces are literal.            -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity                     -->
                <!-- Default: only default Content-Length and Host headers will be sent. -->
                <headers>
                    X-My-Header: header body
                    X-My-Another-Header: another header body
                </headers>

                <!-- Specifies: HTTP request line template. Default value should suit for most cases. -->
                <!-- Value: template. Before template parsing: CRLF is added to the end. Spaces are literal -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!-- Default: $action.method $action.uri HTTP/1.1 -->
                <request_line>$action.method $action.uri HTTP/1.1</request_line>

                <!-- Specifies: HTTP request template. Default value should suit for most cases. -->
                <!-- Value: template. Spaces are removed. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!--                          $action.request_line                                  -->
                <!-- Default: $action.request_line $action.headers $crlf $action.entity -->
                <request>$action.request_line $action.headers $crlf $action.entity</request>

                <!-- Specifies: maximum size of response. -->
                <!-- Value: template for unsigned integer that specifies number of bytes. Spaces are removed. -->
                <!-- Available action macros: $action.method, $action.uri, $action.uri.scheme,      -->
                <!--                          $action.uri.host, $action.uri.post, $action.uri.path, -->
                <!--                          $action.uri.query, $action.entity, $action.headers    -->
                <!--                          $action.request_line, $action.request                 -->
                <!-- Default: 1048576 -->
                <response_max_size>1048576</response_max_size>

                <!-- Specifies: condition at which response isn't expected to have entity. -->
                <!--            Default value should suit for most cases.                  -->
                <!--            Not applicable for HTTP/1.0 responses.                     -->
                <!-- Value: condition. -->
                <!-- Available action macros:                                                            -->
                <!--     $action.uri.host, $action.uri.post, $action.uri.path,                           -->
                <!--     $action.uri.host, $action.uri.post, $action.uri.path,                           -->
                <!--     $action.uri.query, $action.entity, $action.headers                              -->
                <!--     $action.request_line, $action.request, $action.response_max_size,               -->
                <!--     $action.response.version, $action.response.code, $action.response.code.category -->
                <!--     $action.response.reason, $action.response.headers, ${http_header ...}           -->
                <!-- Default:                                         -->
                <!--         $action.method                  $EQ HEAD -->
                <!--     $OR $action.response.code.category  $EQ 1    -->
                <!--     $OR $action.response.code           $EQ 204  -->
                <!--     $OR $action.response.code           $EQ 304  --> 
                <response_without_entity>
                        $action.method                  $EQ HEAD
                    $OR $action.response.code.category  $EQ 1
                    $OR $action.response.code           $EQ 204
                    $OR $action.response.code           $EQ 304
                </response_without_entity>
  
        </http>
    </engines>
</mppd>

Macros of query method

The following engine specific macros are available for query method:

${xml_xpath_make_list //*[local-name()='Id'] ,$sp ' ' ' \\' \\ \\\\}

'0038000000jGpR0AAK', '0039000000PGKR0AAK', '0048000000gGpR0AOK'

Implicit results of query method

Query method doesn't export any action result implicitly. Required action results should be exported explicitly with <result> options.

See also