File Engine

From ESM Wiki
Jump to: navigation, search

Contents

Introduction

File Engine is a Generic Engine that encapsulates different methods for working with files: saving, comparing, etc. File Engine node is <file>:

<mppd>
    <engines>
        <file id=”data_store”>
            ...
        </file>
    </engines>
</mppd>

Primitives

Primitives are more-or-less complex data structures that are used through out this document. In this chapter primitives are described in details so that when they are used they are just referenced – no need to explain its structure again and again.

Path

Path is a primitive that defines most common properties of a file system object: path, user, group and mode. In XML it is defined as a node with four children with corresponding names. Node name depends on context for which path primitive is used. If a node is omitted then default value is provided depending on a context. Example of a path primitive:

<relative>
    <path>/$recipient/$year-$month-$day/$time6$counter16</path>
    <user>$recipient.local</user>
    <group>$group</group>
    <mode>0620</mode>
</relative>

In this XML <relative> is a path primitive with all options specified.

Options and their properties are described in the following table:

OptionProperties
<path> Description: Path to an object. Option is a template that evaluates to a string. There are two regimes of how the option is read and processed. The regime is defined with read attribute for the option. The following regimes are supported:
  • by_components -- first of all components are extracted that are strings between '/' characters; components are treated as templates; at runtime components are evaluated and unsafe characters and character sequences are replaced with 'x' characters; then components are combined back into path; this is a default behavior that prevents generation of unwanted locations due to “special” values of macros.
  • entire -- option is not split on components but is treated whole as a template. This should be used only if macros involved are known to have "safe" values.

Example:

<path read=”entire”>$engines.db.search.path</path>
Type: Template (with additional semantics). Spaces are removed. Evaluates to a string.
Default: Depends on a context.
<user> Description: Unix user name. Must evaluates to an existing Unix user name. Otherwise runtime error occurs.
Type: Template. Spaces are removed. Evaluates to a string.
Default: Depends on a context. If not specified then it is effective user of current process.
<group> Description: Unix group name. Must evaluates to an existing Unix group name. Otherwise runtime error occurs.
Type: Template. Spaces are removed. Evaluates to a string.
Default: Depends on a context. If not specified then it is effective group of current process.
<mode> Description: Access mode for an object. Must evaluates to an octal number with usual meaning for Unix access modes. Otherwise runtime error occurs.
Type: Template. Spaces are removed. Evaluates to an octal number.
Default: Depends on a context. If not specified it is 0620 for files and 0730 for directories.

Common options

Common options are specified under engine node and affect all methods of an engine that depend on corresponding options. The following common options are available:

OptionProperties
<base> Description: Base path for all relative paths. Paths defined in methods are often relative to base path. This allow for more flexibility in path manipulations. Base path should point on a directory. If directory doesn't exist it is created with all intermediate directories using specified path primitive options. Example:
<file>
    <base>
        <path>/usr/local/MPP/$group</path>
        <user>mpp</user>
        <group>$group</group>
        <mode>0730</mode>
    </base>
    ...
</file>
Type: Path primitive.
Default: path = /usr/local/MPP/working/file, other are same as in primitive.
<tmp> Description: Temporary file path template. Provides template from which temporary file paths are generated. Temporary files may be use by methods for different purposes. A method will use this template if it doesn't define own. File name component of a path should evaluate to unique file name. For this purpose the following macros and their combinations may be used: $gethostname, $pid, $thread, $time6, $counter8, $counter16, $counter32, $counter64. More on temporary files can be found in method descriptions. Example:
<file>
    <tmp>
        <path>/var/tmp/$pid$time6$counter16</path>
        <mode>0600</mode>
    </tmp>
    ...
</file>
Type: Path primitive.
Default: None.

Save method

Save method performs saving of data generated from a template to a file with specified path. It does the following:

  • calculates data to be saved from provided data template;
  • calculates base and relative paths;
  • calculates full path by concatenating base and relative paths;
  • if temporary file path template is defined it calculates the path and writes data to a temporary file;
  • then it moves a file from temporary location to full path location through hard linking;
  • if temporary file path template is NOT define it writes data directly to a file at full path;
  • all intermediate directories are created if they doesn't exist;
  • exports calculated base, relative and full paths as engine results;

Save method is defined with <save> node under <file> engine node. For example:

<mppd>
    <engines>
        <file id="data_store">
            <save id="save_message">
                <base>
                    <path>/usr/local/MPP/mail</path>
                    <user>mail</user>
                    <group>mail</group>
                    <mode>0770</mode>
                </base>
                <tmp>
                    <path>/usr/local/MPP/working/${time6}${counter16}</path>
                    <mode>0600</mode>
                </tmp>
                <relative>
                    <path>$recipient/${time6}${counter16}.eml</path>
                    <user>$recipient.local</user>
                    <group>mail</group>
                    <mode>0660</mode>
                </relative>
                <data>$data</data>
            </save>
        </file>
    </engines>
</mppd>

Options of save method

The following options are available for save method:

OptionProperties
<base> Description: Base path for a relative one.
Type: Path primitive.
Default: Inherited from engine's common option.
<tmp> Description: Temporary file path template. If this option is defined then data will first be written to a temporary file, then this file will be relinked (using hard file system links) to final full path. If this option is not defined (whether in method or in engine) then data will be directly written to final full path. Use temporary files to avoid reading of partially written files. Use temporary files on NFS to avoid data corruption due to file name collisions.
Type: Path primitive.
Default: Inherited from engine's common option.
<relative> Description: Relative file path template to save data to. Final full path for file data is combined from <base> and <relative> paths.
Type: Path primitive.
Default: No default – this is mandatory option.
<data> Description: Template for file data. Data template is evaluated and obtained string (possibly binary string) is written to a file.
Type: Template. Evaluates to a string (binary). Spaces are removed.
Default: No default – this is mandatory option.
<acquire_duplicates> Description: Specifies whether duplicates must be acquired or not. Normally if there is a file already exists at final full path then file name is altered until unique final path is found. This behavior for “no” value of the option. But in some circumstances it might be desired to check if existing file data is equivalent to the one that is going to be saved and if it is equivalent then “pretend” that data was saved at specified location without actually do this. This may be used to track duplicates using file names of body parts.
Type: Constant. Possible values are boolean: yes/no (true/false).
Default: no (false)

Macros of save method

The following engine specific macros are available for save method:

MacroProperties
$counter8
$counter16
$counter32
$counter64
Description: 8-, 16-, 32- and 64-bit counters respectively. Each counters starts from 1 and increments at each evaluation of a macro. A counter wraps around to zero when it reaches maximum value and then starts incrementing again. A counter is not persistent - it is destroyed when MPP stops. Main purpose of counters is to be used in generation of unique file names. For example the following template is used to generate unique temporary file names:
<tmp>
    <path>/var/tmp/mpp/$pid$time6$counter16</path>
</tmp>
Scope: any
Dimensions: none
Stages: all
Arguments: none
$action.base.path
$action.relative.path
$action.full.path
Description: Base, relative and full paths respectively as generated during action execution. This macros are intended to be used in explicit engine results to refer back to recent action results. Also values of this macros are implicit engine results with ID's base.path, relative.path and full.path respectively (note that action. prefix is omitted). For example the following explicit result exposes full.path with shorter ID path:
    <result id="path">
        <result>$action.full.path</result>
    </result>
Scope: results
Dimensions: same as action dimensions
Stages: all
Arguments: none

Implicit results of save method

The following implicit results are calculated by save method:

ResultProperties
base.path Description: Base path that was generated and used during action execution.
Dimensions: same as action dimensions
relative.path Description: Relative path that was generated and used during action execution.
Dimensions: same as action dimensions
full.path Description: Full path that was generated and used during action execution. It is a concatenation of base and relative paths.
Dimensions: same as action dimensions

Compare method

Compare method does comparison of data generated from a template with data from a file on disk. It works a following:

  • generates data from a provided data template;
  • generates full path from provided full path template or from base and relative path templates;
  • reads data (by 4k chunks) from a file at generated full path and compares the data with generated data;
  • exports the result of comparison as implicit result;

Example:

<mppd>
    <engines>
        <file>
            <compare id="compare">
                <full>
                    <path>$engines.db.search.full_path</path>
                </full>
                <data>$body_part</data>
            </compare>

            <save id="save">
                <precondition>file.compare.is_equal $EQ no</precondition>
                ...
            </save>

        </file>
    </engines>
</mppd>

In this example compare method compares file at full path obtained as a result of previous engine with data from message body part. Result of comparison is exported as implicit result with ID is_equal and is used in precondition of next save engine.

Options of compare method

The following options are available for compare method:

OptionProperties
<base> Description: Base path for a relative one. Ignore if <full> is defined.
Type: Path primitive.
Default: Inherited from engine's common option.
<relative> Description: Relative file path template for file to compare with data. Final full path for file data is combined from <base> and <relative> paths. Ignore if <full> is defined.
Type: Path primitive.
Default: No default – this is mandatory option if <full> is not defined.
<full> Description: Full file path template for file to compare with data. Options <base> and <relative> are ignored if this one is defined.
Type: Path primitive.
Default: No default – this is mandatory option if <relative> is not defined.
<data> Description: Template for data to compare file data with. Data template is evaluated and obtained string (possibly binary string) is compared with a file.
Type: Template. Evaluates to a string (binary). Spaces are removed.
Default: No default – this is mandatory option.

Macros of compare method

The following engine specific macros are available for compare method:

MacroProperties
$action.is_equal Description: Flag that indicates whether generated data and file data are equal -- yes value, or not -- no value.
Scope: results
Dimensions: same as action dimensions
Stages: all
Arguments: none

Implicit results of compare method

The following implicit results are calculated by compare method:

ResultProperties
is_equal Description: Flag that indicates whether generated data and file data are equal -- yes value, or not -- no value.
Dimensions: same as action dimensions

See also

Personal tools