XML::XQL - A perl module for querying XML tree structures with XQL

# NAME

XML::XQL - A perl module for querying XML tree structures with XQL

# SYNOPSIS

     use XML::XQL;
     use XML::XQL::DOM;

     $parser = new XML::DOM::Parser;
     $doc = $parser->parsefile ("file.xml");

     # Return all elements with tagName='title' under the root element 'book'
     $query = new XML::XQL::Query (Expr => "book/title");
     @result = $query->solve ($doc);

     # Or (to save some typing)
     @result = XML::XQL::solve ("book/title", $doc);

# DESCRIPTION

The XML::XQL module implements the XQL (XML Query Language) proposal submitted to the XSL Working Group in September 1998. The spec can be found at: http://www.w3.org/TandS/QL/QL98/pp/xql.html Most of the contents related to the XQL syntax can also be found in the XML::XQL::Tutorial that comes with this distribution. Note that XQL is not the same as XML-QL!

The current implementation only works with the XML::DOM module, but once the design is stable and the major bugs are flushed out, other extensions might follow, e.g. for XML::Grove.

XQL was designed to be extensible and this implementation tries to stick to that. Users can add their own functions, methods, comparison operators and data types. Plugging in a new XML tree structure (like XML::Grove) should be a piece of cake.

To use the XQL module, either

      use XML::XQL;

or

      use XML::XQL::Strict;

The Strict module only provides the core XQL functionality as found in the XQL spec. By default (i.e. by using XML::XQL) you get 'XQL+', which has some additional features.

See the section Additional Features in XQL+ for the differences.

This module is still in development. See the To-do list in XQL.pm for what still needs to be done. Any suggestions are welcome, the sooner these implementation issues are resolved, the faster we can all use this module.

If you find a bug, you would do me great favor by sending it to me in the form of a test case. See the file t/xql_template.t that comes with this distribution.

If you have written a cool comparison operator, function, method or XQL data type that you would like to share, send it to enno@att.com and I will add it to this module.

# XML::XQL global functions

solve (QUERY_STRING, INPUT_LIST...)

     @result = XML::XQL::solve ("doc//book", $doc);

  This is provided as a shortcut for:

       $query = new XML::XQL::Query (Expr => "doc//book");
       @result = $query->solve ($doc);

  Note that with XML::XQL::DOM, you can also write (see XML::DOM::Node for details):

       @result = $doc->xql ("doc//book");

defineFunction (NAME, FUNCREF, ARGCOUNT [, ALLOWED_OUTSIDE [, CONST, [QUERY_ARG]]])

  Defines the XQL function (at the global level, i.e. for all newly created queries) with the specified NAME. The ARGCOUNT parameter can either be a single number or a reference to a list with numbers. A single number expands to [ARGCOUNT, ARGCOUNT]. The list contains pairs of numbers, indicating the number of arguments that the function allows. The value -1 means infinity. E.g. [2, 5, 7, 9, 12, -1] means that the function can have 2, 3, 4, 5, 7, 8, 9, 12 or more arguments. The number of arguments is checked when parsing the XQL query string.

  The second parameter must be a reference to a Perl function or an anonymous sub. E.g. '\&my_func' or 'sub { ... code ... }'

  If ALLOWED_OUTSIDE (default is 0) is set to 1, the function or method may also be used outside subqueries in node queries. (See NodeQuery parameter in Query constructor)

  If CONST (default is 0) is set to 1, the function is considered to be "constant". See Constant Function Invocations for details.

  If QUERY_ARG (default is 0) is not -1, the argument with that index is considered to be a 'query parameter'. If the query parameter is a subquery, that returns multiple values, the result list of the function invocation will contain one result value for each value of the subquery. E.g. 'length(book/author)' will return a list of Numbers, denoting the string lengths of all the author elements returned by 'book/author'.

  Note that only methods (not functions) may appear after a Bang "!" operator. This is checked when parsing the XQL query string.

  See also: defineMethod

generateFunction (NAME, FUNCNAME, RETURN_TYPE [, ARGCOUNT [, ALLOWED_OUTSIDE [, CONST [, QUERY_ARG]]]])

  Generates and defines an XQL function wrapper for the Perl function with the name FUNCNAME. The function name will be NAME in XQL query expressions. The return type should be one of the builtin XQL Data Types or a class derived from XML::XQL::PrimitiveType (see Adding Data Types.) See defineFunction for the meaning of ARGCOUNT, ALLOWED_OUTSIDE, CONST and QUERY_ARG.

  Function values are always converted to Perl strings with xql_toString before they are passed to the Perl function implementation. The function return value is cast to an object of type RETURN_TYPE, or to the empty list [] if the result is undef. It uses expandType to expand XQL primitive type names. If RETURN_TYPE is "*", it returns the function result as is, unless the function result is undef, in which case it returns [].

defineMethod (NAME, FUNCREF, ARGCOUNT [, ALLOWED_OUTSIDE])

  Defines the XQL method (at the global level, i.e. for all newly created queries) with the specified NAME. The ARGCOUNT parameter can either be a single number or a reference to a list with numbers. A single number expands to [ARGCOUNT, ARGCOUNT]. The list contains pairs of numbers, indicating the number of arguments that the method allows. The value -1 means infinity. E.g. [2, 5, 7, 9, 12, -1] means that the method can have 2, 3, 4, 5, 7, 8, 9, 12 or more arguments. The number of arguments is checked when parsing the XQL query string.

  The second parameter must be a reference to a Perl function or an anonymous sub. E.g. '\&my_func' or 'sub { ... code ... }'

  If ALLOWED_OUTSIDE (default is 0) is set to 1, the function or method may also be used outside subqueries in node queries. (See NodeQuery parameter in Query constructor)

  Note that only methods (not functions) may appear after a Bang "!" operator. This is checked when parsing the XQL query string.

  See also: defineFunction

defineComparisonOperators (NAME => FUNCREF [, NAME => FUNCREF]*)

  Defines XQL comparison operators at the global level. The FUNCREF parameters must be a references to a Perl function or an anonymous sub. E.g. '\&my_func' or 'sub { ... code ... }'

  E.g. define the operators $my_op$ and $my_op2$:

       defineComparisonOperators ('my_op' => \&my_op,
                                  'my_op2' => sub { ... insert code here ... });

defineElementValueConvertor (TAG_NAME, FUNCREF)

  Defines that the result of the value() call for Elements with the specified TAG_NAME uses the specified function. The function will receive two parameters. The second one is the TAG_NAME of the Element node and the first parameter is the Element node itself. FUNCREF should be a reference to a Perl function, e.g. \&my_sub, or an anonymous sub.

  E.g. to define that all Elements with tag name 'date-of-birth' should return XML::XQL::Date objects:

      	defineElementValueConvertor ('date-of-birth', sub {
      		my $elem = shift;
      		# Always pass in the node as the second parameter. This is
      		# the reference node for the object, which is used when
      		# sorting values in document order.
      		new XML::XQL::Date ($elem->xql_text, $elem);
      	});

  These convertors can only be specified at a global level, not on a per query basis. To undefine a convertor, simply pass a FUNCREF of undef.

defineAttrValueConvertor (ELEM_TAG_NAME, ATTR_NAME, FUNCREF)

  Defines that the result of the value() call for Attributes with the specified ATTR_NAME and a parent Element with the specified ELEM_TAG_NAME uses the specified function. An ELEM_TAG_NAME of "*" will match regardless of the tag name of the parent Element. The function will receive 3 parameters. The third one is the tag name of the parent Element (even if ELEM_TAG_NAME was "*"), the second is the ATTR_NAME and the first is the Attribute node itself. FUNCREF should be a reference to a Perl function, e.g. \&my_sub, or an anonymous sub.

  These convertors can only be specified at a global level, not on a per query basis. To undefine a convertor, simply pass a FUNCREF of undef.

defineTokenQ (Q)

  Defines the token for the q// string delimiters at a global level. The default value for XQL+ is 'q', for XML::XQL::Strict it is undef. A value of undef will deactivate this feature.

defineTokenQQ (QQ)

  Defines the token for the qq// string delimiters at a global level. The default value for XQL+ is 'qq', for XML::XQL::Strict it is undef. A value of undef will deactivate this feature.

expandType (TYPE)

  Used internally to expand type names of XQL primitive types. E.g. it expands "Number" to "XML::XQL::Number" and is not case-sensitive, so "number" and "NuMbEr" will both expand correctly.

defineExpandedTypes (ALIAS, FULL_NAME [, ...])

  For each pair of arguments it allows the class name FULL_NAME to be abbreviated with ALIAS. The definitions are used by expandType(). (ALIAS is always converted to lowercase internally, because expandType is case-insensitive.)

  Overriding the ALIAS for "date", also affects the object type returned by the date() function.

setErrorContextDelimiters (START, END, BOLD_ON, BOLD_OFF)

  Sets the delimiters used when printing error messages during query evaluation. The default delimiters on Unix are `tput smul` (underline on) and `tput rmal` (underline off). On other systems (that don't have tput), the delimiters are ">>" and "<<" resp.

  When printing the error message, the subexpression that caused the error will be enclosed by the delimiters, i.e. underlined on Unix.

  For certain subexpressions the significant keyword, e.g. "$and$" is enclosed in the bold delimiters BOLD_ON (default: `tput bold` on Unix, "" elsewhere) and BOLD_OFF (default: (`tput rmul` . `tput smul`) on Unix, "" elsewhere, see $BoldOff in XML::XQL::XQL.pm for details.)

isEmptyList (VAR)

  Returns 1 if VAR is [], else 0. Can be used in user defined functions.

# Additional Features in XQL+

Parent operator '..'

  The '..' operator returns the parent of the current node, where '.' would return the current node. This is not part of any XQL standard, because you would normally use return operators, which are not implemented here.

Sequence operators ';' and ';;'

  The sequence operators ';' (precedes) and ';;' (immediately precedes) are not in the XQL spec, but are described in 'The Design of XQL' by Jonathan Robie who is one of the designers of XQL. It can be found at http://www.texcel.no/whitepapers/xql-design.html See also the XQL Tutorial for a description of what they mean.

q// and qq// String Tokens

  String tokens a la q// and qq// are allowed. q// evaluates like Perl's single quotes and qq// like Perl's double quotes. Note that the default XQL strings do not allow escaping etc., so it's not possible to define a string with both single and double quotes. If 'q' and 'qq' are not to your liking, you may redefine them to something else or undefine them altogether, by assigning undef to them. E.g:

       # at a global level - shared by all queries (that don't (re)define 'q')
       XML::XQL::defineTokenQ ('k');
       XML::XQL::defineTokenQQ (undef);

       # at a query level - only defined for this query
       $query = new XML::XQL::Query (Expr => "book/title", q => 'k', qq => undef);

  From now on k// works like q// did and qq// doesn't work at all anymore.

Query strings can have embedded Comments

  For example:

       $queryExpr = "book/title          # this comment is inside the query string
      	       [. = 'Moby Dick']"; # this comment is outside

Optional dollar delimiters and case-insensitive XQL keywords

  The following XQL keywords are case-insensitive and the dollar sign delimiters may be omitted: $and$, $or$, $not$, $union$, $intersect$, $to$, $any$, $all$, $eq$, $ne$, $lt$, $gt$, $ge$, $le$, $ieq$, $ine$, $ilt$, $igt$, $ige$, $ile$.

  E.g. $AND$, $And$, $aNd$, and, And, aNd are all valid replacements for $and$.

  Note that XQL+ comparison operators ($match$, $no_match$, $isa$, $can$) still require dollar delimiters and are case-sensitive.

Comparison operator: $match$ or '=~'

  E.g. "book/title =~ '/(Moby|Dick)/']" will return all book titles containing Moby or Dick. Note that the match expression needs to be quoted and should contain the // or m// delimiters for Perl.

  When casting the values to be matched, both are converted to Text.

Comparison operator: $no_match$ or '!~'

  E.g. "book/title !~ '/(Moby|Dick)/']" will return all book titles that don't contain Moby or Dick. Note that the match expression needs to be quoted and should contain the // or m// delimiters for Perl.

  When casting the values to be matched, both are converted to Text.

Comparison operator: $isa$

  E.g. '//. $isa$ "XML::XQL::Date"' returns all elements for which the value() function returns an XML::XQL::Date object. (Note that the value() function can be overridden to return a specific object type for certain elements and attributes.) It uses expandType to expand XQL primitive type names.

Comparison operator: $can$

  E.g. '//. $can$ "swim"' returns all elements for which the value() function returns an object that implements the (Perl) swim() method. (Note that the value() function can be overridden to return a specific object type for certain elements and attributes.)

Function: once (QUERY)

  E.g. 'once(id("foo"))' will evaluate the QUERY expression only once per query. Certain query results (like the above example) will always return the same value within a query. Using once() will cache the QUERY result for the rest of the query.

  Note that "constant" function invocations are always cached. See also Constant Function Invocations

Function: subst (QUERY, EXPR, EXPR [,MODIFIERS, [MODE]])

  E.g. 'subst(book/title, "[M|m]oby", "Dick", "g")' will replace Moby or moby with Dick globally ("g") in all book title elements. Underneath it uses Perl's substitute operator s///. Don't worry about which delimiters are used underneath. The function returns all the book/titles for which a substitution occurred. The default MODIFIERS string is "" (empty.) The function name may be abbreviated to "s".

  For most Node types, it converts the value() to a string (with xql_toString) to match the string and xql_setValue to set the new value in case it matched. For XQL primitives (Boolean, Number, Text) and other data types (e.g. Date) it uses xql_toString to match the String and xql_setValue to set the result. Beware that performing a substitution on a primitive that was found in the original XQL query expression, changes the value of that constant.

  If MODE is 0 (default), it treats Element nodes differently by matching and replacing text blocks occurring in the Element node. A text block is defined as the concatenation of the raw text of subsequent Text, CDATASection and EntityReference nodes. In this mode it skips embedded Element nodes. If a text block matches, it is replaced by a single Text node, regardless of the original node type(s).

  If MODE is 1, it treats Element nodes like the other nodes, i.e. it converts the value() to a string etc. Note that the default implementation of value() calls text(), which normalizes whitespace and includes embedded Element descendants (recursively.) This is probably not what you want to use in most cases, but since I'm not a professional psychic... :-)

Function: map (QUERY, CODE)

  E.g. 'map(book/title, "s/[M|m]oby/Dick/g; $_")' will replace Moby or moby with Dick globally ("g") in all book title elements. Underneath it uses Perl's map operator. The function returns all the book/titles for which a change occurred.

  ??? add more specifics

Function: eval (EXPR [,TYPE])

  Evaluates the Perl expression EXPR and returns an object of the specified TYPE. It uses expandType to expand XQL primitive type names. If the result of the eval was undef, the empty list [] is returned.

  E.g. 'eval("2 + 5", "Number")' returns a Number object with the value 7, and 'eval("%ENV{USER}")' returns a Text object with the user name.

  Consider using once() to cache the return value, when the invocation will return the same result for each invocation within a query.

  ??? add more specifics

Function: new (TYPE [, QUERY [, PAR] *])

  Creates a new object of the specified object TYPE. The constructor may have any number of arguments. The first argument of the constructor (the 2nd argument of the new() function) is considered to be a 'query parameter'. See defineFunction for a definition of query parameter. It uses expandType to expand XQL primitive type names.

Method: DOM_nodeType ()

  Returns the DOM node type. Note that these are mostly the same as nodeType(), except for CDATASection and EntityReference nodes. DOM_nodeType() returns 4 and 5 respectively, whereas nodeType() returns 3, because they are considered text nodes.

Function wrappers for Perl builtin functions

  XQL function wrappers have been provided for most Perl builtin functions. When using a Perl builtin function like "substr" in an XQL+ querry, an XQL function wrapper will be generated on the fly. The arguments to these functions may be regular XQL+ subqueries (that return one or more values) for a query parameter (see generateFunction for a definition.) Most wrappers of Perl builtin functions have argument 0 for a query parameter, except for: chmod (parameter 1 is the query parameter), chown (2) and utime (2). The following funcitons have no query parameter, which means that all parameters should be a single value: atan2, rand, srand, sprintf, rename, unlink, system.

  The function result is casted to the appropriate XQL primitive type (Number, Text or Boolean), or to an empty list if the result was undef.

# Implementation Details

XQL Builtin Data Types

  The XQL engine uses the following object classes internally. Only Number, Boolean and Text are considered primitive XQL types:

  * XML::XQL::Number

    For integers and floating point numbers.

  * XML::XQL::Boolean

    For booleans, e.g returned by true() and false().

  * XML::XQL::Text

    For string values.

  * XML::XQL::Date

    For date, time and date/time values. E.g. returned by the date() function.

  * XML::XQL::Node

    Superclass of all XML node types. E.g. all subclasses of XML::DOM::Node subclass from this.

  * Perl list reference

    Lists of values are passed by reference (i.e. using [] delimiters). The empty list [] has a double meaning. It also means 'undef' in certain situations, e.g. when a function invocation or comparison failed.

Type casting in comparisons

  When two values are compared in an XML comparison (e.g. $eq$) the values are first casted to the same data type. Node values are first replaced by their value() (i.e. the XQL value() function is used, which returns a Text value by default, but may return any data type if the user so chooses.) The resulting values are then casted to the type of the object with the highest xql_primType() value. They are as follows: Node (0), Text (1), Number (2), Boolean (3), Date (4), other data types (4 by default, but this may be overriden by the user.)

  E.g. if one value is a Text value and the other is a Number, the Text value is cast to a Number and the resulting low-level (Perl) comparison is (for $eq$):

       $number->xql_toString == $text->xql_toString

  If both were Text values, it would have been

       $text1->xql_toString eq $text2->xql_toString

  Note that the XQL spec is vague and even conflicting where it concerns type casting. This implementation resulted after talking to Joe Lapp, one of the spec writers.

Adding Data Types

  If you want to add your own data type, make sure it derives from XML::XQL::PrimitiveType and implements the necessary methods.

  I will add more stuff here to explain it all, but for now, look at the code for the primitive XQL types or the Date class (XML::XQL::Date in Date.pm.)

Document Order

  The XQL spec states that query results always return their values in document order, which means the order in which they appeared in the original XML document. Values extracted from Nodes (e.g. with value(), text(), rawText(), nodeName(), etc.) always have a pointer to the reference node (i.e. the Node from which the value was extracted.) These pointers are acknowledged when (intermediate) result lists are sorted. Currently, the only place where a result list is sorted is in a $union$ expression, which is the only place where the result list can be unordered. (If you find that this is not true, let me know.)

  Non-node values that have no associated reference node, always end up at the end of the result list in the order that they were added. The XQL spec states that the reference node for an XML Attribute is the Element to which it belongs, and that the order of values with the same reference node is undefined. This means that the order of an Element and its attributes would be undefined. But since the XML::DOM module keeps track of the order of the attributes, the XQL engine does the same, and therefore, the attributes of an Element are sorted and appear after their parent Element in a sorted result list.

Constant Function Invocations

  If a function always returns the same value when given "constant" arguments, the function is considered to be "constant". A "constant" argument can be either an XQL primitive (Number, Boolean, Text) or a "constant" function invocation. E.g.

       date("12-03-1998")
       true()
       sin(0.3)
       length("abc")
       date(substr("12-03-1998 is the date", 0, 10))

  are constant, but not:

       length(book[2])

  Results of constant function invocations are cached and calculated only once for each query. See also the CONST parameter in defineFunction. It is not necessary to wrap constant function invocations in a once() call.

  Constant XQL functions are: date, true, false and a lot of the XQL+ wrappers for Perl builtin functions. Function wrappers for certain builtins are not made constant on purpose to force the invocation to be evaluated every time, e.g. 'mkdir("/user/enno/my_dir", "0644")' (although constant in appearance) may return different results for multiple invocations. See %PerlFunc in Plus.pm for details.

Function: count ([QUERY])

  The count() function has no parameters in the XQL spec. In this implementation it will return the number of QUERY results when passed a QUERY parameter.

Method: text ([RECURSE])

  When expanding an Element node, the text() method adds the expanded text() value of sub-Elements. When RECURSE is set to 0 (default is 1), it will not include sub-elements. This is useful e.g. when using the $match$ operator in a recursive context (using the // operator), so it won't return parent Elements when one of the children matches.

Method: rawText ([RECURSE])

  See text().

# SEE ALSO

XML::XQL::Query, XML::XQL::DOM, XML::XQL::Date

The Japanese version of this document can be found on-line at http://member.nifty.ne.jp/hippo2000/perltips/xml/xql.htm

The XML::XQL::Tutorial manual page. The Japanese version can be found at http://member.nifty.ne.jp/hippo2000/perltips/xml/xql/tutorial.htm

The XQL spec at http://www.w3.org/TandS/QL/QL98/pp/xql.html

The Design of XQL at http://www.texcel.no/whitepapers/xql-design.html

The DOM Level 1 specification at http://www.w3.org/TR/REC-DOM-Level-1

The XML spec (Extensible Markup Language 1.0) at http://www.w3.org/TR/REC-xml

The XML::Parser and XML::Parser::Expat manual pages.

# AUTHOR

Please send bugs, comments and suggestions to Enno Derksen <enno@att.com>

Last updated: Fri Oct 22 18:47:39 1999