
package String::Tokenizer;

use strict;
use warnings;

our $VERSION = '0.02';

### constructor

sub new {
	my ($_class, @args) = @_;
	my $class = ref($_class) || $_class;
	my $string_tokenizer = {
		tokens => []
		};
	bless($string_tokenizer, $class);
	$string_tokenizer->tokenize(@args) if @args;
	return $string_tokenizer;
}

### methods

sub tokenize {
	my ($self, $string, $delimiter) = @_;
	# split everything by whitespace no matter what
	# (possible multiple occurances of white space too) 
	my @tokens = split /\s+/ => $string;
	if ($delimiter) {
		# create the delimiter reg-ex
		# escape all non-alpha-numeric 
		# characters, just to be safe
		$delimiter = join "\|" => map { s/(\W)/\\$1/g; $_ } split // => $delimiter;
		# loop through the tokens
		@tokens = map {
				# if the token contains a delimiter then ...
				if (/$delimiter/) {
					my ($token, @_tokens);
					# split the token up into characters
					# and the loop through all the characters
					foreach my $char (split //) {
						# if the character is a delimiter
						if ($char =~ /^$delimiter$/) {
							# and we already have a token in the works
							if (defined($token) && $token =~ /^.*$/) {
								# add the token to the 
								# temp tokens list
								push @_tokens => $token;
							}
							# and then push our delimiter character
							# onto the temp tokens list
							push @_tokens => $char;
							# now we need to undefine our token
							$token = undef;
						}
						# if the character is not a delimiter then
						else {
							# check to make sure the token is defined
							$token = "" unless defined $token;
							# and then add the chracter to it
							$token .= $char;
						}
					}
					# now push any remaining token onto 
					# the temp tokens list
					push @_tokens => $token if $token;
					# and return tokens
					@_tokens;
				}
				# if our token does not have 
				# the delimiter in it
				else {
					# just return it
					$_
				}
			} @tokens;
	}
	$self->{tokens} = \@tokens;
}

sub getTokens {
	my ($self) = @_;
	return wantarray ?
			@{$self->{tokens}}
			:
			$self->{tokens};
}

1;

__END__

=head1 NAME

String::Tokenizer - A simple string tokenizer.

=head1 SYNOPSIS

  use String::Tokenizer;
  
  # create the tokenizer and tokenize input
  my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');
  
  # create tokenizer
  my $tokenizer = String::Tokenizer->new();
  # ... then tokenize the string
  $tokenizer->tokenize("((5 + 5) - 10)", '()');
  
  # will print '(, (, 5, +, 5, ), -, 10, )'
  print join ", " => $tokenizer->getTokens();

=head1 DESCRIPTION

A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. 

This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple C<split / /, $string> and the other options that involve much larger and complex modules.

Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.

=head1 METHODS

=over 4

=item new ($string, $delimiters)

If you do not supply any parameters, nothing happens, the instance is just created. But if you do supply parameters, they are passed on to the C<tokenize> method and that method is run. For information about those arguments, see C<tokenize> below.

=item tokenize ($string, $delimiters)

Takes a C<$string> to tokenize, and optionally a set of C<$delimiter> characters to facilitate the tokenization. The C<$string> parameter is obvious, the C<$delimiter> parameter is not as transparent. C<$delimiter> is a string of characters, these characters are then seperated into individual characters and are used to split the C<$string> with. So given this string:

  (5 + (100 * (20 - 35)) + 4)

The C<tokenize> method without a C<$delimiter> parameter would return the following comma seperated list of tokens:

  (5, +, (100, *, (20, -, 35)), +, 4)

However, if you were to pass the following set of delimiters [C<(, )>] to C<tokenize>, you would get the following comma seperated list of tokens:

  (, 5, +, (, 100, *, (, 20, -, 35, ), ), +, 4, )

We now can differentiate the parens from the numbers, and no globbing occurs. If you wanted to allow for optionally leaving out the whitespace in the expression, like this:

  (5+(100*(20-35))+4)

as some languages do. Then you would give this delimiter [C<+*-()>] to arrive at the same result.

=item getTokens

Simply returns the array of tokens. It returns an array-ref in scalar context.

=back

=head1 TO DO

This module is very simple, which is much of what I am going for with it. But still it could use some other features. 

One is a better means of token iteration, once they are generated they are available as an array, which is good, but a custom token iterator or set of iterators might be nice. Ones specicially desinged to serve the lexical-analysis and parsing needs. B<String::Tokeniser> actually provides something like that I am talking about. 

Another possible feature is the ability to subsitute regular expressions as delimiters. I had given this a lot of thought originally, but had opted not too, since it had the potential to greatly incresed the complexity of things, and I was really striving for simplicity. But as they say, easy things should be easy, and hard thing possible, so we shall see.

=head1 BUGS

None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to fix it. 

=head1 CODE COVERAGE

I use B<Devel::Cover> to test the code coverage of my tests, below is the B<Devel::Cover> report on this module's test suite.

 ---------------------------- ------ ------ ------ ------ ------ ------ ------
 File                           stmt branch   cond    sub    pod   time  total
 ---------------------------- ------ ------ ------ ------ ------ ------ ------
 /String/Tokenizer.pm          100.0  100.0   50.0  100.0  100.0   27.7   94.9
 t/10_String_Tokenizer_test.t  100.0    n/a    n/a    n/a    n/a  100.0  100.0
 ---------------------------- ------ ------ ------ ------ ------ ------ ------
 Total                         100.0  100.0   50.0  100.0  100.0  100.0   95.9
 ---------------------------- ------ ------ ------ ------ ------ ------ ------

=head1 SEE ALSO

The interface and workings of this module are based largely on the StringTokenizer class from the Java standard library. 

Below is a short list of other modules that might be considered similar to this one. I put this here for 2 reasons. One is that this module might be to simple for your usage, and I provide here a list of others that might be of use to you instead. And secondly because I feel that people tend to be hasty in declaring something "CPAN pollution" and "reinventing the wheel". There are many good modules out there, but they don't always fit peoples needs. Some are abandoned, and no longer maintained, others lack good test suites, and still more have just grown too complex with features and such to be useful in some contexts. With this module I aim to provide a simple clean String Tokenizer, based largely on the one found in the Java standard library. 

=over 4

=item B<String::Tokeniser>

This module looks as if it hasnt been updated from 0.01 and that was uploaded in since 2002. So my guess is it has been abandonded. Along with being a tokenizer, it also provides a means of moving through the resulting tokens, allowing for skipping of tokens and such. These are nice features I must admit, and possibly these (or similar) features may make it into String::Tokenizer in future releases. 

=item B<Parse::Tokens>

This one hasn't been touched since 2001, although it did get up to version 0.27. It looks to lean over more towards the parser side than a basic tokenizer. 

=item B<Text::Tokenizer>

This one looks more up to date (updated as recently as March 2004), but is both a lexical analyzer and a tokenizer. It also uses XS, mine is Pure Perl. This is something maybe to look into if you were to need a more beefy solution that what B<String::Tokenizer> provides.

=back

=head1 AUTHOR

stevan little, E<lt>stevan@iinteractive.comE<gt>

=head1 COPYRIGHT AND LICENSE

Copyright 2004 by Infinity Interactive, Inc.

L<http://www.iinteractive.com>

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself. 

=cut
