    All Packages  Class Hierarchy  This Package  Previous  Next  Index

# Class sdsu.util.SimpleTokenizer

    java.lang.Object
       |
       +----sdsu.util.SimpleTokenizer

public class SimpleTokenizer
extends Object

This class performs some simple parsing of strings or streams. The input is a sequence of ascii characters. The sequence is divided into tokens, whitespace, and comments. Comments start with the comment character and continue to the next newline (\n ) character. Comments are removed from the input characters and not returned as part of a token. A token is string from the current location to the next separator or whitespace character. Characters defined as whitespace (tab, newline, and space default values ) help delineate tokens but are not part of tokens. That is whitespace characters are removed after finding a token. If a token must contain whitespace character, a possible separator, or comment character, the token must be placed between two quote characters. A quoted token can contain a quote character.

Author:
  Roger Whitney (whitney@cs.sdsu.edu)
See Also:
  Stringizer, TokenCharacters

##

SimpleTokenizer(InputStream)
  Create a SimpleTokenizer on tokenSource with default settings
SimpleTokenizer(InputStream, TokenCharacters)
  Create a SimpleTokenizer on tokenSource
SimpleTokenizer(String)
  Create a SimpleTokenizer on string with default settings
SimpleTokenizer(String, TokenCharacters)
  Create a SimpleTokenizer on string

##

hasMoreElements()
  Returns true if not at end of source stream or source string
hasMoreTokens()
  Returns true if not at end of source stream or source string
nextToken()
  Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found.
nextToken(String)
  Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found.
separator()
  Returns the separator found by the last call to nextToken
setEatEscapeChar(boolean)
  Enable/Disable the eating of the escape character.

##

SimpleTokenizer

     public SimpleTokenizer(String parsable)

  Create a SimpleTokenizer on string with default settings

SimpleTokenizer

     public SimpleTokenizer(String parsable,
                            TokenCharacters charTable)

  Create a SimpleTokenizer on string

  Parameters:
    commentChar - character used to indicate start of a comment
    quoteChar - character used to quote a string containing special characters
    whitespace - characters used for whitespace. Use null or empty string for no whitespace characters

SimpleTokenizer

     public SimpleTokenizer(InputStream tokenSource)

  Create a SimpleTokenizer on tokenSource with default settings

SimpleTokenizer

     public SimpleTokenizer(InputStream tokenSource,
                            TokenCharacters charTable)

  Create a SimpleTokenizer on tokenSource

  Parameters:
    commentChar - character used to indicate start of a comment
    beginQuoteChar - character used to start a quote of a string containing special characters
    endQuoteChar - character used to end a quote of a string containing special characters
    whitespace - characters used for whitespace. Use null or empty string for no whitespace characters

##

hasMoreTokens

     public boolean hasMoreTokens()

  Returns true if not at end of source stream or source string

hasMoreElements

     public boolean hasMoreElements()

  Returns true if not at end of source stream or source string

separator

     public char separator()

  Returns the separator found by the last call to nextToken

setEatEscapeChar

     public void setEatEscapeChar(boolean value)

  Enable/Disable the eating of the escape character.

nextToken

     public String nextToken(String newSeparators) throws IOException

  Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found. The separator is removed from the stream, but not returned as part of token.

  Parameters:
    separator - can be any character except the current comment or quote character
  Throws: IOException
    If separator or EOF does not follow this token

nextToken

     public String nextToken() throws IOException

  Returns string containing all characters up to the given separator, unquoted whitespace, or EOF if the separator is not found. The separator is removed from the stream, but not returned as part of token.

  Parameters:
    separator - set of characters to be used as separator after token. Can be any non-null or nonempty string of characters except the current comment or quote character
  Throws: IOException
    If separator or EOF does not follow this token

    All Packages  Class Hierarchy  This Package  Previous  Next  Index