    All Packages  Class Hierarchy  This Package  Previous  Next  Index

# Class sdsu.util.TokenCharacters

    java.lang.Object
       |
       +----sdsu.util.TokenCharacters

public class TokenCharacters
extends Object

This class maintains special characters used in parsing strings into tokens. It keeps track of whitespace, characters to indicate the start of a comment, quote characters used to quote tokens that contain special characters, an escape character, and characters that separate tokens.

Author:
  Roger Whitney (whitney@cs.sdsu.edu)
See Also:
  SimpleTokenizer, Stringizer

##

COMMENT_CHAR
  Default character (#) used to indicate start of comment.
ESCAPE_CHAR
  Character (/) used to preceed a quote character or escape character in a quoted token.
QUOTE_CHAR
  Default character (') used to delineate the start and end of a quoted token.
WHITESPACE
  Default characters treated as whitespace.

##

TokenCharacters()
  Create TokenCharacters with default values.
TokenCharacters(String)
  Create TokenCharacters with given characters for token separators and default values for the rest of parameters.
TokenCharacters(String, String, char, char, String)
  Create a TokenCharacters object with given values

##

addQuoteChars(char, char)
  Add the quote pair beginQuote-endQuote to the pairs recognized as char pairs to quote a token.
containsEscapeableChar(String)
  Returns true if c needs to be escaped in a quoted token.
containsTokenTerminator(String)
  Returns true if contains a character that indicates the end of a token.
escapeToken(String)
  Places escape character before any quote character or the escape character Returns the modified token
getCommentChar()
  Returns a character that indicates start of a comment.
isBeginQuote(char)
  Returns true if c indicates the start of a quoted token
isComment(char)
  Returns true if c indicates the start of a comment
isEndQuote(char)
  Returns true if c indicates the end of a quoted token
isEOL(char)
  Returns true if c is Mac, Unix, or PC EOL character
isEscape(char)
  Returns true if c is an escape character
isQuotePair(char, char)
  Returns true if beginQuote and endQuote are matching begin ending quote characters
isSeparator(char)
  Returns true if c is a separator character
isTokenTerminator(char)
  Returns true if c indicates the end of an unquoted token IE c is a whitespace, separator or comment character
isWhitespace(char)
  Returns true if c is a whitespace character
quoteToken(String)
  Surrounds token with begin-end quote pair Returns the quoted token
requiresEscaping(char)
  Returns true if c needs to be escaped in a quoted token.
setSeparatorChars(String)
  Set the current set of separators to newSeparators

##

WHITESPACE

     public static final String WHITESPACE

  Default characters treated as whitespace. Default is space, tab, line feed (ascii 10, \n in unix) and carriage return ( ascii 13, usd as newline char on PC and Macs )

ESCAPE_CHAR

     public static final char ESCAPE_CHAR

  Character (/) used to preceed a quote character or escape character in a quoted token.

COMMENT_CHAR

     public static final String COMMENT_CHAR

  Default character (#) used to indicate start of comment.

QUOTE_CHAR

     public static final char QUOTE_CHAR

  Default character (') used to delineate the start and end of a quoted token. Tokens are quoted when they contain special characters.

##

TokenCharacters

     public TokenCharacters()

  Create TokenCharacters with default values. You must set separators before using the new object.

TokenCharacters

     public TokenCharacters(String separators)

  Create TokenCharacters with given characters for token separators and default values for the rest of parameters.

TokenCharacters

     public TokenCharacters(String separators,
                            String commentChars,
                            char beginQuoteChar,
                            char endQuoteChar,
                            String whitespace)

  Create a TokenCharacters object with given values

  Parameters:
    commentChar - character used to indicate start of a comment
    beginQuoteChar - character used to start a quote of a string containing special characters
    endQuoteChar - character used to end a quote of a string containing special characters
    whitespace - characters used for whitespace. Use null or empty string for no whitespace characters

##

addQuoteChars

     public void addQuoteChars(char beginQuote,
                               char endQuote)

  Add the quote pair beginQuote-endQuote to the pairs recognized as char pairs to quote a token.

getCommentChar

     public char getCommentChar()

  Returns a character that indicates start of a comment.

setSeparatorChars

     public void setSeparatorChars(String newSeparators)

  Set the current set of separators to newSeparators

isEOL

     public boolean isEOL(char c)

  Returns true if c is Mac, Unix, or PC EOL character

isEscape

     public boolean isEscape(char c)

  Returns true if c is an escape character

isWhitespace

     public boolean isWhitespace(char c)

  Returns true if c is a whitespace character

isSeparator

     public boolean isSeparator(char c)

  Returns true if c is a separator character

isBeginQuote

     public boolean isBeginQuote(char c)

  Returns true if c indicates the start of a quoted token

isEndQuote

     public boolean isEndQuote(char c)

  Returns true if c indicates the end of a quoted token

isComment

     public boolean isComment(char c)

  Returns true if c indicates the start of a comment

isTokenTerminator

     public boolean isTokenTerminator(char c)

  Returns true if c indicates the end of an unquoted token IE c is a whitespace, separator or comment character

isQuotePair

     public boolean isQuotePair(char beginQuote,
                                char endQuote)

  Returns true if beginQuote and endQuote are matching begin ending quote characters

requiresEscaping

     public boolean requiresEscaping(char c)

  Returns true if c needs to be escaped in a quoted token. That is if c is a quote character or the escape character

containsEscapeableChar

     public boolean containsEscapeableChar(String token)

  Returns true if c needs to be escaped in a quoted token. That is if c is a quote character or the escape character

escapeToken

     public String escapeToken(String token)

  Places escape character before any quote character or the escape character Returns the modified token

containsTokenTerminator

     public boolean containsTokenTerminator(String token)

  Returns true if contains a character that indicates the end of a token. That is whitespace, comment char or a separator

quoteToken

     public String quoteToken(String token)

  Surrounds token with begin-end quote pair Returns the quoted token

    All Packages  Class Hierarchy  This Package  Previous  Next  Index