Text representation
ASCII
-
Text is usually represented as a simple string of characters.
-
The characters are members of a character set.
-
One of the most commonly used character sets is ASCII.
-
The ASCII character set is represented in 7 bits.
-
With 7 bits 128 different characters can be represented.
-
The characters are stored in bytes of 8 bytes.
-
So another 128 characters can be used (usually as special characters of
different text processing systems)
-
Tommi's Page about ASCII
character set:
-
The basic ASCII
set
ISO Character Sets
ISO (International Standardization Organization) has standardized the ASCIIs
extensions which contain also "non-english" characters accented charaecters
(finnish, spanish, swedish, et c.)
Unicode Character Set
Java has adopted the Unicode character set. Unicode characters are two-byte
characters. Most international character sets are included in the Unicode
character set.
Marked-up text
HTML, SGML and LATEX are the most common mark-up systems for text. Some
special characters are reserved as delimiters (in HTML "<" and " >"
). Special reserved words represent typically different formatting commands
in ASCII files. Older text processing systems use some special characters
to denote formatting on screen or printer.
Structured text
Text can be structured to emphasize the content of text (chapters - pages
- index - et c.) In a simple ASCII text these can not be represented.
The structure usually partitions the text in hierarchy
Most text processing systems have their own structures making it difficult
to exchange text between systems (Word, WordPerfert, ...)
Some standards exists.
Hypertext
Hypertext is a form of structured document which may consist of several
text files connected with links.
Operations
Character operations
Typically a text file is s string of characters each occupying a byte in
the string. A Character operation is any operation that assigns a value
on a character position, checks the value of a character position or changes
the content of a character position.
| T |
H |
I |
S |
_ |
I |
S |
_ |
T |
E |
X |
T |
_ |
_ |
_ |
Character operations don't consider the text as consisting of words or
any such entities. They consider each character as a separate unit.
String operations
Special operation can be defined on text if words are defined in the text
as separate objects. In most text processing systems the concept of word
has special purpose and words vcan be used in operations (move one word
forward, move word, compare words, make index of words.) Words can be defined
as separate structures or the word can be defined implicietely be defining
separators that delimit a word (SPACE , . ; : " " CR LF )
Text can be represented as a sequence of words.
Editing
Select, Cut, Paste, Move, Change.
Editing operations can operate on characters or words.
Formatting
Typically a sequence of characters can be assigned a set of attributes
that define the appearance of the text. Attributes can appear as TAG:s
or Markups (as in HTML and SGML) or as special built in attributes as in
Word. (In Word-text processing system the attributes are stored at the
end of the sequence that they format) For example the appearance of a paragraph
is situated at the end of the paragraph. - The font is assigned at the
end of the text string of that particular font.
Pattern-matching and searching
Typical actions:
-
Search,
-
Replace,
-
Global replace.
Efficient pattern matching and string search requires an index to be built
of the words of the text. Faloutsos (1985).
Sorting
The user may want to represent the text sorted in ascending or descending
order of lines. One form of sorting is so called KWIC - sort (Key Words
In Context). Lines of text (e.g. Titles of books, articles, Web pages)
are sorted with respect to the words that appear in the text. ( see for
example some index catalogues, say Computing Literature by ACM). An example
of a KWIC
index
Compression
See Shannon:
Information Theory
Text can be compressed to enable faster transmission of text of storage
of text in smaller space. See for example Compression
in MM applications
Typical compression algorithms are
Huffman
-
Text is analysed to find the frequencies of letters (or groups of letters)
-
codes of variable lengths are assigned to each of the letter in an optimal
manner: shorter codes for frequent letters - longer for less frequent
-
text is stored using these codes together with the code table.
Ziv-Lempel
-
a table is constructed for the most frequently appearing strings in the
text
-
these strings are replaced by the table indexes
The compression methods can compress a standard text file to about one
half. If information about the topic of the text could be utilize one could
reach compression of about 1:4.
This is the result of the famous Shannon test. (Also known as the hangman-game)
-
I think of a sentence
-
You have to guess it letter by letter
-
Count the number of guesses for each letter
-
After you guess the whole sentence, count the average number of guesses.
-
This number can be then compared with the ordinary text representation,
which (depending how you define the basic assumptions) can be 256 (ASCII)
or 28 (number of letters in alphabet) or something in between.
Encryption
Several methods exist for hiding the informatioon in a message , or text
file so that only the people who know the key to the method can read it.
Used in hiding passwords, messages, bank messages, text in general. See
e.g. DES
Language-Specific operations
Spelling - Grammatical analysis
mj@uwasa.fi 24.2.97