ECE291

Computer Engineering II

Lockwood, Spring 1998

Introduction

Non-lossy compression algorithms, such as the one studied in this MP, completely preserve the data they compress. By representing the most frequently recurring data patterns using the smallest bit sequence, the total size of the data can be greatly reduced.

In this MP, we will write an interactive assembly program that encodes and decodes text messages using Huffman compression. The text message will consist of symbols that include: the 26 letters ('A'..'Z'), the space (' '), and the asterisk ('*'). Each symbol will be represented using a variable-bit-length encoding.

Huffman Encoding

The main body of the algorithm involves sequentially combining the least frequenctly occuring pairs of nodes into a tree-structured supernode. The parent of a supernode is given a weight equal to the sum of the weighs of the children. The algorithm continues combining nodes and supernodes until all nodes have been combined into a single tree. Once finished, all of the symbols appear as leaves in the tree.

The encoding of the symbols is determined by assigning '0' and '1' values to the branches of the tree. The encoded value is determined by following the branches of the tree, starting at the root. The number of bits required to represent a symbol is equal to the depth of the leaf. Frequent letters are represented with fewer bits, while infrequent letters are represented with more.

For this MP, you are given the Huffman encoding table. This table was generated using Prof. Lockwood's PhD dissertation as a representative sample of standard writing. (The question of whether or not Prof. Lockwood's writing actually reflects standard writing is an entirely different question...). A Perl script was run that scanned the entire file and counted the frequency of each letter in the document. The results of this are shown below:

e 0.121 t 0.105 i 0.081 o 0.072 a 0.068 n 0.065 s 0.062 r 0.061 c 0.046 h 0.044 l 0.043 u 0.033 d 0.033 p 0.030 m 0.028 f 0.024 g 0.019 b 0.017 w 0.014 v 0.012 y 0.009 k 0.005 q 0.004 x 0.003 z 0.001 j 0.001

As expected, vowels and common letters such as 'E', 'T', 'I', and 'O' appeared most often. The letters 'Z' and 'J' appeared least frequently. Using the algorithm described above, the following encoding tree was generated: (The space and asterisk symbols were added later by splitting the nodes for nodes for S and V)

In this tree, the letter 'E' can be found by following the right-left-right branches. By assigning '0' and and '1' values to the left and right branches, respectively, this corresponds to an encoding pattern of 101. Likewise, the letter 'Z' has an encoding of: 110000110. Note that decoding of Huffman codes is somewhat tricky due to the fact that symbols can be represented with a differing number of bits.

Messages and documents can be formed by appending symbols together. The message 'HELLO', for example, can be compactly represented with only 22 bits as: 00101 101 00100 00100 0001. To clearify the encoding, spaces were shown between the symbols. In memory, however, adjacent symbols would occupy adjacent bit positions. In general, we will be treating memory as an array of bits. Because the symbols do not fall on even byte boundries, it is up to the decoding algorithm to decide where one symbol stops and the next begins.

User Interface

• You are given the framework of a program which provides a menu-driven interface.

• By selecting an option from the menu, the user can:
  • Enter text message or binary data
  • Print the contents of the message or buffer
  • Encode a message or decode the buffer

• All binary data is entered and displayed from left-to-right with the Least Significant Bit (LSB) first.

Sample Input & Output

• Results from a sample run of the program are shown below:
  (Boldface letters are were provided as input)

  ---------- MP2 Menu ----------- Enter (T)ext (B)inary Print (M)essage (R)buffeR Huffman (E)ncode (D)ecode -------- [ESC] = exit --------- B Enter Binary Data (LSBit .. MSBit): 101 D Text Message= E T Enter Text Message: Z E Compressed Size = 9 bits ~= 1 bytes. (LSBit .. MSBit) 11000011 0 T Enter Text Message: HELLO E Compressed Size = 22 bits ~= 2 bytes. (LSBit .. MSBit) 00101101 00100001 000001

• To understanding the program, run the library-based MP2.EXE program interactively. Try compressing and decompressing your own data.

• A sample input data, test1.in, is included with this MP.
  Use the following command to pipes this file into MP2:
  mp2 < test1.in
  Your results should match test1.out.

• Your program should work for all types of data.
  TAs will provide you with additional test cases when you demo your MP.

Data Structures

• A few variables have already been defined for you in the program framework.
  • TextMsg: String of ASCII bytes holding un-encoded text.
    The end of this string should be always be marked with the '$' character.

  • Buffer: A packed array of bytes that hold the Huffman-encoded data.
    The first huffman code begins with the least significant bit (LSBit) of the first byte (Buffer[0]) in the array. The next huffman code begins at the next bit location in the buffer. Note that one variable-length symbol may overlap multiple byte locations, and that no bits are wasted between symbols.

  • BufferLength: A word which indicates how many bits are stored in buffer.

  • HuffCodes: An array of structures that that define how symbols are encoded.
    This table is defined in the file HUFFCODE.INC and is Include'd into MP2.ASM. For each of the 28 symbols (letters+space+asterisk), this array contains: the ASCII letter, the Huffman-encoded bit pattern (in two formats), and the number of bits in the symbol. The DownEncoding specifies the symbol with the most significant bit listed first. The UpEncoding specifies the same symbol in reverse bit order (i.e., with the Least significant bit first). Both formats are given for your convenience. It is easy to confuse the bit ordering. Recall that MASM and CodeView both present data with the most significant bit listed first.

• A few constants have also been defined:
  • TextMsgMaxLength == 70 bytes,
    i.e., Just less than one line of full-screen text.

  • BufferMaxLengthBits == 9*TextMsgMaxLength bits,
    i.e., the longest encoding times the longest Buffer length.

  • BufferMaxLength == 1+BufferMaxLengthBits/8 bytes,
    Number of bytes required to hold Buffer, rounded up to next byte.

Procedures

• This assignment has eight procedures. You will receive credit for this assignment by replacing each of the eight procedures listed below with your own code.
• Experiment with the working code to gain a full understanding of how the programs works.
• Your program should exactly match the functionality of the library subroutines.
• All subroutines should be modular. They should use the stack to preserve the value of any registers they may modify.
• Five of the fifty points are earned by comparing the effiency of your your code against the library routines.

• PrintTextMsg
  • Purpose: Prints the contents of the TextMsg variable.
  • Inputs: TextMsg variable
  • Outputs: Writes to screen
  • Hints: Use LIB291 Functions where appropriate.
  • Points: 5

• ReadTextMsg
  • Purpose: Read TextMsg from the keyboard.
    • Converts lowercase letters to uppercase letters
    • Allows backspacing (BS = ASCII 8)
    • Ignores line feeds (LF = ASCII 10)
    • Prevents Underflow and Overflow of variables
      (i.e., writing past the beginning or end of TextMsg)
    • Terminates with a carriage return (CR = ASCII 13)
    • Rejects any invalid input and beeps
    • Marks end of TextMsg with the '$'.
  • Inputs: Keyboard
  • Outputs: TextMsg variable
  • Hint: To erase a character from the screen, print a backspace, then whitespace, then backspace.
  • Points: 5

• PrintBuffer
  • Purpose: Prints the size contents of the binary Buffer array
    • Prints 8 bits at a time
    • Prints least significant bit of each symbol first.
    • Prints 5 bytes (40 bits) per line
  • Inputs: Buffer & BufferLength variables
  • Outputs: Writes to screen
  • Hint: Review your shifting techniques and modulo arithmetic!
  • Points: 5

• ReadBuffer
  • Purpose: Read binary Buffer from the keyboard.
    • Accepts only 0's and 1's
    • Allows backspacing
    • Prevents Underflow and Overflow of variable
    • Terminates with a carriage return (CR = ASCII 13)
    • Ignores line feeds (LF = ASCII 10)
    • Rejects any invalid input and beeps
    • Sets BufferLength = number of bits read
  • Inputs: Keyboard
  • Outputs: Buffer and BufferLength variable
  • Points: 5

• Encode
  • Purpose: Encodes a single ASCII character into a huffman symbol, as defined above.
  • Input: AL = ASCII character
  • Outputs:
    • AX = Symbol
    • DH = Symbol-size (Number of bits)
  • Notes: This function is called by EncodeHuff
  • Points: 5

• AppendBufferN
  • Purpose: Appends a variable-length symbol to the end of the Buffer array.
  • Inputs:
    • AX = Symbol
    • DH = Symbol-size (Number of bits)
  • Output: Appends symbol-size bits to Buffer then adds symbol-size to BufferLength
  • Notes: This function is called by EncodeHuff
  • Points: 5

• EncodeHuff
  • Purpose: Encodes TextBuf to Buffer as described above
  • Input: TextMsg
  • Output: Buffer & BufferLength Variables
  • Notes: Calls AppendBufferN and Encode
  • Points: 5

• DecodeHuff
  • Purpose: Decodes Buffer to TextBuf as described above
  • Inputs: Buffer & BufferLength Variables
  • Output: TextMsg
  • Hints: Create a suitable data stucture to assist in decoding.
  • Points: 10

• Performance
  • Purpose: You are encouraged to write efficient code. The 'P' button allows you to measure the performance of your code. For any given message in TextMsg, the PerformanceTest routine will count how many iterations of DecodeHuff and EncodeHuff your program can execute within 1/18th of a second. The higher the number, the better.
  • You can only get performance points if you complete the Encode, AppendBufferN, DecodeHuff, and EncodeHuff routines.
  • Points
    • 10% or more faster than crippled library code: 5 points
    • As fast as library code: 4 points
    • Within 90% as fast as library code: 3 point
    • Within 50% as fast as library code: 2 point
    • Within 10% as fast as library code: 1 point
    • Incomplete or non-functional routine: 0 points

Preliminary Procedure

• Copy the empty MP2 program (MP2.ASM), huffman encoding table (HUFFCODE.INC), sample input file (test1.in), corresponding output files (test1.out), libraries (libmp2.lib, lib291.lib), and Makefile from the network drive to your home directory with the following command:
  xcopy /s I:\ece291\mp2 F:\mp2
  Alternatively, from home, you can download the same files as mp2.zip.
• As with MP0 and MP1, run NMake to build an executable program using the given ECE291 library functions.
• As with MP0 and MP1, use a text editor to modify the program. As given, the program uses LIBMP2 routines to implement all functionality. To receive full credit for the assignment, you will need to implement each of the subroutines described above with your own code.
• As with MP0 and MP1, use CodeView (CV) to debug and test your program. Because you only receive credit for procedures that function completely as specified, it is best to debug each routine individually.
• By modifying a few comments, you can mix and match usage of your own code and Library routines. You may notice that the LIBMP2 routines contain extraneous and difficult-to-read code. They are not meant to be unassembled!

Final Steps

1. Print a copy of the MP2 grading sheet.
2. Demonstrate MP2.EXE to a TA or to the instructor.
   • Be prepared to answer questions about any aspect of the operation of your program. The TAs will not accept an MP if you cannot fully explain all operations of your code.
   • Your program must work for all possible data patterns. Additional test patterns will be listed on the grade sheet.
   • The TA will determine performance points using a sample message
3. Handin in your program by running:
   A:\Handin YourWindowsLogin
4. Print MP2.ASM
5. Staple the MP2 grading sheet to the front of your MP2.ASM file and give both to the same TA that approved your demonstration.

HUFFCODE.INC (Huffman Encoding Table)

MP2.ASM (Program framework)