CS535 Final Project

FPX-IDE Interface

(Public Report)

Benjamin West (bmw3@ccrc.wustl.edu)

Qiong (Maggie) Zhang (mzhang@ee.wustl.edu)

Project Webpage: http://students.cec.wustl.edu/~bmw3/hwdb/cs535.shtml

Washington University in St. Louis

12/13/2001

Table of Contents

1) Introduction

2) Background and Motivation

3) Project Delegation

4) Components of the Project

a) Hardware

i) Printed Circuit Board (PCB) for Arbitration b/w IDE and the FPX

ii) PC with Host IDE Controller and Hard Drive

b) Software

5) Results

6) Plans for Further Refinement

7) Conclusion

8) References

Figure 1: Block DIagram Overview of FPX-IDE Inteface2

Figure 2: PCB with Voltage Conversion and Termination4

Figure 3: Components of FPX_IDE Module5

Figure 8: Example of IDE-Extracted Data (and resulting data corruption)7

1) Introduction

Figure 1: Block DIagram Overview of FPX-IDE Inteface

For our final project in CS535, we implemented an AT-Attachment Packet Interface (ATAPI, also referred to as IDE or EIDE) for the Field Programmable Port-Extender (FPX) of the Wash. U. Gigabit Switch (WUGS). The purpose of our interface, at least in the scope of this course, was to capture the contents of data packets that traverse the IDE peripheral interconnection bus, and then to forward that data over the FPX’s network interface to an observer on a remote workstation. To this end we constructed a hardware component to physically attach the IDE bus to the test pins of the Reprogrammable Application Device (RAD) on the FPX, and then a software module to control the RAD. Given the tight schedule for project completion, we placed conservative constraints on our project’s scope and complexity, so that we could successfully build a system that works both in hardware and software. With the CS535 project now complete, we hope to scale this functional base to greater capacity and greater capability after the semester end.

2) Background and Motivation

We undertook this project because of its potential usefulness to our common research project outside of CS535: an off-the-shelf consumer hard drive modified to accept

search queries directly, and then to perform that search on its contents at (or near) the disk rotation speed. From comparing the data rate for the read head flying over a magnetic disk (>1Gbit/s) to data rate of the I/O bus that connects a hard drive to its host (~50MB/s), we predict our device could provide up to two magnitudes’ improvement in search time. Naturally, the process of modifying this drive to suit our research goals would require a high degree of familiarity with the IDE protocol, and of its typical implementation on hard drive interfaces. Unfortunately, such reverse-engineering would involve a massive learning curve, especially because of the limited propriety information which hard drive manufacturers make available to us. So, as a first step, we chose to construct a device that sits on the IDE bus and snoops traffic between the hard drive and the bus controller on the host PC. With the addition of simple, streaming-data processing capabilities to the IDE snooper (i.e. string matching), our passive IDE snooper could by itself offer significant improvement in search time, if one assumes that an IDE bus can accommodate data throughput rates that exceed the host PC’s capability to digest such data.

The custom IDE-bus snooping device we plan to implement (outside of the CS535 final project) will essentially contain an FPGA programmed with an IDE interface, but because of the turn-around time typically required for manufacturing such a prototype board, we chose to use the RAD FPGA on the FPX as a readily available testbed. The general idea is that once we demonstrate successful operation of our IDE snooper on the RAD, we can readily migrate the snooper to our own prototype board.

3) Project Delegation

Ben West

· Designed, implemented, and tested the printed circuit board (PCB) that arbitrates electrical connection between the IDE bus and the RAD test pins.

· Setup the desktop PC (with IDE bus controller) and hard drive as the target system of our bus snooper.

· Modified VHDL code for the IDE Finite State Machine (FSM) and GetCell process to improve software robustness and online troubleshooting capability.

· Performed simulation/verification of all VHDL code (pre-synthesis, post-synthesis, and post-place & route).

Qiong (Maggie) Zhang

· Designed, wrote and tested VHDL code for the IDE finite state machine (i.e. the software component that interprets the IDE protocol).

· Designed, wrote and tested VHDL code for the GetCell module, which accepts data extracted by the IDE state machine, encapsulates that data into ATM cells, and then stores those cells in a FIFO for eventual forwarding onto the FPX’s network interface.

· Designed, wrote and tested VHDL code for the SDRAM-based cell FIFO.

· Performed simulation/verification of all VHDL code (pre-synthesis, post-synthesis, and post-place & route).

4) Components of the Project

a) Hardware

i) Printed Circuit Board (PCB) for Arbitration b/w IDE and the FPX

Figure 2: PCB with Voltage Conversion and Termination

The IDE bus is a multi-drop bus that operates at 5.0V. The RAD on the FPX operates at 3.3V, and has no termination elements on its test pins. Thus, we designed and etched a small PCB to provide both electrical termination and voltage conversion for the physical interface between the RAD and the IDE bus. The composite artwork for this board is shown above in Figure 2, with blue showing the bottom layer (mostly ground plane), red the top layer, and green showing pads and vias. This board is actually the same size as the FPX board, and the remaining space not shown in the figure above was used to support the hard drive attached to the IDE bus. We originally intended for this PCB and hard drive assembly to sit on risers directly on top of the FPX, with 3-4” ribbon cables running from the test connectors on our PCB to those on the FPX. However, the length limitation on the ribbon cable that connected our PCB to the IDE bus controller the PC made this arrangement untenable, and we simply lengthened the ribbon cables between our PCB and the FPX so that our assembly could sit entirely outside of the WUGS chassis.

ii) PC with Host IDE Controller and Hard Drive

The second hardware element to our project was the actual IDE bus to be snooped. This comprised of a desktop PC with an IDE host controller and a IDE hard drive, as illustrated in Figure 1. Both of these pieces were retrieved from spare equipment sitting in the MISC and CCRC labs, and were adapted to serve our project’s purpose. That this, we installed the Linux Operating System (OS) on the PC to allow precise control of both the IDE controller and attached hard drive, and then replaced the hard drive’s original contents with test data. Using the hdparm program, we set this IDE bus to operate specifically in the mode Ultra-DMA (UDMA) 0, the slowest data-burst mode supported by the IDE protocol (120ns cycle time), and then mounted the hard drive to be examined in synchronous mode, to ensure tight correlation between filesystem operations in the OS (reading a file) and actual IDE bus transfers. The buffer cache maintained by the OS proved rather difficult to disable completely, meaning bus transfers didn’t always follow OS filesystem operations exactly, but we found a simple workaround by unmounting and then remounting the test hard drive before each experiment.

b) Software

The software components of our project were all within an FPGA bit image programmed into the RAD of the FPX. As shown in Figure 1 in the Introduction, this FPGA image contained our core FPX_IDE module, a controller for interfacing with SDRAM (provided in CS535), and a loopback module (also provided in CS535). The purpose of the loopback module was to route network traffic back to it its sender, allowing us to capture the FPX_IDE module’s output traffic, which would otherwise proceed through the Network Interface Device (NID) to the WUGS linecard.

Figure 3: Components of FPX_IDE Module

The FPX_IDE module shown in detail above in Figure 3 was the core of our project’s software component, and it consisted of the following parts:

· 7x64bit InCellData buffer. Stored each ATM cell that arrives from the NID, until the CellFIFO could write that cell into SDRAM. This buffer was 64bits wide to fit the data path of the SDRAM. Taken from CS535 Machine Problem 5 (MP5) unmodified.

· 14x32bit IDECellData buffer. Collected into ATM cell-sized portions the IDE-extracted data outputted by the IDE finite state machine.

· GetCell process. Managed the buffering of incoming ATM cells, and emptied contents of IDECellData into InCellData when enough IDE data to fill an ATM cell had been captured. Taken from MP5 and modified.

· IDE Finite State Machine (FSM). When enabled by the CellFIFO, the IDE FSM monitored the IDE control signals, looking for the start of an Ultra-DMA data burst from the hard drive. When it recognized such a burst, it copied each 16bit IDE data value that it saw into the 32bit IDECellData buffer (with each 32bit value latched by the signal Data_IDE_en).

· PushCell process. When InCellData filled up, meaning a complete ATM cell was present, this process requested the buffer contents be written into the CellFIFO (i.e. in the SDRAM). Taken unmodified from MP5.

· CellFIFO module. Maintained the read/write pointers in the SDRAM-based FIFO, and translated write cell/read cell requests from the PushCell/PullCell processes into requests understandable by the SDRAM controller. Since the SDRAM couldn’t always accept a read/write request immediately (because of its refresh cycle), the CellFIFO managed cell flow control between the PushCell and PullCell processes, ensuring that each request from those two processes was answered in a fair and ordered manner. The CellFIFO also interpreted NCHARGE-format control cells sent by the user, and controlled/queried the IDE FSM accordingly. Taken from MP5 and modified.

· PullCell process. Emptied cells from the cell FIFO into the outgoing buffer OutCellData at regular intervals. Taken unmodified from MP5.

· SendCell process. Copied the ATM cell stored in OutCellData out back to the NID. Taken unmodified from MP5.

5) Results

As stated above in the Introduction, our general strategy for better ensuring that our project worked, was to keep our goals and project scope simple. This has worked out to our benefit, because our final result is operational (although still a very rough prototype in the context of our research outside of CS535). The FPGA bit image was downloaded and successfully started on the RAD, and we were able to send NCHARGE control cells and receive proper responses. The image was rated by the compiler for operation at 88MHz, although the interface with the off-chip SDRAM placed an upper limit of about 70MHz. The actual hardware demonstration was performed with a 62MHz clock, based on equipment available in the lab at the time, which in any case was still significantly greater than the expected 8MHz speed of the IDE bus. Once our custom PCB was attached to the FPX and the IDE FSM initialized, we began experimenting with reading small ASCII files off the test hard drive. We successfully captured ATM cells outputted by the FPX_IDE, whose contents correlated quite closely to the file contents retrieved from the hard drive.

However, the extracted IDE data didn’t match their source exactly, as it became quite apparent our FPX_IDE module was missing some of the 16bit data values that traversed the bus. Figure 4 below illustrates this quite clearly, showing how a file containing the alphabet (upper case ASCII characters) was only retrieved in portions through our IDE bus snooper.

Original File	ATM Cell (Hex)	ATM Cell (ASCII)
ABCDEFGHIJKLMNOPQRSTUVWXYZ	00014091 00000910 41424546 47484c4d 50515354 575a1b5b 337e0000 00000000 00000000 00000000 00000000 00000000 00000000 00000000	A B E F G H L M P Q S T W Z [ 3 ~

Figure 4: Example of IDE-Extracted Data (and resulting data corruption)

Investigation into potential causes for this data corruption lead us to doubt the electrical signal quality over the ribbon cables extending from our PCB to the FPX. Random signals which we sampled with an oscilloscope did indeed exhibit rather severe problems with crosstalk, and with reflections from impedance mismatch (on the order 30% VCC). While this didn’t conclusively explain the skipped IDE data values, it lead us to believe the FPX receiver that sampled the DSTROBE signal off the IDE bus was likely not seeing clean, unambiguous voltage swings. Strangely enough, this presumed interference on DSTROBE did not lead to duplicate data values extracted from the IDE bus (i.e. the same value sampled twice), only skipped values.

6) Plans for Further Refinement

Certainly, solving the problem with corrupted data being sampled off the IDE bus is the first of many fixes and refinements we hope to implement on our prototype. More robust electrical termination on the affected signals paths is planned, along with using ribbon cables with better grounding and dielectric characteristics. Once these signaling issues are resolved, we anticipate our device will be able to sample an IDE bus running at speeds up to 33Mhz, or at half the RAD clock frequency, a 4-fold increase over its current speed.

Another avenue for potential improvement is to filter the data sampled off the IDE bus, stripping away headers and framing elements left by the filesystem, by the hard drive itself, etc. This is unfortunately not trivial, as the hard drive transfers data over the IDE bus only in the form of 4KB blocks, and not as individual, variable-length files. Just from observing the contents of the ATM cells generated by our FPX_IDE module, the separation between file data and the rest of the 4K block was not apparent, unless the observer knew the contents of the file in advance. Thus, such file-system recognition might possibly require that our bus-snooper have access to the File Allocation Table (FAT) or I-Node Table of the filesystem that it is sampling from.

7) Conclusion

While our project developed some non-trivial hardware problems that sharply limited the usefulness of what data we did collect, we believe we met our original goal of finding insight into the workings of the IDE interconnection bus, into its implementation in hard drives, and into the format in which data traverses the bus. The working prototype we’ve built for this project will go on to feature prominently in our research, as we incrementally refine the device for greater capacity. Certainly, this project could have taken on a far greater degree of sophistication, had it not been bound by the physical hardware component, but it was precisely that physical interface between the FPX and an IDE bus which defined this project from Day One.

We can both safely say this project has helped us clear the initial hurdles of building the real prototype of our self-searching hard drive.

8) References

ATA/ATAPI 5 Protocol Specification (draft revision 3) http://www.t13.org/project/d1321r3.pdf

Braun, Florian, John Lockwood, and Marcel Waldvogel. Layered Protocol Wrappers for Internet Packet Processing in Reconfigurable Hardware, Technical Report WUCS-01-10, Washington University Department of Computer Science, July 2001. http://www.arl.wustl.edu/arl/projects/fpx/wrappers/wucs-01-10.pdf

Dharmapurikar, Sarang and John Lockwood. Synthesizable Design of a Multi-Module Memory Controller, Technical Report WUCS-01-26, Washington University, Department of Computer Science, October, 2001. http://www.cs.wustl.edu/cs/techreports/2001/wucs-01-26.pdf

FPX Webpage. http://www.arl.wustl.edu/arl/projects/fpx/.

Lockwood, John W., Jon S. Turner, and David E. Taylor. “Field Programmable Port Extender (FPX) for Distributed Routing and Queuing”, ACM International Symposium on Field Programmable Gate Arrays (FPGA'2000), Monterey, CA, February 2000, pages 137-144.