Reviewer: Mike Wilson
Date: 12-1-2005
How would you rate this paper, relative to others we have read? top 50%, but not top 25%
How would you rate your kowledge of the topic of this paper? novice
What problem or issue does the paper address? Why is it important?
Storage is becoming more and more distributed, and current storage technologies attach to hosts via network protocols (TCP/IP). Research in this arena has focused on speeding up the network connection by offloading protocol processing to dedicated hardware, freeing up the CPU for other work. However, this direction for research is founded on the assumption that this will speed up processing. The authors question this assumption and devise experiments to test it directly.
The importance lies in redirecting research into better directions for solving these problems. If the underlying assumptions about inefficiencies are wrong, then current research is largely a waste of time and should be redirected.
What are the main contributions of the paper and why are they important?
The authors succeed in comparing three different technologies: TCP offloading, Host Bus Adaptors, and pure software. The performance analysis is directly to the point of selecting research paths for SAN technology. While no new solutions are contributed, this is less important than ensuring we're trying to solve the right problems.
The results point out that TCP offloading functions best at large block sizes, where per-byte overhead dominates, and worst at small block sizes where per-operation overhead dominates. However, in no case did hardware solutions of any kind perform as well as software solutions. In this experiment, the hardware solutions never even reached line speed.
While the software approaches do take a great deal of CPU time from other processing, the throughput is superior to anything the hardware approaches can provide. The disparity is judged to be due to the slower processors used in the hardware approaches. Additional effort should be directed into ASIC solutions for protocol offloading, and offload processors must be increased in speed (ad therefore cost) to stay competitive with software solutions.
How significant are these contributions relative to previous work?
Prior to this study, most performance investigations were focused on general-purpose protocol use, not storage area networks. More closely related work on storage over IP has looked at wide-area networks - high latency transfers. This papers looks specifically at SAN technology in the usual environment, high bandwidth with low latency. The studies are complementary, not redundant.
Give detailed comments justifying your view of the paper.
Given the goals, this paper must be judged primarily on whether it succeeds in analyzing performance correctly for these three technologies. An analysis of their experimental setup is in order.
It should be noted that, in this analysis, I rely on the fact that the software generally performed better than the hardware. That is, experimental variations that would serve to penalize the software tests or advance the hardware tests do not call the results into question, only the magnitude of the difference.
The experimental setup used a single client machine of moderate performance, running a stock Windows 2000, one of the most common operating systems in use today. The comparisons were made by swapping different cards in this same machine. The authors also compared multiple cards within each category and found that they were all within a 15% performance range, so while they used the best cards for the study, they properly verified that the card was not atypical. The server was a lower grade machine running linux. Care was taken to avoid bus contention on the datapath from the network to the disks, so the data is as unpolluted as possible. However, the server only ran the software solution, not a hardware test. The authors assert that their results show that this was irrelevant. I would prefer to see a sampling of the CPU utilization of the server side to verify that the experiment never became CPU bound at the server. The authors note that they measured these values, but only provide utilization for the initiator. The results on the initiator allow an inference that the server side utilization should be around 76%, but it would be better to show it. (The authors do assert that the server CPU was never saturated at any time.)
The actual tests relied on reading and re-reading the same locations repeatedly. The authors sought to remove any RAID performance differences from the experiment by relying on cached data, effectively emulating much faster disk speeds. As noted, writes were found to perform the same as reads, so were not reported.
The authors showed by analysis that the PCI bus and memory latency could not account for the results, since the throughput was always well below PCI bus limits. From this, the authors conclude that the bottleneck is in the offload hardware, not in some other portion of the data path.
One point of reduced value is that the software approach did not provide zero-copy support. This does imply that the comparisons between software and the hardware solutions place the software at a slightly unfair disadvantage. Since this penalizes the software, the results are not in question (as explained above).
At larger block sizes, the authors discovered that neither the CPU nor the network were the limiting factor, and performed additional investigations to determine the culprit. While this is outside the original scope of investigation, it is an urgent necessity: only by finding all of the relevant variables can future research be properly targetted.
A larger problem is that while the block sizes were varied, the network speed was a constant. The argument could be made that, since the network was never saturated by any of the solutions, this is irrelevant. However, in the high-performance arena, with higher speed offload engines and CPUs, it is suggested that software solutions will no longer be able to keep up with network speeds, and that hardware solutions would be superior. This is something that should have been researched, ideally by charting results at 10, 100 mbps, and 1, 10 gbps.
While there is no discussion of future work, this is not really a drawback for a paper that attempts to provide a sound basis for future research. The paper isn't about solving a problem: it is about defining that problem correctly.