Reviewer: Michela Becchi
Date: 12-1-2005
How would you rate this paper, relative to others we have read? bottom 50%
How would you rate your kowledge of the topic of this paper? novice
What problem or issue does the paper address? Why is it important?
The paper presents an experimental evaluation of three approaches to build an IP storage area network. The approaches differ in the level of hardware support used: (1) the software approach maintains all the functionality in the host system, (2) the TOE approach offloads the TCP/IP functionalities to the network adapter, (3) the HBA approach moves both the TCP/IP and the storage protocol processing to the network adapter.
The significance of the study consists in highlighting the limitations of current hardware based solutions compared to software ones and giving guidance for improving hardware based systems.
What are the main contributions of the paper and why are they important?
As mentioned, the paper does not invent anything new, but provides an evaluation of existing technologies. As such, it should be evaluated more as an experimental study than as a research paper. Its based contribution consists in the systematic analysis and in pointing out the limitations of hardware approaches, especially on small block sizes.
How significant are these contributions relative to previous work?
Franking speaking I am not aware about previous work focusing on this kind of evaluation. I assume that the reported analysis is novel.
Give detailed comments justifying your view of the paper.
The paper presents an experimental study and evaluation of three approaches (one software and two hardware based solutions) for building IP storage area networks. The analysis consists of two parts. First, synthetic “micro-benchmarks” are used in order to evaluate the sensitivity of the three solutions to varying block size, CPU, memory and I/O bus speed. Second, the same approaches are compared on “macro-benchmarks” consisting of real world applications.
The based conclusions carried out by the authors are the followings. (1) Software solutions are more performing than hardware ones, especially on small block sizes (where the per-byte cost is less). (2) Hardware solutions offer a better CPU utilization-to-throughput ratio especially for large block size. Since none of the configurations analyzed reaches a full CPU utilization even with the software approach, this benefit does not show to have a great exploitation. (3) All the three solutions are sensible to I/O bus speed and less to CPU and memory speed (which do not currently constitute a bottleneck). (4) The basic cause of performance bottleneck in current hardware solution is the disparity between host and adapter.
In my opinion this paper is very easy to read and to follow. As mentioned, it presents an experimental study, and as such it should be evaluated. Even if the experiments and their analysis sound, I cannot say that at the end I feel convinced by the conclusions. In particular, I have one basic remark and few little observations.
The strongest point in the paper is highlighting how the hardware solutions are in most cases less suitable than the software ones. This is motivated through the mismatch between host system and adapter speed. Now, before investing in ASIC technology for hardware offload, I would like to see more evidence of this conclusion. The CPU speed sensitivity study has been carried out for frequency ranges which do not allow proving this concept. My first impression in that, being this an important point in the paper, the discussion would have been more convincing if more evidence on that point had been provided.
A second observation is that the analysis does no take into consideration scenarios where the software solution would show more limitations. This is anyway partially corrected by the fact that those scenarios (large number of initiators, application with high communication overhead) are mentioned in the conclusions.
Finally, there are a couple of points where I have been not convinced by the data or discussion. In pg. 239 it is mentioned that PCI bus is an important performance factor especially for the software approach; while it seems me from Fig. 4 that it affects more TOE solution. I found surprising on Fig. 240 how the CPU utilization is similar across the solutions. An explanation I gave (which is confirmed in the conclusion section) is that, for that benchmark, the communication task does not take much of the CPU utilization. If this is true across all the benchmark, this seems to me a better and concrete motivation for software solutions rather than hardware (in other words: why caring about offloading a task which does not take much CPU time?). However, this aspect is not discussed. Finally, on the second column on page 240, it is told that the software approach starts showing superiority for larger file size. This seems to me in contrast with what seen in the micro-benchmarks analysis, but there is no real discussion on that.
In conclusion, I found the topic of the paper interesting, and I for sure see some value in the analysis. However, at the end I don’t feel really convinced by the conclusions, and there are a couple of points which seem to me not so consistent. I would rank this as average paper. No being particularly convinced by it, bottom half seems to me more appropriate than top half.