Although figure 3 suggests that
the performance of remote memory (parity logging) is significantly better than
the performance of disk, the completion time
of an application even under remote memory
may be unacceptably high.
Hopefully, the performance of remote memory
will be improved as soon as the Ethernet interconnection network is
substituted by a faster one (e.g. FDDI, ATM, FCS, etc.).
To evaluate the performance of the
applications on top of faster networks we
make detailed performance
measurements that separate
the completion time of the application
into the following factors:
(i) user time (utime),
(ii) system time (systime)
(iii) initialization time (inittime)
(iv) page transfer time (ptime).
Using the provided time command we measure the utime, systime, and
elapsed time (etime) for each application. Subtracting the utime
and systime
from the etime for instances of the applications that
perform no paging we calculate the inittime, that is the
time it takes the operating system to load and start executing the application.
The ptime consists of the protocol processing time (pptime) and
the bandwidth dependent blocking time (btime). We measured the
pptime and found it to be equal to 1.6 ms per page for TCP/IP.
We calculate the btime using the formula :
Assuming that a network with X times higher bandwidth will decrease
btime by a factor of X, we can predict the etime of the application
over this high bandwidth network. Thus, the formula used is :
We made all these measurements on our FFT application, and predict its performance on a system with an interconnection network which provides ten times more bandwidth than the Ethernet. We also predict its completion time on a system that has enough memory to hold all the working set of the application (ALL_MEMORY) by adding the utime, systime and inittime. The predicted execution times, along with the measured execution times of DISK and PARITY_LOGGING are plotted in figure 4. We see that ETHERNET*10 performs very close to ALL_MEMORY, and significantly better than both ETHERNET and DISK.
To understand the results shown in figure 4, we analyze the execution time of FFT with 24MBytes of input when PARITY_LOGGING is used. The measured elapsed time is 130.76 seconds, consisting of 66.138 sec of useful user time, 3.133 sec of system time, 0.21 sec of initialization time and 61.279 sec of page transfer time. During the same run, the application suffered 2718 pageouts and 2055 pageins. Since 4 servers were used plus a parity server the number of page transfers was equal to 3397 + 2055 = 5452. Thus the protocol overhead was equal to 5452*0.0016, or about 8.723 sec. The bandwidth dependent blocking time was equal to 61.279 - 8.723, or about 52.556 sec. Using a ten times faster interconnection network, the bandwidth dependent waiting time will be reduced to 5.255 sec. Thus, the total completion time of FFT would be 66.138 + 3.133 + 0.21 + 8.723 + 5.255 sec, or 83.459 sec, divided as follows: 79.246% in user time, 3.754% in system time, 0.252% in initialization time and 16.748% in page transfer time. We see that a 100 Mbit/sec interconnection network reduces the total paging overhead to less than 17% of the total application execution time. We believe that most users would be willing to pay such an overhead in order to run an application that does not fit in main memory. After all, the only other option they have is to suffer from disk thrashing.
Figure: Performance of FFT for various Architecture Alternatives.
DISK is the measured completion time when paging to
a local disk.
ETHERNET is the measured completion time of parity logging to
remote memory on top of the Ethernet.
ETHERNET*10 is the predicted completion time when
using remote memory as a paging device, on top of a network
that provides ten times
more bandwidth than the Ethernet interconnection network.
ALL_MEMORY is the predicted completion time of FFT
when we use the same workstation but with enough memory to
hold its entire working set.