Although figure 3 suggests that
the performance of REMOTE_MEMORY is significantly better than
the performance of DISK, the completion time
of an application even under REMOTE_MEMORY
may be unacceptably high.
Hopefully, the performance of REMOTE_MEMORY
will be improved as soon as the Ethernet interconnection network is
substituted with a faster one (e.g. FDDI, ATM, FCS, etc.).
To evaluate the performance of the
applications on top of faster networks, or faster disks we
make detailed performance
measurements that separate
the completion time of the application
into three factors:
(i) bandwidth-dependent blocking time,
(ii) useful user time, and
(iii) protocol-dependent systems overhead.
Using the provided time command we measure the elapsed time
for each application which is the sum of factors (i)-(iii). The
same command also provides the user-time (factor (ii)), and the
system time (factor (iii)).
If from the elapsed time, we subtract
the user plus the system time, we get the time the system was idle
waiting for pages to go through the interconnection network (factor (i)).
By dividing this idle time
with the number of page ins plus page outs, we get the average time
the application waits for each page to go through
the interconnection network. Assuming that an X
times faster interconnection network will reduce this waiting time by
a factor of X, we can predict the
completion time of the application on the faster network 
by adding the measures user and system times, with the
predicted blocking time.
We made all these measurements on our FFT application, and predict its performance on a system with an interconnection network which is two and ten times as fast as the Ethernet. We also predict its completion time on a system with twice as fast disk ( DISK*2), and on a system that has enough memory to hold all the working set of the application ( ALL_MAIN_MEMORY). The predicted execution times, along with the measured execution times of DISK and REMOTE_MEMORY are plotted in figure 4. We see that ETHERNET*10 performs very close to ALL_MAIN_MEMORY, and significantly better than both REMOTE_MEMORY and DISK.
To understand the results shown in figure 4,
we analyze the execution time of FFT
with 28Mbytes of input. The measured elapsed time is 208 seconds,
consisting of
78.5 sec of useful user time,
5 sec of system time,
and 124 sec of network blocking time,
spent waiting for pages to go through the Ethernet.
During the same run, the application suffered
6520 page-outs and 7791 page-ins.
The average waiting time for a page transfer
(both for page ins and page outs) on top of the Ethernet
is 
, or about 8.6 ms. Using a ten times
faster interconnection network, the average waiting time will be reduced
at least to 0.86 ms. Thus, the total completion time of FFT would be
at most 
sec,
divided as follows: 82% in user time, 5% in system time, and
13% in network blocking time.
We see that a 100 Mbit/sec interconnection network reduces the total
paging overhead to a mere 17% of the total applications execution time.
We believe that most users would be willing to pay such an overhead
in order to run an application that does not fit in main memory.
Figure: Performance of FFT for various Architecture Alternatives.
DISK is the measured completion time when paging to
a local disk.
REMOTE_MEMORY is the measured completion time when paging to
remote memory on top of the Ethernet.
ETHERNET*2 and ETHERNET*10 is the predicted completion time when
using remote memory as a paging device, on top of a network
that is twice and ten times
as fast as the Ethernet interconnection network.
DISK*2 is the predicted completion time when using
a twice as fast disk for paging.
ALL_MAIN_MEMORY is the predicted completion time of FFT
when we use the same workstation but with enough memory to
hold its entire working set.