\documentclass{article} % For LaTeX2e
\usepackage{nips10submit_e,times}
\usepackage{amsfonts,url, amsmath}
\usepackage{hyperref}
\hypersetup{
    colorlinks,%
    citecolor=black,%
    filecolor=black,%
    linkcolor=black,%
    urlcolor=blue
}

\newcommand{\Rb}[1]{\ensuremath{\mathbb{R}^{#1}}}

\title{K-means on Azure\\ Supplementary material}

\author{
Matthieu Durut \\
Lokad\\
70, rue Lemercier, 75018 Paris -- France\\
\texttt{matthieu.durut@lokad.com}\\
\And
Fabrice Rossi\\
BILab, T\'el\'ecom ParisTech\\
LTCI - UMR CNRS 5141 \\
46, rue Barrault, 75013 Paris -- France\\
\texttt{fabrice.rossi@telecom-paristech.fr}\\
}

\nipsfinalcopy
\begin{document}
\maketitle

\section{Azure base performances}

One can find some Azure performance benchmarks on \textit{AzureScope}\footnote{\url{http://azurescope.cloudapp.net/}}.
Azure performance largely depends on our algorithm, our implementation and the framework (Lokad-cloud) we use.
In order to use our model as a predictive tool to determine the optimal number of workers $P^{*}$ and the expected 
speedup, we estimated Azure performances by running some isolated jobs and recording time.

\subsection{Read bandwidth}

We pushed 8 blobs of 8 MB into the storage (this is the size of map results we are going to load back in reducers), and
we measured the time spent to retrieve them. For a single worker using a single thread, we retrieved the blobs in
7.79 seconds, implying a 8.21 MB/sec read bandwidth. We tried to use multiple threads on a single worker as
advised by AzureScope to speedup the download process, and we found the best read bandwidth was obtained using
5 threads: we retrieved the 8 blobs in 6,13 seconds, implying a 10.44 MB/sec read bandwidth. The multithreads
read bandwidth we observed differs from what is achieved in AzureScope. This may be explained by two potential factors:
we observed average bandwidth whereas 
AzureScope observed peak performances and we did not have the same experiment environment. Especially, all the parallel
read requests were run on a single container instead of being distributed on several BlobStorage containers.

This benchmark is very optimistic as opposed to our clustering experiments: the bandwidth we estimated was recorded while 
no other workers were reading or writing into the storage. Other parallel reads and writes can have two effects 
on the read bandwidth performance. Firstly, they can limit the bandwidth because of aggregated bandwidth boundaries 
(AzureScope is reporting 800 MB/sec bandwidth for reading operations). Secondly, they can increase the latency between 
a write completion in the blob storage and the availability of the blob to other workers. When running experiments 
with a large number of workers reading and writing in parallel, we sometimes experienced blobs already pushed into the storage 
becoming available only after 1 or 2 minutes. To use our model as a predictive tool to determine $P^{*}$, we used a value 
of bandwidth equals to 8 MB/sec, which is a very conservative assumption that leads us to underestimate communication costs.

\subsection{Workers Flops performances}

All our experiments were run on Azure small VM that are guaranteed to run on 1.6 GHz CPU. Yet because
of virtualization we do not have any warranty in term of floating point operations speed. Therefore, to fit
our predictive speedup model, we ran some map jobs (where the number of floating point operations is known)
to determine how fast could our algorithm be run on these VM. The code was first run on an Intel Core
2 Duo T7250 2*2GHz using only one core. We estimated our map jobs to be run at approximatively 750 MFlops
on average. We then ran some of our map jobs in Azure and we get for the same code performance of 669 MFlops on 
the small VM.\\

It is worth mentioning that with the number of workers we used (at max 160 mappers), we observed very little differences
in processing time between workers. Flops performance may slightly move with time, but at a given time,
workers performed at very similar speed.

\section{Benchmarks}

\subsection{Speedup}

The next table shows speedups with varying values of P. All the runs listed below were using
N = 500000, K = 1000, D = 1000 and I = 10. Theoretically, K and D should have the same impact on map
and reduce phases since map and reduce costs are supposed to be proportional of KD. Yet, since our
model does not take into account latency, having very small values of K and D would lead to underestimate
communication cost by neglecting latency. We have set K and D to be much smaller than N, but also big
enough to neglect latency. 

The following table reports the total running time in seconds (including data loading) of the proposed
implementation for different numbers of mapping PU (P). We report the speedup over the
theoretical total running time, the efficiency (speedup divided by the total number of
PU : $P$ mappers and $\sqrt{P} + 1$ reducers) and the theoretical efficiency predicted by the model.

\begin{center}\scriptsize
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
P &10&50&60&70&80&90&95&100&110&120&130&140&150&160\\
\hline
Wall Time & 2223 & 657 & 574 & 551 & 560 & 525& 521 & 539 & 544 & 544 & 574 & 603 & 605 & 674\\
\hline
SpeedUp & 10.0 & 34.1 & 39.0 & 40.6 & 40.0 & 42.6 & 43.0 & 41.5 & 41.2 & 41.2 & 39.0 & 37.1 & 37.0 & 33.2\\
\hline
Efficiency & 0.67 & 0.58 & 0.57 & 0.51 & 0.44 & 0.42 & 0.41 & 0.37 & 0.34 & 0.31 & 0.27 & 0.24 & 0.23 & 0.19\\
\hline
Theo Efficiency & 0.63 & 0.61 & 0.60 & 0.55 & 0.53 & 0.49 & 0.48 & 0.47 & 0.43 & 0.41 & 0.37 & 0.36 & 0.33 & 0.32\\

\end{tabular}
\end{center}

As expected, the total processing time is minimal for a specific value of P (here 95), for which
one third of the total time is spent in the map phase (as predicted by the model). One can notice that 
the speedup curve is relatively flat in a neighborhood of $P^{*}$. This value of $P^{*}$ is lower
than the predicted value (170 for the chosen constants) as a consequence of the overestimation of
the storage efficiency by the theoretical model. As explained above, our model made the assumption 
that bandwidth is scalable with the number of workers, and this is no more true when the number of workers 
is too high. For example, when running 160 workers, some workers spend one minute to load the dataset from 
the storage, and for 2 experiments we ran with more than 100 workers, one blob took more than 2 minutes to 
be retrieved by a reducer after being pushed by a mapper.

\subsection{Scaleup}

Since in our model $P^{*}$ is not proportional to $N$ anymore (as this was the case using MPI's broadcast), we cannot hope to
achieve linear scaleup. Therefore, the scaleup challenge turns into minimizing growth of wall time as N grows, using 
$P^{*}(N)$ mappers. Theoretically, as our model gives a processing cost proportional to $\frac{N}{P}$ and a communication
cost proportional to $\sqrt{P}$, for $ P = P^{*} =(3NF/B)^{2/3}$ (where $B$ is the time needed
to read a double from the BlobStorage and $F$ is the time needed to perform
one floating point operation), we should get a global cost proportional to $N^{\frac{1}{3}}$. 
Therefore, if N is expected to be multiplied by 8, wall time of the algorithm running $P^{*}(N)$ mappers should be multiplied by 2. 
We estimated by running experiments for some given values of $N$ the best value $P^{*}(N)$ and we observed the corresponding amount 
of wall time. 
Values of K and D (K=1000 and D=1000) were kept constant for all the experiments to ease comparisons.

\begin{center}\scriptsize
\begin{tabular}{c|c|c|c|c|c|c}
 & N & $P^{*}$ & Wall Time & Sequential theoretic time & Effective Speedup & Estimated Speedup ($=\frac{P^{*}}{3}$)\\
\hline
Run 1 & 62500 & 27 & 264 & 2798 & 10.6 & 9\\
\hline
Run 2 & 125000 & 45 & 306 & 5597 & 18.29 & 15\\
\hline
Run 3 & 250000 & 78 & 384 & 11194 & 29.15 & 26\\
\hline
Run 4 & 500000 & 95 & 521 & 22388 & 43.0 & 31.6\\

\end{tabular}
\end{center}

One can see that $P^{*}(N)$ does not grow as fast as $N$. For all the experiments, speedup achieved 
with $P^{*}$ is approximatively  $\frac{P^{*}}{3}$, and reducing step took about twice the 
time spent in map step for this number of mappers, as predicted by our cost model. While the size 
of the dataset is multiplied by 8, wall time is exactly multiplied by 2, as predicted by the model.

\subsection{Price}

Without any commitment and package, 1 hour of CPU on small VM is charged 0.1 dollar and 1000000 
transactions between workers and storage (QueueStorage or BlobStorage) are charged 1 dollar.
Therefore, one clustering experiment with 10 workers (decreasing wall time from 6 hours to 37 minutes) 
is charged less than 2 dollars, and our biggest experiment running 175 workers is charged less than    
20 dollars. Since our experiments are not one hour long but 10 minutes long at worst, if one can recycle 
the 50 other minutes running other computations, then the cost of our biggest experiment drops to 4 dollars
on average. For bigger uses, some packages are available to decrease charges. 
%\section{Conclusions}

%We overestimate the bandwidth.
%As cloud provides an on-demand scale-up our down, one can seek to determine optimal number of mappers $P^{*}$ 
%Our modelisation has a predictive utility in estimating $P^{*}$, but overestimates $P^{*}$ by underestimating communication costs.
%For all the experiments, while running $P^{*}$ mappers, reduce step took twice the time spent in map step.
%Communications in cloud are very important and should therefore not be neglected as on DMM architectures.
%Flops are perfectly scaling as the number of workers grows.
%Communication is not scaling as well. It may be because of aggregated bandwidth boundaries or because 
%blobs are sometimes not instantly available when running multiple parallel push blobs and pull blobs.   
%Throughout, Azure provides a satisfactory scaling up to 200 workers.
%Cost very small
%As the speedup curve is relatively flat in the neighborhood of $P^{*}$, one could choose to run the 
%algorithm with half mappers as $P^{*}$ to save up resources.


%\section{Future works}

%Better modelisation of communication costs while all reducers are reading.
%Try to store blobs in different BlobStorage containers. 
%Implement direct communication between workers.


\end{document}
