\section{Introduction to clustering and distributed Batch K-Means}

Clustering is the task of unsupervised classification that assigns a set of objects into groups so that each group is composed of similar objects. Non-hierarchical clustering has been widely used for more than half a century as a summarizing technique to build a simplified data representation. It is one of the primary tools of unsupervised learning. It plays an outstanding role in several pattern analyses, in decision making, proximity exploration or machine-learning situations including text mining, pattern or speech recognition, web analysis and recommendation, marketing, computational biology, etc.\\

Two main clustering questions have been addressed in many contexts, reflecting the clustering's broad utility. The first question deals with the notion of similarity between objects. In general, pattern proximity is based on some similarity measure defined for any pair of data elements. Such a similarity measure is of prime interest since it totally defines the proximity notion and therefore needs to reflect what the user considers as close or similar. When the data set is contained in a vectorial space, a simple euclidean distance is often used. In many other situations, a more specific similarity measure is used, such as in the context of DNA microarrays exploration. In other cases, clustering output is used as the training input of a set of supervised learning functions. This latter case, often referred to as clusterwise linear regression (see for example \cite{Hennig99}), is of prime interest for Lokad since the time series clustering is used to produce groups of time series that are processed group by group. From a computation point of view, the complexity of the similarity measure has a significant impact on the computational cost of clustering.\\

The second clustering aspect that has been widely addressed is the algorithm choice. For a given similarity measure, one can define a performance criterion (often referred to in the literature as a loss function) for a clustering result to quantify how close to its assigned cluster each point is. The problem of returning the optimal clustering solution in regard to this loss function is known to be in many cases computationally untractable (see for example \cite{KMeansIsNPhard} in the case of the K-Means). Many algorithms are devised to return approximations of the optimal solution, often by converging to a local minimum of the loss function. We refer the reader to \cite{DataClusteringReview} for an in-depth clustering algorithm review. Some well-known techniques of non-hierarchical clustering are Batch K-Means, Vector Quantization (VQ), Neural Gas, Kohonen Maps, etc.\\

This chapter focuses on the Batch K-Means clustering algorithm. The choice of Batch K-Means has been motivated by several reasons. Firstly, while better clustering methods are available, Batch K-Means remains a useful tool, especially in the context of data summarization where a very large data set is reduced to a smaller set of prototypes. As such, Batch K-Means is at least a standard pre-processing tool. Secondly, Batch K-Means has a low processing cost, proportional to the data size and the number of clusters. Thus, it is a good candidate for processing very large data sets. Finally, apart from the number of iterations to convergence, the processing time of Batch K-Means depends only on the data dimensions and on K, rather than on the actual data values: timing results obtained on simulated data apply to any data set with the same dimensions.\\

Distributed computing in machine-learning or data-mining arises when the computation time to run some sequential algorithm is too long or when the data volume is too big to fit into the memory of a single computing device. Such algorithms have already been successfully investigated for fifteen years (see for example \cite{ParallelMiningAssociationRules}, \cite{DhillonModha2000} or \cite{ScalableParallelClassifier}) and applications built on top of these distributed algorithms are presently used in a wide range of areas, including scientific computing or simulations, web indexing applications such as Google Search, social network exploration applications such as Facebook, sparse regression in computer vision, etc. Parallelization is today one of the most promising ways to harness greater computing resources, whereas the volume of data sets keeps growing at a much faster rate than the sequential processing power. In addition, most of recent CPU are now multi-cores, implying parallelization to benefit from these supplementary resources.\\

In this chapter, we investigate the parallelization of Batch K-Means over different computing platforms. Batch K-Means is known to be easy to parallelize on shared memory computers and on local clusters of workstations: numerous publications (see e.g. \cite{DhillonModha2000}) report linear speedup up to at least 16 processing units (which can be CPU cores or workstations). There are two reasons why the Batch K-Means algorithm is suited for parallelization. The first reason is that distributed Batch K-Means produces exactly the same result as sequential Batch K-Means. For algorithms where the sequential and the distributed versions produce different results, it is necessary to confront the two algorithm versions on both speedup and accuracy criteria. In the case of Batch K-Means, the comparison of the sequential and distributed versions is limited to the speedup criterion. In addition, the exact matching of the results of the two algorithm version provides an easy mechanism to guarantee that the distributed algorithm version is correctly implemented. From an engineering point of view, this property makes the development process much easier.\\

The second reason why distributed K-Means is easy to parallelize is that it has been claimed to be an embarrassingly parallel algorithm (we refer the reader to Subsection \ref{sec_MapreduceConsiderations} for a definition of embarrassingly parallel problems). Most of the computation time is spent on evaluating distance between data points and prototypes. These evaluations can easily be distributed on multiple processing units. Distributed K-Means can then be viewed as an iterated MapReduce algorithm: firstly, the data set is split into $M$ subsets of equal size (where $M$ is the number of processing units), each processor (mapper) being responsible for computing distances between its data points and the prototypes. In the end, all the processors compute a version of local prototypes, then forward this version to a unique processor responsible for gathering all the versions of local prototypes versions, computing the prototypes' shared version (reduce step) and sending it back to all the units. Forwarding the versions of local prototypes can be done in several ways, depending on the hardware/software framework: broadcasting can be done using MPI (Message Passing Interface) implementation or web services for example.\\

Batch K-Means has already been successfully parallelized on DMM architectures using MPI (see for example \cite{DhillonModha2000} or \cite{KMeansJoshi}). This algorithm has also already been implemented on shared-nothing platforms, for example using Hadoop, but to our knowledge no theoretical study of the behavior of this algorithm on such a platform has been done yet. This chapter therefore investigates parallelization techniques of Batch K-Means over a platform of Cloud Computing (in the present case Azure). The main difficulty of the cloud algorithm consists in implementing synchronization and communication between the processing units, using the facilities provided by Windows Azure cloud operating system. We detail this technical challenge, provide theoretical analyses of speedup that can be achieved on such a platform and the corresponding experimental results.\\

Let us now briefly outline the chapter. In Section \ref{Sequential K-Means Algorithm Section}, we present the sequential Batch K-Means algorithm and its computational cost. In Section \ref{Distributed K-Means Algorithm Section}, we describe how the Batch K-Means is often distributed on several processors and examine the new corresponding computational cost. We also build some model of the real cost of a distributed Batch K-Means on DMM and develop a bandwidth condition inequality. Section \ref{sec_KMeans_Azure} is devoted to some specificities of the cloud that prevent previous developments of DMM Batch K-Means cost from being applied on the cloud. We provide a new parallelization implementation to adapt the distributed Batch K-Means to the cloud specificities. Section \ref{sec_Cloud_KMeans_Results} presents the experimental results of our distributed Batch K-Means on the cloud.

\section{Sequential K-Means}
\label{Sequential K-Means Algorithm Section}

\subsection{Batch K-Means algorithm}

%Suppose we are given a dataset of n points in $\Re^{d}$, $(x_{i})_{i=1..n}$ . We want to construct K points $(c_{k})_{k=1..K}$ also in $\Re^{d}$ to minimize the objective function :
%\begin{equation}
%\sum_{i=1}^{n} \displaystyle \min_{k=1..K} (||x_{i}-c_{k}||_{2}^{2})
%\end{equation}

Let us consider the following problem: given a data set of $N$ points $\{\zbf_{t}\}_{t=1}^N$ of a $d$ dimensional space, we want to construct $K$ points $\{w_{k}\}_{k=1}^{K}$, referred to in the following as prototypes or centroids, as a summary of the data set using the euclidean distance as a similarity measure. Through this similarity measure, one can define the empirical distortion:

\begin{equation*}
\label{eq:EmpiricalDistortion}
C_{N}(w) = \sum_{t=1}^N{\min_{\ell = 1, \ldots, K} \left\|\zbf_t  - w_\ell\right\|^2}, \quad \mbox{$w \in \left(\real^d\right)^K$}.
\end{equation*}

$C_{N}$ is guaranteed to have at least one minimizer, as it is both continuous and coercive. As already stated, the computation of an exact minimizer is often untractable, and this problem is proved to be NP-Hard, even in the easier case of $d=2$ (see \cite{KMeansIsNPhard}). Among the many algorithms that compute an approximation of the optimal minimizer, the Batch K-Means is a well known algorithm, widely used because it is easy to implement and provides overall satisfactory results. \\

The Batch K-Means algorithm belongs to the class of alternating optimization algorithms. Indeed, it alternates two optimization phases iteratively. The first phase, called the assignment phase, takes as input a given set of prototypes, and assigns each point in the data set to its nearest prototype. The second phase, referred to as the recalculation phase, is run after the assignment phase has been completed. During this second phase, each prototype is recomputed as the average of all the points in the data set that has been assigned to him. Once the second phase has been completed, phase 1 is run again, and phase 1 and 2 are run iteratively until a stopping criterion is met. Algorithm \ref{algo_SequentialKMeans} logical code describes the alternation of the two phases.\\

\begin{algorithm}
    \caption{Sequential Batch K-Means}
    \begin{algorithmic}
    \label{algo_SequentialKMeans}
        \STATE Select $K$ initial prototypes $(w_{k})_{k=1}^{K}$
        \REPEAT
        \FOR{$t = 1$ to $N$}
        \FOR{$k=1$ to $K$}
        \STATE compute $||\zbf_{t}-w_{k}||_{2}^{2}$
        \ENDFOR
        \STATE find the closest centroid $w_{k^{*}(t)}$ from $\zbf_{t}$;
        \ENDFOR
        \FOR{$k=1$ to $K$}
        \STATE $w_{k} = \frac{1}{\# \{ t, k^{*}(t)=k \}} \displaystyle\sum_{\{ t, k^{*}(t)=k \}} \zbf_{t}$
        \ENDFOR
        \UNTIL {the stopping criterion is met}
    \end{algorithmic}
\end{algorithm}

Batch K-Means is an algorithm that produces, by construction, improved prototypes (in regards to the objective function \eqref{eq:EmpiricalDistortion}) for each iteration and that stabilizes on a local minimum of this objective function. In many cases, the prototypes $w$ and the corresponding empirical loss $C_{N}(w)$ are deeply modified in the first iterations of the Batch K-Means then they move much more slowly in the latter iterations, and after several dozens iterations the prototypes and the corresponding empirical loss are totally fixed. Yet, such a behavior is not systematic and Batch K-Means may need more iterations before stabilizing (see e.g. \cite{KMeansSlow}). The classical stopping criteria are: wait until the algorithm is completed (prototypes remain unmodified between two consecutive iterations), or run an a-priori fixed number of iterations or run the algorithm until the empirical loss gain between two iterations is below a given threshold. \\

Batch K-Means can be rewritten as a gradient-descent algorithm (see \cite{BottouBengioKMeans}). As one can notice in many other gradient-descent algorithms, Batch K-Means is very sensitive to initialization. More specifically, both the time to convergence and the quality of the clustering are strongly impacted by the prototypes initialization. We refer the reader to \cite{BRA1} for an in-depth review of different K-Means initialization techniques. In the following, the $K$ prototypes are initialized with the values of the first $K$ points of our data set and in the case of a distributed algorithm, all the computing units will be initialized with the same prototypes.\\

\subsection{Complexity cost}

The Batch K-Means cost per iteration does not depend on the actual data values but only on the data size. It is therefore possible to provide a precise cost for a Batch K-Means for a given data size. This cost has already been studied by Dhillon and Modha in \cite{DhillonModha2000}. Let us briefly list the different operations required to complete the algorithm. For a given number of iterations $I$, Batch K-Means requires: $I(K+N)d + IKd$ read operations, $IKNd$ subtractions, $IKNd$ square operations, $IKN(d-1) + I(N-K)d$ additions, $IKd$ divisions, $2IN + IKd$ write operations, $IKd$ double comparisons and an enumeration of $K$ sets whose cumulated size is $N$.\\

If the data set is small enough to fit into the RAM memory and if specific care is made to avoid most of cache miss (see for example \cite{Drepper}), the read and write operation costs can be neglected for the sequential Batch K-Means version. Making the reasonable approximation that additions, subtractions and square operations are all made in a single CPU clock cycle and that $N>>K$ and $N>>d$, one can model the time to run $I$ iterations of Batch K-Means by:\\
\begin{equation*}
SequentialBatch^{Wall time} = (3KNd+KN+Nd)IT^{flop},
\label{sequential KMeans cost}
\end{equation*}

where $T^{flop}$ denotes the time for a floating point operation to be evaluated.

\section{Distributed K-Means Algorithm on SMP and DMM architectures}
\label{Distributed K-Means Algorithm Section}

\subsection{Distribution scheme}
\label{sec_KMeans_SMP_Algo}

As already observed in \cite{DhillonModha2000}, the Batch K-Means algorithm is inherently data-parallel: the assignment phase which is the CPU-intensive phase of the sequential algorithm version, consists in the same computation (distance calculations) applied on all the points of the data set. The distance calculations are intrinsically parallel, both over the data points and the prototypes. It is therefore natural to split the computational load by allocating disjoint subsets of points to the different processing units. This property makes the Batch K-Means algorithm suitable for many distributed architectures: the assignment phase is shortened by distributing point assignments tasks over the different processing units. Overall, the distributed Batch K-Means wall time ---i.e. the human perception of the passage of time from the start to the completion of the algorithm, as opposed to the CPU time--- is reduced compared to its sequential counterpart because of the shortening of this assignment phase duration.\\

Let us assume that we own $M$ computing units and that we want to run a Batch K-Means over a data set composed of $N$ data points $\{\zbf_{t}\}_{t=1}^{N}$. The initial data set is split into $M$ parts of homogeneous size $S^{i}=\{\zbf_{t}^{i}\}_{t=1}^{n}$ for $ 1\leq i\leq M$ with $n=N/M$. The computing units are assigned an Id from $1$ to $M$ and the computing unit $m$ processes the data set $S^{m}$. Each computing unit is given the same computation load, and the different processors complete their respective tasks in similar amount of time. Once all the computing units have completed their task, one or multiple units can proceed to the recalculation phase, before the reassignment phase is run again. We detail this distributed algorithm version in the following logical code (Algorithm  \ref{algo_SMP_Distributed_KMeans}).\\
% has already been successfully distributed on several architectures.   can be distributed on several processors. We describe here a version of synchronous distributed K-Means. Results of the K-Means stay unchanged compared with the sequential Batch K-Means.

\begin{algorithm}
    \caption{Distributed Batch K-Means on SMP architectures}
    \begin{algorithmic}
    \label{algo_SMP_Distributed_KMeans}
    \STATE Code run by the computing unit m
    \STATE Get same initial prototypes $(w_{k})_{k=1}^{K}$ as other units
    \REPEAT
    \FOR{$t = 1$ to $n$}
        \FOR{$k=1$ to $K$}
            \STATE compute $||z_{t}^{m}-w_{k}||_{2}^{2}$
        \ENDFOR
        \STATE find the closest prototype $w_{k^{*}(t,m)}$ to $\zbf_{t}^{m}$
    \ENDFOR
    \FOR{$k=1$ to $K$}
        \STATE Set $p_{k}^{m} = \# \{ t, ~\zbf_t^{m} \in S^{m} ~\& ~k^{*}(t,m)=k \}$
    \ENDFOR
    \FOR{$k=1$ to $K$}
        \STATE $w_{k}^{m} = \displaystyle\frac{1}{p_{k}^{m}} \displaystyle\sum_{\{ t, ~\zbf_t^{m} \in S^{m} ~\& ~k^{*}(t,m)=k \}} {\zbf_{t}^{m}}$
    \ENDFOR
    \STATE Wait for other processors to finish the for loops
    \IF{i==0}
        \FOR{$k=1$ to $K$}
            \STATE Set $w_{k} = \displaystyle\frac{1}{\displaystyle\sum_{m=1}^{M}p_{k}^{m}} \displaystyle \sum_{m=1}^{M} p_{k}^{m}w_{k}^{m}$
        \ENDFOR
        \STATE Write $w$ into the shared memory so it is available for the other processing units
    \ENDIF
    \UNTIL {the stopping criterion is met}
    \end{algorithmic}
\end{algorithm}

As previously stated, this distributed algorithm produces after each iteration the very same result as the sequential Batch K-Means. The distributed Batch K-Means is therefore only evaluated on a speedup criterion: how much does this distributed algorithm version reduce the total time of execution? In many implementations, the overall time of execution is significantly reduced by the parallelization of the assignments and slightly increased by the communication between the different processors that is induced by the parallelization. We can therefore model the total time of execution by the equation \eqref{Synchronous Distributed K-Means Cost}.\\

\begin{equation}
Distributed ~Batch^{Wall Time} = T_{M}^{comp} + T_{M}^{comm},\\
\label{Synchronous Distributed K-Means Cost}
\end{equation}
where $T_{M}^{comp}$ refers to the wall time of the assignment phase and $T_{M}^{comm}$ refers to the wall time of the recalculation phase (mostly spent in communications).\\

The wall time of the assignment phase ($T_{M}^{comp}$) does not depend on the computing platform. This wall time roughly equals the assignment phase time of the sequential algorithm divided by the number of processing units $M$. Indeed, for this phase the translation of the algorithm toward the distributed version does neither introduce nor remove any computation cost. It only distributes this computation load on $M$ different processing units. Therefore, this wall time is modeled by the following equation \eqref{eq_TMComp}.\\

\begin{equation}
T_{M}^{comp} = \displaystyle \frac{(3KNd+KN+Nd)IT^{flop}}{M}.\\
\label{eq_TMComp}
\end{equation}

As previously stated, the wall time of the reassignment phase is mostly spent in communications. During this phase, the different $M$ prototypes versions computed by the $M$ processing units are merged together to produce an aggregated prototypes version (which is exactly the prototypes version that would have been produced by the sequential algorithm), and then this prototypes version (referred to as shared version in the following) is made available to each computing unit before the reassignment phase is restarted. In the following subsections, we investigate how the shared prototypes version is made available depending on the hardware architecture and how this communication process impacts the wall time of the recalculation phase on Symmetric Multi-Processors (SMP) and DMM architectures.\\

\subsection{Communication costs in SMP architectures}

SMP refers to a multiprocessor computer architecture where several identical processors are connected to a shared memory and controlled by a single OS instance. A typical SMP configuration is a multi-core processor, where each core is treated as a separate processor. In SMP architectures, the different processing units, the main memory and the hard-disks are connected through dedicated hardware such as buses, switches, etc. In this context, the communication is very fast.\\

In such an architecture, the communication costs of the recalculation phase are therefore very small compared to the processing costs of the reassignment phase. This brings us to neglect the communication costs in the case of a SMP architecture ($T_{M}^{comm, SMP} = 0$). One can then simplify the Batch K-Means cost on SMP architecture by the simplified equation \eqref{eq_SMP_Cost}.

\begin{equation}
Distributed ~Batch_{SMP}^{Wall Time}=T_{M}^{comp} = \displaystyle \frac{(3KNd+KN+Nd)IT^{flop}}{M}.
\label{eq_SMP_Cost}
\end{equation}

This model of Batch K-Means on SMP architectures provides a perfect theoretical model where the neglect of SMP communication costs leads to a theoretical perfect speedup of a factor $M$ for $M$ processing units.\\

\subsection{Communication costs in DMM architectures}
\label{sec_DMMCosts}
In contrast with SMP architectures, a DMM system refers to a multi-processor in which each processor has a private memory and no shared memory is available. In such a system, the prototypes version computed by the processing unit $i$ is accessible by the processing unit $j$ (with $i \neq j$), if and only if the processing unit $i$ explicitly sends its result to the processing unit $j$. Since all the communications in DMM architectures need to be explicit, several frameworks have been devised to provide communication and synchronization primitives. These frameworks disburden the application developer from specific communication handling by providing a higher level communication layer. Among such frameworks, the most well-known ones are Message Passing Interface (MPI) (see e.g. \cite{MPI1996}) or PVM (see e.g. \cite{PVM}). \\

The wall time of the synchronous distributed Batch K-Means algorithm on DMM has been studied by Dhillon and Modha in \cite{DhillonModha2000} then by Joshi in \cite{KMeansJoshi}. In both cases, Batch K-Means is distributed on DMM architecture using a MPI framework. In \cite{DhillonModha2000}, this wall time is modeled by the following equation \eqref{eq_Dhillon}.\\

\begin{align}
\label{eq_Dhillon}
Distributed ~Batch_{DMM}^{Wall Time} & = T_{M}^{comp} + T_{M}^{comm, DMM}\\
& = \frac{(3KNd+KN+Nd)IT^{flop}}{M} + O(dKIT_{M}^{reduce}),
\end{align}

%\begin{align}
%\label{eq_Dhillon}
%\(Distributed ~Batch_{DMM}^{Wall Time}\) & \(=\) \(T_{M}^{comp} + T_{M}^{comm, DMM}\) \\
%& \(=\) \(\frac{(3KNd+KN+Nd)IT^{flop}}{M} + O(dKIT_{M}^{reduce}) \) \\
%\end{align}

%\begin{tabular}{ r c l }
%\(Distributed ~Batch_{DMM}^{Wall Time}\) & \(=\) & \(T_{M}^{comp} + T_{M}^{comm, DMM}\) \\
%   & \(=\) & \(\frac{(3KNd+KN+Nd)IT^{flop}}{M} + O(dKIT_{M}^{reduce}) \) \\
%\label{eq_Dhillon}
%\end{tabular}\\

where $T_{M}^{reduce}$, following the notation of \cite{DhillonModha2000}, denotes the time required to perform a sum or an average operation of $M$ doubles distributed on $M$ processing units. \\

To determine the wall time of the recalculation phase on DMM architectures using MPI, one therefore needs to determine the time to perform a sum or an average operation distributed on $M$ processing units, referred to as $T_{M}^{reduce}$. This quantity is determined by the design of MPI: the MPI framework can perform communication and can broadcast data between the processing units using a tree-like topology that is described in the following subsection (\ref{section_InDepthMpiCosts}). For such a tree-like topology, the communication and broadcasting primitives are reported to be performed in $O(log(M))$. As a consequence, $T_{M}^{reduce}$ is also reported to be performed in $O(log(M))$ (see e.g. \cite{MPI_performance}).\\

In many cases, the hardware architecture provides enough bandwidth for MPI to be very efficient. In the experiments made in \cite{DhillonModha2000}, the communication costs are very acceptable and have little impact on the distributed Batch K-Means speedup (see Subsection \ref{sec_Dhillon_Experiments}). Yet, the communication latency and the bandwidth between processing units should not be neglected in many other cases. The following subsections provide a more detailed evaluation of $T_{M}^{reduce}$ and compute some bandwidth condition to prevent communications from becoming a bottleneck of the algorithm.\\

\subsection{Modeling of real communication costs}
\label{section_InDepthMpiCosts}

A lot of work has been done to provide precise model of the MPI primitives performances over the last two decades. In \cite{MPI_performance}, numerical values of many MPI execution performances are provided. As outlined in \cite{MPI_performance_measurements}, achieving such numerical measurements is a very difficult task that highly depends on the underlying hardware topology, on the synchronization methods, on the network congestion, on the different MPI implementations, etc. In this subsection we do not aim to provide accurate performance evaluation of the MPI behavior in our clustering context but rather to explicit qualitative patterns of the constraints brought by communications.\\

Let us consider the communication scheme described in Figure \ref{fig_MPI_Tree}. This overly-simplified communication scheme highlights the tree-like nature of the communication patterns adopted by many MPI implementations. More specifically, the figure highlights the logarithmic scaling of the communication mechanism: to sum or average values across computing units, many MPI implementations have developed a structure where data chunks are sent in multiple steps. Each step of this communication pattern consists in two actions: a processing unit sends its data to a second one, the receiving unit merges the two data chunks into a data chunk of the same size. After each step, the number of data chunks that are sent in parallel is divided by 2. Such a pattern induces $\lceil log_{2}(M) \rceil$ communication steps. \\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale=0.6]{../../Pictures/HardwareSchemes/Merginglogic.png}
\end{center}
\caption{This non-realistic scheme of the merging prototype logic highlights the tree structure of many MPI primitives. To begin with, the first worker sends a data chunk to the second worker, while the third worker sends simultaneously its data chunk to the fourth worker. A second step follows the first one in which the merging result of the first and second workers are sent to the fourth worker that already owns the merging result of the third and fourth workers. In two steps, the data chunks of the four workers are merged. }
\label{fig_MPI_Tree}
\end{figure}

In the following equations, we neglect communication latencies as well as the time to merge the data chunk (which is small in comparison with the data communication between the two processing units) and model communication costs as the ratio of the quantity of data to be sent divided by the bandwidth. We therefore provide a theoretic model of $T_{M}^{comm, DMM}$ by:\\

\begin{equation*}
T_{M}^{comm, DMM} = \displaystyle \sum_{m=1}^{\lceil log_{2}(M) \rceil} \frac{IKdS}{B_{\frac{M}{2^{m}}}^{DMM, MPI}},
\end{equation*}

where $S$ refers to the size of a double in memory (8 bytes in the following) and $B_{x}^{DMM, MPI}$ refers to the communication bandwidth per machine while $x$ processing units are communicating at the same time. The number of processing units that are communicating at the same time $x$ has a strong impact on $B_{x}^{DMM, MPI}$ because of a phenomenon referred to as aggregated bandwidth bounds.\\

The aggregated bandwidth refers to the maximum number of bytes the actual network can transfer per time unit. We refer the reader to Subsection \ref{sec_AggregatedBandwidth} for a more precise explanation. As regards our model, let us make the strong simplification that the bandwidth per worker only depends on the hardware context and on the total number of processing units, and not on the actual number of processing units communicating at the same time. We therefore note this bandwidth $B_{M}^{DMM, MPI}$.\\

The time spent in communication is then written:
\begin{align*}
T_{M}^{comm} & = \displaystyle \sum_{m=1}^{\lceil log_{2}(M) \rceil} (\frac{IKdS}{B_{M}^{DMM, MPI}})\\
& = \lceil log_{2}(M)\rceil \frac{IKdS}{B_{M}^{DMM, MPI}}.
\end{align*}

We can then deduce an estimation of the speedup rate on DMM architectures more detailed than in \cite{DhillonModha2000}, where $T_{M}^{comm}$ is specified :

\begin{align}
\label{SpeedupFormula}
Speedup(M,N) & = \frac{SequentialBatch^{Wall time}}{Distributed ~Batch_{DMM}^{Wall Time}}\\
& = \frac{T_{1}^{comp}}{T_{M}^{comp} +T_{M}^{comm}}\\
& = \frac{3NKdIT^{flop}}{\frac{3NKdIT^{flop}}{M} + \frac{IKdS}{B_{M}^{DMM, MPI}} \lceil log_{2}(M) \rceil}\\
& = \frac{3NT^{flop}}{\frac{3NT^{flop}}{M} +  \frac{S}{B_{M}^{DMM, MPI}} \lceil log_{2}(M) \rceil}.
\end{align}

Using the previous speedup model, one can deduce the optimal number of processing units for distributed Batch K-Means implemented on DMM architectures:

\begin{align}
\label{eq_MStar_DMM}
M_{DMM}^{*}  =  \frac{3NT^{flop}B_{M}^{DMM,MPI}}{S}.
\end{align}

\subsection{Comments}
\label{sec_DMM_Comments}

The speedup model provided in the previous Subsection \ref{section_InDepthMpiCosts} is too simple to provide a satisfactory forecasting tool to anticipate with enough precision the actual speedup provided by a distributed Batch K-Means implementation on a given architecture. Yet, this naive model allows us to draw some qualitative conclusions shared by many distributed Batch K-Means implementations on shared-nothing architectures:\\

\begin{itemize}
\item The speedup depends neither on the number of prototypes ($K$), nor on the data dimension ($d$), neither does it depend on the data distribution. It only depends on the number of points in the data set ($N$) and on architecture characteristics such as bandwidth or CPU frequency.\\

\item The speedup is a monotonically increasing function of $N$. In particular, the previous equation leads to $\displaystyle\lim_{N\rightarrow +\infty} SpeedUp(M,N) = M $. From this theoretical result, a conclusion can be drawn: the more loaded the RAM of workers is, the more efficient the workers are. Provided the processing units are given enough RAM, it is possible to get an efficiency per worker arbitrarily close to 100\%.\\

\item The actual speedup of the distributed implementation might even be higher than the theoretical speedup model since we have not taken into account the fact that the RAM is bounded.
    Indeed, the accumulated RAM from each of the machines may enable the data set to move from disk into RAM thereby drastically reducing the time access for read and write operations. This phenomenon is often referred to as superlinear speedup.\\

\item The previous equation shows that speedup is a monotonically decreasing function of $T^{flop}$ and a monotonically increasing function of $B_{M}^{DMM, MPI}$. This result shows that the parallelization is best suited on slow machines with high communication bandwidth.\\

\item Because of the actual values of bandwidths and CPU frequencies in present real world DMM architectures, it is always efficient to use all the available processing units of a DMM architecture to perform a distributed Batch K-Means.\\
\end{itemize}

\subsection{Bandwidth Condition}

In this subsection we briefly determine some theoretical conditions that guarantee that the recalculation phase is small in comparison with the reassignment phase. This condition turns into:\\

\begin{equation*}
T_{M}^{comm} << T_{M}^{comp}.
\label{DMM1 GeneralCase}
\end{equation*}

Such a condition translates into:

\begin{equation}
\frac{IKdS}{B_{M}^{DMM, MPI}} \lceil log_{2}(M) \rceil << \frac{(3NKd+NK+Nd)IT^{flop}}{M}.
\label{DMM2 GeneralCase}
\end{equation}

A sufficient condition for equation \eqref{DMM2 GeneralCase} to be verified is that the following equation \eqref{DMM3 GeneralCase} is verified:\\

\begin{equation}
\frac{IKdS}{B_{M}^{DMM, MPI}} \lceil log_{2}(M) \rceil <<\frac{3NKdIT^{flop}}{M}.
\label{DMM3 GeneralCase}
\end{equation}

Finally we get the following condition:

\begin{align*}
\frac{N}{M\lceil log_{2}(M) \rceil} >> \frac{S}{3T^{flop}B_{M}^{DMM, MPI}}.
\label{GeneralBandwidthCondition}
\end{align*}

For example, in the context of a DMM architecture with 50 computing units composed of a single mono-core retail processor (2Ghz) and a network interface controller with a $1$ Gbit/sec bandwidth, the condition turns into:

\begin{equation*}
N >> 12,000.
\end{equation*}

Assuming that the dimension $d=1000$, it is therefore necessary to have a data set of more than 10 GBytes (i.e. 200 MBytes per machine) on such an architecture, to ensure that communications will cost less than 1\% of the computation duration.

\subsection{Dhillon and Modha case study}
\label{sec_Dhillon_Experiments}

The experiments of \cite{DhillonModha2000} have been run on an IBM SP2 platform with a maximum of 16 nodes. Each node is an IBM POWER2 processor running at 160MHz with 256 MBytes of main memory. Processors communicate through the High-Performance Switch (HPS) with HPS-2 adapters. Performance of the HPS has been discussed in \cite{HPS_2} and \cite{HPS_1}. In \cite{HPS_1} for example, point-to-point bandwidth in a SP2 with HPS and HPS-2 adapters using MPI is reported to be about 35 MBytes/sec.\\

It is difficult to estimate the bandwidth actually obtained during the experiments of \cite{DhillonModha2000}, as it is also difficult to estimate effective Flops. In \cite{DhillonModha2000}, no comments are made about bandwidth, yet Flops are reported to be very fluctuent: 1.2 GFlops in some experiments, 1.8 GFlops in others (the maximum would be 16 * 160MFlops = 2,5GFlops). Yet, on smaller data sets, Flops might be very much lower than the reported numbers. Since we cannot guess the actual bandwidth and Flops during these experiments, theoretical bandwidth and Flops are used in the following to provide approximative results.\\

Dhillon and Modha report that they have obtained a speedup factor of 6.22 on 16 processors when using $2^{11}$ points.
Using condition \eqref{SpeedupFormula}, we can estimate speedup when using $2^{11}$ points :

\begin{align*}
Estimated Speedup & = \frac{3NT^{flop}}{\frac{3NT^{flop}}{M} +  \left(2 \frac{S}{B_{M}^{DMM, MPI}} + 5T^{flop}\right) \lceil log_{2}(M) \rceil}\\
& \simeq 11.
\end{align*}

We can see the estimated speedup is not accurate, but we are in the range where communication is important enough to prevent a perfect speedup and small enough though for a significant speedup to be observed.\\

When using $2^{21}$ points, we get :

\begin{align*}
Estimated Speedup & = \frac{3NT^{flop}}{\frac{3NT^{flop}}{M} +  \left(2 \frac{S}{B_{M}^{DMM, MPI}} + 5T^{flop}\right) \lceil log_{2}(M) \rceil}\\
& \simeq 15.996.
\end{align*}

\cite{DhillonModha2000} reports that for $2^{21}$ points, a 15.62 speedup is observed. Again, anticipated speedup indicates there will be little issue to parallelize, and observed speedup is indeed excellent.

%
%\section{Some Case study of K-Means on DMM architectures}
%\label{Distributed K-Means Case Study}
%
%Equation (\ref{GeneralBandwidthCondition}) sets general condition for avoiding bandwidth to be a bottleneck. When dealing with a local cluster, the design of the cluster can be chosen so the condition holds. In this section, we examine some case study and check if this condition holds or not.
%
%\subsection{First case study}
%First case study, we want to cluster through a K-Means all the electricity French load curves. The time series are observed every hour for 10 years. We want the number of clusters to increase with the number of series to cluster, but not as fast as linearly, so we will say we want as many clusters as the square root of the number of time series. We suppose for the modelisation that the clustering processus will stop after 100 iterations. The computation are run on a network of monocore processors of 1GFlops, which makes 1 000 000 000 operations per second.
%Numbers in this experience are :
%
%\begin{enumerate}
%\item n = 20 000 000 time series
%\item d = 24 hours * 365 days * 10 years = 87600 values per series
%\item K = $\sqrt{n}$ = 4472 clusters
%\end{enumerate}
%
%Note : volume of time series is 12,3 TeraBytes.
%
%Note : setting K as $\sqrt{n}$ is an heuristic to ensure K growths with n, but not so fast than n.\\
%
%We will estimate cost of clustering theses data on 3 different hardware cases :
%
%case 1 : a monocore 2GFlops processor (we do not take into account this amount of data cannot fit into memory). Using modelisation \ref{sequential KMeans cost} this would take 37 years to process.
%
%case 2 : The hardware used at Yahoo \cite{TeraSortWinnerYahoo} to win the TeraSort contest \cite{SortBenchmark}.\\
%This hardware is mainly caracterised by :
%\begin{itemize}
%\item 910 nodes
%\item 4 dual core Xeons @ 2.0ghz per a node
%\item 1 gigabit ethernet on each node
%\item 40 nodes per a rack
%\item 8 gigabit ethernet uplinks from each rack to the core
%\end{itemize}
%
%We would get :\\
%
%\begin{equation}
%DMM ~StudyCase1 ~Cost = T_{P}^{comp} + T_{P}^{comm}
%\label{DMM1 StudyCase1}
%\end{equation}
%
%where $T_{P}^{comp}$ is :
%
%\begin{align*}
%T_{P}^{comp} & = \displaystyle \frac{(3nKd+nK+dK+nd)*I*T^{flop}}{P}\\
%& = 322893 seconds\\
%\label{DMM2 StudyCase1}
%\end{align*}
%
%and $T_{P}^{comm}$ became :
%
%\begin{align*}
%T_{P}^{comm} & = (2*~Broadcast ~time ~between ~2 ~processors + Merging ~time) * \lceil log_{2}(P) \rceil\\
%& = (2\frac{IKd*SoD}{B_{P,avg}^{DMM, MPI}} + 5IKd*T^{flop}) * \lceil log_{2}(P) \rceil\\
%& = 225866 seconds\\
%\end{align*}
%
%Therefore, it would take about 6 days to finish the computation and computation would take more time than communication .\\
%
%case 3 : this time we suppose we can run on the same hardware, but the switch on top of all the racks has a limited aggregated bandwidth of 100Gbits/secs. Aggregated bandwidth is now limiting and  we can modelize  $B_{P,avg}^{DMM, MPI} = 109Mbits$.
%
%\begin{align*}
%T_{P}^{comm} & = (2*~Broadcast ~time ~between ~2 ~processors + Merging ~time) * \lceil log_{2}(P) \rceil\\
%& = (2\frac{IKd*SoD}{B_{P,avg}^{DMM, MPI}} + 5IKd*T^{flop}) * \lceil log_{2}(P) \rceil\\
%& = 434155 seconds\\
%\end{align*}
%
%ADD EVALUATION OF COST OF THE CLUSTER.\\
%ADD EVALUATION OF COST ON THE CLOUD IF FOLLOWING ALGORITHM DEVELOPED IN SECTION 7\\


\section{Implementing Distributed Batch K-Means on Azure}
\label{sec_KMeans_Azure}

\subsection{Recall of some Azure specificities}

The architecture of most Azure hosted applications is based on two components: \textit{web roles} and \textit{worker roles}. Web roles are designed for web application programming. In the context of our cloud-distributed Batch K-Means prototype, a single web role is used for monitoring the algorithm behaviour, observing the queues load, noticing the BlobStorage short outages or unavailability, profiling our algorithm and redeploying quickly our application using the Appdomain trick (see Subsection \ref{sec_PersonalNotes} and \cite{AppDomainTrick}). Worker roles are designed to run general background processing. Each worker role typically gathers several \emph{cloud services} and uses many \emph{workers} (Azure's processing units) to execute them. Our cloud-distributed Batch K-Means prototype uses only one worker role, several services and tens or hundreds of workers. This worker role is used to run the actual Batch K-Means algorithm.\\

The computing power is provided by the workers, while the Azure storage system is used to implement synchronization and communication between workers. It must be noted indeed that Azure does not currently offer any standard API for distributed computation, neither a low-level one such as MPI, nor a higher-level one such as MapReduce (\cite{MapReduce2004}) or Dryad (\cite{Dryad2007}). MapReduce could be implemented using Azure components (following the strategy of
\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{Graphlab2010},
those high-level API might be inappropriate for iterative machine-learning
algorithms such as Batch K-Means. Therefore we rely directly on the Azure queues and the
BlobStorage.\\

%\textbf{Azure Queues} provide a message delivery mechanism through distributed
%queues. Queues are designed to store a large amount of small messages (with
%maximal individual size of 8 KB). Using queues to communicate helps building
%loosely coupled components and mitigates the impact of individual component
%failure. Here, queues are used to store job messages. Messages stored in a
%queue are guaranteed to be returned at least once, but possibly several times:
%this requires one to design idempotent jobs.  When a message is unqueued by a
%worker, this message is not deleted but it becomes invisible for other
%workers. If a worker fails to complete the corresponding job (because it
%throws some exception or because the worker dies), the message becomes
%available after a certain period of time. Through this process, one can make
%sure no job is lost because of e.g., a hardware failure.\\
%
%\textbf{Azure Blob Storage} enables applications to store large objects, up to
%50 GB each.  It supports a massively scalable blob system, where hot blobs will
%be served from many servers to scale read access. Blobs are composed of a
%string (that is used as a key to store the value), a value composed of a
%binary object, and a timestamp (etag) (that indicates the last write on this
%blob). In addition of Get and Put methods, blobs whose key shares a given
%prefix can be listed. Optimistic non locking atomic read-modify-write
%operations can be implemented using a timestamp matching condition: a write
%succeed if and only if the timestamp of the storage matches the one provided
%by the write operation.\\

Our prototype uses
\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}} (see Subsection \ref{sec_LokadCloud}), an
open-source framework that adds a small abstraction layer to ease Azure
workers startup and life cycle management, and storage access.\\

\subsection{The cloud Batch K-Means algorithm}
\label{sec_CloudKMeans_Algorithm}

As already outlined, the distributed Batch K-Means can be reformulated as an iterated MapReduce where the assignment phases are run by the mappers and the recalculation phases are performed by the reducers. The design of our distributed cloud Batch K-Means is therefore vastly inspired by the MapReduce abstraction and follows the considerations introduced in Chapter \ref{chap:conceptionCloud}. This section presents our cloud implementation. Figure \ref{fig_CloudKMeans} provides a scheme of the cloud implementation of our distributed Batch K-Means. Algorithms \ref{algo_CloudKMeansMapper} (resp. \ref{algo_CloudKMeansPartialReducer} and \ref{algo_CloudKMeansFinalReducer}) reproduce the logical code run by the mappers (resp. the partial reducers and the final reducer).\\

\begin{figure}
    \centering
    \includegraphics[scale=0.6]{../../Pictures/MapReduce/DistributedKMeans}
    \caption{Distribution scheme of our cloud-distributed Batch K-Means. The communications between workers are conveyed through the BlobStorage. The recalculation phase is a two-step process run by the partial reducers and the final reducer to reduce I/O contention.}
    \label{fig_CloudKMeans}
\end{figure}

\begin{algorithm}
    \caption{Distributed Cloud Batch K-Means : Mapper}
    \begin{algorithmic}
    \label{algo_CloudKMeansMapper}
    \STATE Dequeue a message from the Map Queue.
    \STATE Get taskId, groupId and IterationMax from the message.
    \STATE Set m=taskId
    \STATE Retrieve the partial data set $S^{m}= \{\zbf_{i}^{m}\}_{i=1}^{N}$ from the storage
    \STATE Retrieve the initial prototypes shared version $\{w^{srd}_{k}\}_{k=1}^{K}$
    \STATE Initialize $w^{m}$ as $w^{m}=w^{srd}$
    \FOR{It=0; It < IterationMax ; It++}
        \FOR{$\zbf_{i}^{m} \in S^{m}$}
            \FOR{$k=1$ to $K$}
                \STATE Compute $||\zbf_{i}^{m}-w_{k}||_{2}^{2}$
            \ENDFOR
            \STATE Find the closest prototype $w_{k^{*}(i,m)}$ to $\zbf_{i}^{m}$
        \ENDFOR
        \FOR{$k=1$ to $K$}
            \STATE Set $p_{k}^{m} = \# \{ t, ~\zbf_t^{m} \in S^{m} ~\& ~k^{*}(t,m)=k \}$
        \ENDFOR
        \FOR{$k=1$ to $K$}
            \STATE Set $w_{k}^{m} = \displaystyle\frac{1}{p_{k}^{m}} \displaystyle\sum_{\{ t, ~\zbf_t^{m} \in S^{m} ~\& ~k^{*}(t,m)=k \}} {\zbf_{t}^{m}}$
        \ENDFOR
        \STATE Send $w^{m}$ into the storage in a location depending on the iteration It
        \STATE Send $p^{m}$ into the storage in a location depending on the iteration It
        \STATE Ping the storage every second to check if $w^{srd}$ is available for iteration It
        \STATE Download it when it becomes available
        \STATE Replace $w^{m}$ by the new downloaded shared version
    \ENDFOR
    \end{algorithmic}
\end{algorithm}

\begin{algorithm}
    \caption{Distributed Cloud Batch K-Means : Partial Reducer}
    \begin{algorithmic}
    \label{algo_CloudKMeansPartialReducer}
    \STATE Dequeue a message from the Partial Reduce Queue.
    \STATE Get groupId and IterationMax from the message.
    \STATE Set g= groudId
    \FOR{It=0; It < IterationMax ; It++}
        \STATE Retrieve the prototypes version $\{w^{m}\}_{m=g\sqrt{M}}^{(g+1)\sqrt{M}}$ corresponding to iteration It
        \STATE Retrieve the corresponding weights $\{p^{m}\}_{m=g\sqrt{M}}^{(g+1)\sqrt{M}}$ corresponding to iteration It
        \FOR{$k=1$ to $K$}
            \STATE Set $p_{k}^{g}=\displaystyle \sum_{m=g\sqrt{M}}^{(g+1)\sqrt{M}} p_{k}^{m}$
            \STATE Set $w_{k}^{g}= \frac{1}{p_{k}^{g}} \displaystyle \sum_{m=g\sqrt{M}}^{(g+1)\sqrt{M}} p_{k}^{m} w_{k}^{m} $
        \ENDFOR
        \STATE Send $w^{g}$ into the storage in a location depending on the iteration It
        \STATE Send $p^{g}$ into the storage in a location depending on the iteration It
    \ENDFOR
    \end{algorithmic}
\end{algorithm}

\begin{algorithm}
    \caption{Distributed Cloud Batch K-Means : Final Reducer}
    \begin{algorithmic}
    \label{algo_CloudKMeansFinalReducer}
    \STATE Dequeue the single message from the Final Reduce Queue.
    \STATE Get IterationMax from the message.
    \FOR{It=0; It < IterationMax ; It++}
        \STATE Retrieve the prototypes version $\{w^{g}\}_{g=1..\sqrt{M}}$ corresponding to iteration It
        \STATE Retrieve the corresponding weights $\{p^{g}\}_{g=1..\sqrt{M}}$ corresponding to iteration It
        \FOR{$k=1$ to $K$}
            \STATE Set $p_{k}^{srd}=\displaystyle \sum_{g=1}^{\sqrt{M}} p_{k}^{g}$
            \STATE Set $w_{k}^{srd}= \frac{1}{p_{k}^{srd}} \displaystyle \sum_{g=1}^{\sqrt{M}} p_{k}^{g} w_{k}^{g} $
        \ENDFOR
        \STATE Send $w^{srd}$ into the storage in a location depending on the iteration It
    \ENDFOR
    \end{algorithmic}
\end{algorithm}

Following the MapReduce terminology, we split our algorithm into three cloud services (setup, map and reduce services), each one matching a specific need. A queue is associated to each service; it stores messages specifying the storage location of the data needed for the tasks. The processing units regularly ping the queues to acquire a message. Once it has acquired a message, a worker starts running the corresponding service, and the message becomes invisible till the task is completed or timeouts. Overall, we use $M+\sqrt{M}+1$ processing units in the services described below (we suppose in the following that $\sqrt{M}$ is an integer).\\

The \textit{SetUpService} generates $M$ split data sets of $n=N/M$ points in each and puts them into the BlobStorage. It is also generating the original shared
prototypes which are also stored in the BlobStorage. Once the processing units in charge of the set-up have completed the data generation, they push $M$ messages in the queue corresponding to the ``Map Service''. Each message contains a taskId (from $1$ to $M$) related to a split data set to be processed and a groupId (from $1$ to $\sqrt{M}$), which is described above. The same processing unit also pushes $\sqrt{M}$ messages in the queue corresponding to the ``Reduce Service''. In the current implementation\footnote{available at http://code.google.com/p/clouddalvq/}, the SetUp service is executed by $M$ processing units to speedup the generation process.\\

Once the set-up has been completed, the Map queue is filled with $M$ messages (corresponding to the $M$ map tasks) and the Reduce queue is filled with $\sqrt{M}$ messages (corresponding to the $\sqrt{M}$ reduce tasks). Each processing unit pings the different queues to acquire a Map task or a Reduce task.\\

When executing a \textit{MapService} task, a processing unit is referred to as a mapper. Each mapper first downloads the corresponding partial data set it is in charge of (once for all). Then the mapper loads the initial shared prototypes and starts the distance computations that form the assignment phase. Once the assignment phase has been completed, the mapper builds a local version of the prototypes according to the locally run assignment phase. This prototypes version is sent to the BlobStorage, and the mapper waits for the Reduce service to produce a shared version of the prototypes. When the shared version of the prototypes is made available, the mapper downloads it from the storage and restarts the assignment phase using the new prototypes version thus obtained. The instructions performed by the mappers are detailed in Algorithm \ref{algo_CloudKMeansMapper}.\\

When executing a \textit{ReduceService} task, a processing unit is referred to as a partial reducer. Each partial reducer downloads from the storage multiple prototypes versions that come from different Map tasks, and merges them into an average prototypes version. More specifically, each Reduce task consists in merging $\sqrt{M}$ prototypes versions. The Reduce task message holds an Id called groupId that refers to the group of prototypes versions that this task needs to collect. When the $\sqrt{M}$ prototypes versions it is in charge of are retrieved, the partial reducer merges the prototypes versions using weighted averages and pushes the merged result into the storage. Once all the $\sqrt{M}$ Reduce tasks have been completed, a last reducer, referred to as the final reducer, downloads the $\sqrt{M}$ merged results thus produced and merges all of them into a single prototypes version called shared version. This shared version is pushed into the storage to be made available for the mappers\footnote{More specifically, the final reducer pushes $\sqrt{M}$ copies of the shared prototypes version in the storage instead of one. Each of these copies is read by all the mappers sharing the same groupId.}. When this shared version is read by the mappers, a new iteration of the algorithm is started and the assignment phase is re-run by the mappers. The instructions performed by the partial reducers (resp. the final reducer) are detailed in Algorithm \ref{algo_CloudKMeansPartialReducer} (resp. Algorithm \ref{algo_CloudKMeansFinalReducer}).\\

The two-step design of the ReduceService (partial reducers and the final reducer) is of paramount importance. This design ensures no processing unit needs to download more than $\sqrt{M}$ prototypes versions per iteration. In the same way, $\sqrt{M}$ copies of the shared prototypes version are made available in the storage instead of one, to ensure no blob is requested in parallel by more than $\sqrt{M}$ processing units per iteration.\\

For each iteration of the algorithm, there are three synchronization barriers. Firstly, each partial reducer needs to get the results of the $\sqrt{M}$ mappers it is related to before merging them. Secondly, the final reducer needs to get the results of all the $\sqrt{M}$ partial reducers before merging them into the shared prototypes version. Finally, each mapper needs to wait for the final reducer to push the shared version into the BlobStorage before restarting the reassignment phase. As outlined in Chapter \ref{chap:conceptionCloud}, the synchronization primitive is one of the few primitives that is not directly provided by Azure. Let's consider two different designs to implement this synchronization process.\\

The first solution consists in using a counter to keep track of the number of tasks that need to be completed before the synchronization barrier can be lifted. This counter needs to be idempotent and designed so that the contention should be limited. Such a counter has been already described in the BitTreeCounter section of Chapter \ref{chap:conceptionCloud}. For example, all the mappers that process tasks sharing the same groupId could own such a counter initially set to $\sqrt{M}$; once a mapper is done, it decrements the counter, and when a mapper decrements the counter from 1 to 0, it knows all the other mappers of the same groupId are also done, so that the corresponding partial reduce task can be started.\\

The second solution comes from the ability for each processing unit to ping a specific blob storage location to determine if there is a blob in there. Let us re-examine the previous example. Instead of using a specific counter and starting the reducer only when the counter hits 0, the partial reducer could be started from the beginning, together with the mappers. This partial reducer would then regularly query the storage to detect whether the prototypes versions produced by the mappers have already been made available. When all the $\sqrt{M}$ versions have been retrieved, then the partial reducer can start the merging operation. This second solution requires that each blob put into the storage should have a corresponding blobName that is pre-defined using a fixed rule; in the case of prototypes versions made by mappers, the corresponding blobNames are built using the following addressing rule: prefix/iteration/groupId/jobId. \\

Both solutions provide idempotence and avoid contention. The second solution has been chosen because it helps to reduce stragglers (see Subsection \ref{sec_StragglerIssues}) by overlapping the latest computations of mappers with the retrieval of first available results by the partial reducer. This second solution comes with the drawback of running more workers ($M + \sqrt{M} + 1$) than the first solution ($M$). This drawback is not that significant since ---as this is highlighted in the following subsections--- the scale-up of our algorithm will be more limited by the bandwidth/CPU power of our machines than by the actual cost of these machines.\\

Let us remark that the two-step design of our clustering prototype has been inspired by previous design choices that we adopted unsuccessfully. Indeed, our first design was a single-step reduce run by a single worker, with a synchronization barrier between the assignment phase and the recalculation phase in the form of a single naive blob counter. The contention on this naive blob counter, just as the overload on the single reducer motivated us to the design presented above.\\


%
%In practice, when all the points of a map service task have been processed
%once, the local prototypes version and their weights are pushed into the
%storage according to the following addressing rule: iteration/groupId/jobId.
%The worker then starts waiting, pinging the shared prototypes of the current
%iteration until it becomes available.\\
%
%Each \textbf{Reduce Service} task has a groupID, obtained from one of the
%$\sqrt{M}$ messages pushed into the reduce queue. As explained above, each
%reducer is listing blobnames with prefix: iteration/groupId in the BlobStorage
%until it gets all the results expected within this groupId. Map results
%are downloaded as soon as available. Once all the map results expected have
%been retrieved and loaded locally, the reducer merges the results into a
%partial merged result, and pushed this in a partial reduce results
%directory. One last reducer runs the same process on the partial reduce results
%directory. Once the $\sqrt{P}$ partial reduce results are retrieved, the last
%reducer builds a new shared version out of the partial reduce results and pushes the
%shared prototypes into the storage. After several seconds, every map worker
%has been pinging the now available shared result blob, and the map step can be
%run again.\\
%
%
%The first algorithm we propose is the most adapted to cloud services philosophy of Azure : we are running an Iterative MapReduce, messages queued indicating whether it is a map or reduce operation, and where the data can be retrieved.


%\begin{figure}
%    \centering
%    \includegraphics[scale=0.3]{../../Pictures/HardwareSchemes/centroidComputation1}
%    \caption{centroidComputation1}
%\end{figure}
%
%\begin{figure}
%    \centering
%    \includegraphics[scale=0.3]{../../Pictures/HardwareSchemes/centroidComputation2}
%    \caption{centroidComputation2}
%\end{figure}
%
%\begin{figure}
%    \centering
%    \includegraphics[scale=0.3]{../../Pictures/HardwareSchemes/centroidComputation3}
%    \caption{centroidComputation3}
%\end{figure}

\subsection{Comments on the algorithm }

In the previous proposed algorithm as well as in the DMM implementation of \cite{DhillonModha2000}, the assignment phase of Batch K-Means is perfectly parallelized. As already outlined in the model of speedup in the case of DMM architectures, the global performances are hindered by the wall time duration of the reduction phase. In the case of a SMP architecture, the recalculation phase is approximately instantaneous and the parallelization in such architectures is very efficient. On DMM architectures, we have already shown how the MPI framework provides very efficient primitives to perform the reduction phase in amounts of time that are in most cases negligible (with a $O(log(M))$ cost). In such cases, the DMM implementations lead to a linear speedup nearly as optimal as in the SMP case, as confirmed in the large data set experiments of \cite{DhillonModha2000}.\\

In the case of our Azure implementation that relies on the storage instead of on direct inter-machine communications, the recalculation phase is proportionally much longer than in the DMM case. The two-step design of our algorithm produces a recalculation phase cost of $O(\sqrt{M})$ because the partial reducers as well as the final reducer need to download $\sqrt{M}$ prototypes versions per iteration. Such a cost is asymptotically much higher than the asymptotical cost of the MPI framework ($O(log(M))$). While it would have been possible for our reduction architecture to complete in $O(log(M))$ by implementing a $log(M)$-step architecture, such a design would have led to higher recalculation costs because of frictions. Indeed, the latency between the time a task to being completed and the time the consumer of the former task manages to acknowledge that the result is available, is rather high. This phenomenon highlights a previously stated remark (see Chapter \ref{chap:azurepres}): the cloud platforms are probably not suited to process very fine-grained tasks.\\

We also draw the reader's attention on the importance of the data set location and downloading. The previous algorithm is inspired by an iterated MapReduce with the noticeable exception that no Map task corresponds to a single iteration of K-Means but to all the iterations. Since Azure does not provide any mechanism to run computations on the physical machines where the data are stored through the BlobStorage, each processing unit need to download the data set from the storage (i.e. from distant machines). To prevent each of the workers from re-loading a data set at each iteration, we have chosen that each processing unit should be processing the same data chunk for each iteration.\\

\subsection{Optimizing the number of processing units}

A very important aspect of our cloud implementation of Batch K-Means is the elasticity provided by the cloud. On SMP or DMM architectures, the quantity of CPU facilities is bounded by the actual hardware, and the communications are fast enough. As a result, the user has better run the algorithm on all the available hardware. The situation is significantly different for our cloud implementation. Indeed, the cloud capabilities can be redimensionned on-demand to better suit our algorithm requirements. Besides, the communication costs which are induced show that oversizing the number of workers would lead to increasing the overall algorithm wall time because of these higher communication costs. Running Batch K-Means on Cloud Computing therefore introduces a new interesting question: what is the (optimal) amount of workers that minimize our algorithm wall time?\\

To answer this question, let us re-tailor the speedup formula detailed in Subsection \ref{section_InDepthMpiCosts}. As already explained in Subsection \ref{sec_CloudKMeans_Algorithm}, the symbol $M$ in the following will not refer to the total number of processing units but to the number of processing units that will run the Map tasks. The total amount of processing units could be lowered to this quantity, but as outlined in the same subsection, it would come at the cost of a slightly slower algorithm. As in the DMM case, the wall time of distance calculations remains unmodified:

\begin{align*}
T_{M}^{comp} & = \frac{(3NKd+NK+Nd)IT^{flop}}{M},\\
\end{align*}

where $N$, as in the DMM architecture case, stands for the total number of points in all the data sets gathered. In addition to the distance calculations, a second cost is introduced to model the wall time of the reassignment phase: the time to load the data set from the storage. Let us introduce $T_{Blob}^{read}$ (resp. $T_{Blob}^{write}$) that refers to the time needed by a given processing unit to download (resp. upload) a blob from (resp. to) the storage per memory unit. The cost to load the data set from the storage is then modeled by the following equation:

\begin{align*}
T_{M}^{Load} & = \frac{NdST_{Blob}^{read}}{M}.\\
\end{align*}

Let us keep in mind that by design this loading operation needs to be performed only once, even when $I>1$. This loading operation can be neglected in speedup model if $T_{M}^{Load} << T_{M}^{Comp}$. A sufficient condition for this to be true is:

\begin{align*}
\frac{ST_{Blob}^{read}}{3IKT^{flop}} << 1.
\end{align*}

This condition turns out to be true in almost all the cases. Let us provide a new wall time model of the recalculation phase performed by the two-step reduce architecture provided in Subsection \ref{sec_CloudKMeans_Algorithm}. This recalculation phase is composed of multiple communications between workers and storage as well as average computation of the different prototypes versions. More specifically, each iteration requires:

\begin{itemize}
\item $M$ mappers to write their version ``simultaneously'' in the storage;
\item each of the partial reducers to retrieve $\sqrt{M}$ prototypes versions;
\item each of the partial reducers to compute an average of the versions thus retrieved;
\item each of the partial reducers to write ``simultaneously'' its result in the storage;
\item the final reducer to retrieve the $\sqrt{M}$ partial reducer versions;
\item the final reducer to compute the shared version accordingly;
\item the final reducer to write the $\sqrt{M}$ shared versions in the storage;
\item all the mappers to retrieve the shared version.
\end{itemize}

The recalculation phase per iteration can therefore be modeled by the following equation:

\begin{align*}
T_{M}^{comm, per iteration} & = KdST_{Blob}^{write} + \sqrt{M}KdST_{Blob}^{read}  + 5(\sqrt{M})KdT^{flop}\\
& + KdsT_{Blob}^{read}  + KdST_{Blob}^{write}  + \sqrt{M}KdST_{Blob}^{read}\\
& + 5(\sqrt{M})KdT^{flop} + \sqrt{M}KdST_{Blob}^{write}. \\
\end{align*}

Keeping only the most significant terms, we get:

\begin{align*}
T_{M}^{comm} & \simeq I\sqrt{M}KdS(2T_{Blob}^{read} + T_{Blob}^{write}). \\
\end{align*}

More generally, we can show that a $p$-step reduce process leads to communication cost of the following form:

\begin{align*}
T_{M}^{comm, ~p-step} & \simeq I\sqrt[1/p]{M}KdS(pT_{Blob}^{read} + (p-1)T_{Blob}^{write}). \\
\end{align*}

In the context of our $2$-step reduce process, we can deduce an approximation of the speedup factor :

\begin{align}
\label{eq_CloudKMeansSpeedUp}
SpeedUp & = \frac{T_{1}^{comp}}{T_{M}^{comp} + T_{M}^{comm}}\\
& \simeq \frac{3IKNdT^{flop}}{\frac{3IKNdT^{flop}}{M} +I\sqrt{M}KdS(2T_{Blob}^{read} + T_{Blob}^{write}) } \\
& \simeq \frac{3NT^{flop}}{\frac{3NT^{flop}}{M} + \sqrt{M}S(2T_{Blob}^{read} + T_{Blob}^{write})}.
\end{align}\\

Using this model, which is kept simple, one can see there is an optimal number of workers to use.
This number ($M^{*}$) can be expressed as:\\

\begin{align}
\label{eq_MStar}
M^{*}  =  \sqrt[2/3]{\frac{6NT^{flop}}{S(2T_{Blob}^{read} + T_{Blob}^{write})}}.
\end{align}\\

This quantity must be compared to the optimal number of processing units in the DMM case expressed in equation \eqref{eq_MStar_DMM}. Let us remark that contrary to the DMM model, the best number of processing units in our cloud model does not scale linearly with the number of data points $N$. It directly follows that our cloud implementation suffers from a theoretical impossibility to provide an infinite scale-up with the two-step reduce architecture. Subsection \ref{sec_ScaleUp} investigates how our cloud-distributed Batch K-Means actually performs in terms of practical scale-up.\\

One can verify that for the previous value of $M^{*}$, $T_{M^{*}}^{comp} = \frac{1}{2} T_{M^{*}}^{comm}$, which means that when running the optimal number of processing units, the reassignment phase duration is half of the recalculation phase duration. In such a situation the efficiency of our implementation is rather low: with $M^{*} + \sqrt{M^{*}} + 1$ processing units used in the algorithm, we only get a speedup of $M^{*}/3$.

%\subsection{Read bandwidth}
%
%We pushed 8 blobs of 8 MB into the storage (this is the size of map results we are going to load back in reducers), and
%we measured the time spent to retrieve them. For a single worker using a single thread, we retrieved the blobs in
%7.79 seconds, implying a 8.21 MB/sec read bandwidth. We tried to use multiple threads on a single worker as
%advised by AzureScope to speedup the download process, and we found the best read bandwidth was obtained using
%5 threads: we retrieved the 8 blobs in 6,13 seconds, implying a 10.44 MB/sec read bandwidth. The multithreads
%read bandwidth we observed differs from what is achieved in AzureScope. This may be explained by two potential factors:
%we observed average bandwidth whereas
%AzureScope observed peak performances and we did not have the same experiment environment. Especially, all the parallel
%read requests were run on a single container instead of being distributed on several BlobStorage containers.
%
%This benchmark is very optimistic as opposed to our clustering experiments: the bandwidth we estimated was recorded while
%no other workers were reading or writing into the storage. Other parallel reads and writes can have two effects
%on the read bandwidth performance. Firstly, they can limit the bandwidth because of aggregated bandwidth boundaries
%(AzureScope is reporting 800 MB/sec bandwidth for reading operations). Secondly, they can increase the latency between
%a write completion in the blob storage and the availability of the blob to other workers. When running experiments
%with a large number of workers reading and writing in parallel, we sometimes experienced blobs already pushed into the storage
%becoming available only after 1 or 2 minutes. To use our model as a predictive tool to determine $P^{*}$, we used a value
%of bandwidth equals to 8 MB/sec, which is a very conservative assumption that leads us to underestimate communication costs.

%\subsection{Workers Flops performances}
%
%All our experiments were run on Azure small VM that are guaranteed to run on 1.6 GHz CPU. Yet because
%of virtualization we do not have any warranty in term of floating point operations speed. Therefore, to fit
%our predictive speedup model, we ran some map jobs (where the number of floating point operations is known)
%to determine how fast could our algorithm be run on these VM. The code was first run on an Intel Core
%2 Duo T7250 2*2GHz using only one core. We estimated our map jobs to be run at approximatively 750 MFlops
%on average. We then ran some of our map jobs in Azure and we get for the same code performance of 669 MFlops on
%the small VM.\\
%
%It is worth mentioning that with the number of workers we used (at max 160 mappers), we observed very little differences
%in processing time between workers. Flops performance may slightly move with time, but at a given time,
%workers performed at very similar speed.

%\subsection{Communication benchmark}
%
%We fixed K=1000, D = 1000 (so each version of the prototypes is 8MB big) and I =10. For different
%values of P, we run our clustering, replacing the processing part by waiting
%a fix period of time (15 seconds), so we could record communication time without being affected
%by straggler issues, as reported above. The following table provides wall time of the fake clustering
%(for 10 iterations), and the amount of time spent in communication (Wall Time - 10*15 seconds).
%
%\begin{center}\scriptsize
%\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
%P &5&10&20&30&40&50&60&70&80&90&100&110&120&130\\
%\hline
%Wall Time & 287 & 300 & 335 & 359 & 392 & 421& 434 & 468 & 479 & 509 & 533 & 697 & 591 & 620\\
%\hline
%Communication & 137 & 150 & 185 & 209 & 242 & 271& 284 & 318 & 329 & 359 & 383 & 547 & 441 & 470\\
%\end{tabular}
%\end{center}
%
%One can see communication is not free, especially for small values of P. Yet, communication costs does not
%move much as the number of worker grows, since we design our communication algorithm to limit this. Most of
%communication costs are due to aggregated read bandwidth boundaries while all workers are trying to read one copy
%of the shared prototypes at the same time, the final reducer write bandwidth boundary (this worker has to push
%into the storage $\sqrt{P}$ copies of the shared prototypes in parallel), queues latencies, and some blob temporary unavailability
%(when we stress the storage, a given blob already pushed can take as much as 2 minutes 30 before being available to download).

%\subsection{Azure expected performances}
%
%We will try to give an estimation of the speedup we can get using last algorithm version.
%Yet, the very nature of the cloud will make this estimation rather difficult to make :
%\begin{itemize}
%\item Bandwidth reading and writing on the storage introduce a lot of uncertainty in the estimation. Because it varies over time. But also because before loading up data or after downloading data we must serialize/deserialize them, compress/uncompress them, ... Therefore, bandwidth capacity is not exactly what expected.
%\item To download and upload data, using multi-threads improves a lot total throughput, as observed here \ref{AzureScope}. Throwing multiple threads (up to 30 as advised) comes with some little overhead.
%\item Workers are pinging queues to detect if there are messages to process. To limit the number of network pings, we schedule the pinging process only once per several seconds, introducing again uncertainty and latency.
%\item Virtual Machines upon workers might be shutdown or die, but work will still be executed once the VM is moved or reboot or another worker takes the job. Therefore, it is impossible to note that a VM has been unavailable for a part of the time, yet it will affect the result.
%\item Storage is sometimes unavailable for short periods of time. Retry policies are implemented on different layers of the abstractions so we do not have to  be careful. Yet, time to upload and download will be affected and we won't be able to notice, unless we build our algorithms on lower level of abstraction, increasing a lot the difficulty of the design.
%\item When running this algorithm, workers wait for the others to complete before running the next iteration. On the cloud, stragglers issue are stronger than on a DMM machine. Therefore, synchronization barrier will affect a lot the process, and time consumption will be affected by every bottleneck on every worker.
%\end{itemize}

%NUMBERS

\section{Experimental results}
\label{sec_Cloud_KMeans_Results}

\subsection{Azure base performances}
\label{sec_AzureBasePerformances}
In order to use our cost model as a predictive tool to determine the optimal number of workers $M^{*}$ and our implementation performance, one needs to evaluate the performances of Azure services. These performances have already been briefly analyzed in Subsection \ref{sec_AzureBenchmark}. We refer the reader to this subsection for more explanations and just recall here the recorded performances.\\

\begin{itemize}
\item \textit{BlobStorage Read Bandwidth:} 8MBytes/sec
\item \textit{BlobStorage Write Bandwidth:} 3MBytes/sec
\item \textit{CPU performance while performing distance calculations:} 670 MFlops
\end{itemize}

\subsection{The two-step reduce architecture benchmark}
\label{sec_AggregatedBandwidth}

The aggregated bandwidth refers to the maximum number of bytes the actual network can transfer per time unit. In most cases, such a quantity does not equal the product of maximal bandwidth per processing unit by the number of processing units, since the network may not sustain such maximal bandwidths for each worker when all of them are communicating at the same time. The aggregated bandwidth measurement is a difficult task that is hardware and task dependent. In this subsection we focus on a specific case raised by our clustering algorithm.\\

In the case of the two-step recalculation phase implementation developed in Subsection \ref{sec_CloudKMeans_Algorithm}, it is both difficult and inaccurate to theoretically determine the recalculation phase duration, partly because of the previously mentioned aggregated bandwidth bounds. We therefore produce a custom benchmark to evaluate the time spent in the recalculation phase as follows: the prototypes are designed as a data chunk of 8MBytes and the recalculation phase is run 10 times ($I =10$). For the different
values of $M$, the clustering implementation is run but the processing part is replaced by waiting for a fixed period of time (15 seconds), so that communication time can be recorded without being affected by straggler issues, as reported in Subsection \ref{sec_StragglerIssues}. The following table provides the wall time of this benchmark (for 10 iterations), and the amount of time spent in the recalculation phase (Wall Time - 10*15 seconds).

\begin{table}[!h]
    \centering
    \begin{tiny}
        \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
            \hline
            M &5&10&20&30&40&50&60&70&80\\
            \hline
            Wall Time (in sec) & 287 & 300 & 335 & 359 & 392 & 421& 434 & 468 & 479 \\
            \hline
            Communication (in sec)& 137 & 150 & 185 & 209 & 242 & 271& 284 & 318 & 329\\
            \hline
            $2T_{Blob}^{read} + T_{Blob}^{write}, (in ~10^{-7}sec/Byte)$ & 7.64 & 5.92 & 5.16 & 4.76 & 4.78 & 4.78 & 4.59 & 4.73 & 4.58\\
            \hline
        \end{tabular}

        \vspace{0.5cm}

        \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
            \hline
            M &90&100&110&120&130\\
            \hline
            Wall Time (in sec)& 509 & 533 & 697 & 591 & 620\\
            \hline
            Communication (in sec)& 359 & 383 & 547 & 441 & 470\\
            \hline
            $2T_{Blob}^{read} + T_{Blob}^{write}, (in ~10^{-7}sec/Byte)$ & 4.71 & 4.77 & 6.51 & 5.02 & 5.13\\
            \hline
            \end{tabular}
    \end{tiny}
    \caption{Evaluation of the communication throughput per machine and of the time spent in the recalculation phase for different number of communicating units $M$.}
    \label{table:azureThroughput}
\end{table}

%\begin{center}\tiny
%\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
%\hline
%M &5&10&20&30&40&50&60&70&80\\
%\hline
%Wall Time (in sec) & 287 & 300 & 335 & 359 & 392 & 421& 434 & 468 & 479 \\
%\hline
%Communication (in sec)& 137 & 150 & 185 & 209 & 242 & 271& 284 & 318 & 329\\
%\hline
%$2T_{Blob}^{read} + T_{Blob}^{write}, (in ~10^{-7}sec/Byte)$ & 7.64 & 5.92 & 5.16 & 4.76 & 4.78 & 4.78 & 4.59 & 4.73 & 4.58\\
%\hline
%\end{tabular}
%\end{center}
%
%\begin{center}\tiny
%\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
%\hline
%M &90&100&110&120&130\\
%\hline
%Wall Time (in sec)& 509 & 533 & 697 & 591 & 620\\
%\hline
%Communication (in sec)& 359 & 383 & 547 & 441 & 470\\
%\hline
%$2T_{Blob}^{read} + T_{Blob}^{write}, (in ~10^{-7}sec/Byte)$ & 4.71 & 4.77 & 6.51 & 5.02 & 5.13\\
%\hline
%\end{tabular}
%\end{center}

%\begin{center}\tiny
%\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
%\hline
%M &5&10&20&30&40&50&60&70&80&90&100&110&120&130\\
%\hline
%Wall Time (in sec)& 287 & 300 & 335 & 359 & 392 & 421& 434 & 468 & 479 & 509 & 533 & 697 & 591 & 620\\
%\hline
%Communication (in sec)& 137 & 150 & 185 & 209 & 242 & 271& 284 & 318 & 329 & 359 & 383 & 547 & 441 & 470\\
%\hline
%$T_{Blob}^{read} + T_{Blob}^{write}, (in 10^{-7}sec/Byte$ & 3.82 & 2.96 & 2.58 & 2.38 & 2.39 & 2.39& 2.29 & 2.37 & 2.29 & 2.36 & 2.39 & 3.25 & 2.51 & 2.57
%\hline
%\end{tabular}
%\end{center}

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.4, clip=true, trim = 0mm 25mm 0mm 25mm]{../../Pictures/Cloud/BandwidthReduce.pdf}
\end{center}
\caption{ Time to execute the Reduce phase per unit of memory ($2T_{Blob}^{read} + T_{Blob}^{write}$) in $~10^{-7}$sec/Byte in function of the number of communicating units.}
\label{fig:BandwidthReduce}
\end{figure}

First of all, one can notice that the quantity $2T_{Blob}^{read} + T_{Blob}^{write}$ does not grow with $M$ (at least for $M<100$), which proves that we do not suffer
in our experiment from aggregated bandwidth bounds before using 100 workers. Secondly, we can note that the obtained value is smaller than the value that would be obtained using the values provided in Subsection \ref{sec_AzureBasePerformances} (which corresponds to $5.83*10^{-7}$ sec/Byte): indeed, the parallelization of downloads and uploads in each machine (through multiple threads) reduces enough the communication to compensate the frictions introduced by our two-step design. Finally, the whole process is sometimes behaving much worse than expected (see the outlier $M=110$). The case $M=110$ has been re-launched 1 hour later, obtaining the value 4.85. Figure \ref{fig:BandwidthReduce} sums up the table and shows that aggregated bandwidth bounds are not hit before $100$ hundred processing units are used. Before this threshold, the quantity $2T_{Blob}^{read} + T_{Blob}^{write}$ remains constant.\\

\subsection{Experimental settings}
\label{sec_KMeans_Cloud_Settings}

In the following experiments, Batch K-Means are run on synthetic data. As explained in the introduction to this chapter, the Batch K-Means wall time depends on the data size but not on the actual data values, except for the number of iterations to convergence. Thus, the synthetic nature of the data has no impact on the conclusion that we draw. Besides, the synthetic nature
of our data has allowed us to easily modify parameters such as the dimension $d$ to highlight some results. The synthetic data are generated uniformly in the unit hypercube using the following settings: the dimension $d$ is set to $1000$ and the number of clusters $K$ is set to $1000$. The number of points in the total data set depends on the experiment. For speed-up experiments, the total data set is composed of $500,000$ data points (for a total size of 4 GBytes) that are evenly split among the multiple processing units. For scale-up experiments, the data set total number of points grows with the number of processing units in the scale-up experiments. In all our experiments, the algorithm is run for 10 iterations to get stable timing estimates.\\

Theoretically, K and d should have the same impact on map and reduce phases since Map and Reduce costs are supposed to be proportional to $Kd$. As a consequence, the speedup is theoretically agnostic to $K$ or $d$. Yet, since our model does not take latency into account, having very small values of $K$ and $d$ would lead to underestimate communication costs by neglecting latency. Provided $K$ and $d$ are kept big enough, our speedup/scale-up results do not depend on $K$ or $d$.\\

As explained in the introduction to the chapter, the distributed Batch K-Means produces for each iteration the very same results as what a sequential Batch K-Means would return. Thus, the following experiments only focus on the speedup provided by the parallelization and not the function loss improvements.\\

\subsection{Speedup}

In this subsection we report on the performances of the cloud implementation proposed in Subsection \ref{sec_CloudKMeans_Algorithm} in terms of speedup. In other words, we investigate whether the proposed implementation allows us to reduce the Batch K-Means execution wall time for a given data set. In addition, we compare the observed optimal speedup to the theoretical optimal speedup obtained by the equation \eqref{eq_CloudKMeansSpeedUp}. We report on the results of one out of multiple experiments that we have run, the other experiments having shown the same general patterns and qualitative conclusions.\\

The proposed implementation is tested using the settings described in Subsection \ref{sec_KMeans_Cloud_Settings}. The algorithm is run for 10 iterations to get stable timing estimates. Neglecting loading time and memory issues (a small instance has only 1.75 GBytes of memory), a sequential Batch K-Means implementation would use approximately 6 hours and 13 minutes to run the 10 iterations.\\

The following table reports on the total running time in seconds (including data loading) of the proposed implementation for different numbers of mappers ($M$). We also report on the speedup over the theoretical total running time, and the efficiency (speedup divided by the total number of processing units $M+\sqrt{M}+1$). The speedup as a function of $M$ is plotted in Figure \ref{fig_KMeansCloudScaleup1} (it is the curve with $N=500,000$).\\

%\begin{center}\scriptsize
%\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
%\hline
%M &10&50&60&70&80&90&95&100&110&120&130&140&150&160\\
%\hline
%Time & 2223 & 657 & 574 & 551 & 560 & 525& 521 & 539 & 544 & 544 & 574 & 603 & 605 & 674\\
%\hline
%SpeedUp & 10.0 & 34.1 & 39.0 & 40.6 & 40.0 & 42.6 & 43.0 & 41.5 & 41.2 & 41.2 & 39.0 & 37.1 & 37.0 & 33.2\\
%\hline
%Efficiency & 0.67 & 0.58 & 0.57 & 0.51 & 0.44 & 0.42 & 0.41 & 0.37 & 0.34 & 0.31 & 0.27 & 0.24 & 0.23 & 0.19\\
%\hline
%Theo. Eff. & 0.63 & 0.61 & 0.60 & 0.55 & 0.53 & 0.49 & 0.48 & 0.47 & 0.43 & 0.41 & 0.37 & 0.36 & 0.33 & 0.32\\
%\hline
%\end{tabular}
%\end{center}

\begin{table}[!h]
    \centering
    \begin{tiny}
        \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
            \hline
            M &10&50&60&70&80&90&95&100&110\\
            \hline
            Time & 2223 & 657 & 574 & 551 & 560 & 525& 521 & 539 & 544 \\
            \hline
            SpeedUp & 10.0 & 34.1 & 39.0 & 40.6 & 40.0 & 42.6 & 43.0 & 41.5 & 41.2 \\
            \hline
            Efficiency & 0.67 & 0.58 & 0.57 & 0.51 & 0.44 & 0.42 & 0.41 & 0.37 & 0.34 \\
            \hline
        \end{tabular}

        \vspace{0.5cm}

        \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
            \hline
            M &120&130&140&150&160\\
            \hline
            Time & 544 & 574 & 603 & 605 & 674\\
            \hline
            SpeedUp & 41.2 & 39.0 & 37.1 & 37.0 & 33.2\\
            \hline
            Efficiency & 0.31 & 0.27 & 0.24 & 0.23 & 0.19\\
            \hline
        \end{tabular}
    \end{tiny}
    \caption{Evaluation of our distributed K-Means speedup and efficiency for different number of processing units $M$.}
    \label{table:speedupAchieved1}
\end{table}

As expected, the total processing time is minimal for a specific value of $M$ referred to as $M^{*}_{eff}$ (here 95), for which the speedup (43.0) is comparable to $M^{*}_{eff}/3$ (as predicted by the model). With the values of $T^{flop}$ reported in Subsection \ref{sec_AzureBasePerformances} and the values of $2T_{Blob}^{read} + T_{Blob}^{write}$ evaluated in Subsection \ref{sec_AggregatedBandwidth}, the theoretical value of $M^{*}$ is 112.While the theoretical optimal number of workers $M^{*}$ slightly overestimates the actual optimal number of workers $M^{*}_{eff}$, the equation \eqref{eq_MStar} provides a good first estimate of this number before running the experiments. More specifically, once the values  $2T_{Blob}^{read} + T_{Blob}^{write}$ and $T^{flop}$ have been evaluated, our equations provide an a-priori tool to estimate what the optimal number of machines to use would be.\\


%
%As expected, the total processing time is minimal for a specific value of P (here 95), for which
%one third of the total time is spent in the map phase (as predicted by the model). One can notice that
%the speedup curve is relatively flat in a neighborhood of $P^{*}$. This value of $P^{*}$ is lower
%than the predicted value (170 for the chosen constants) as a consequence of the overestimation of
%the storage efficiency by the theoretical model. As explained above, our model made the assumption
%that bandwidth is scalable with the number of workers, and this is no more true when the number of workers
%is too high. For example, when running 160 workers, some workers spend one minute to load the dataset from
%the storage, and for 2 experiments we ran with more than 100 workers, one blob took more than 2 minutes to
%be retrieved by a reducer after being pushed by a mapper.\\

\subsection{Optimal number of processing units and scale-up}
\label{sec_ScaleUp}

Let us bear in mind that the scale-up is the ability to cope with more data in the same amount of time, provided that the number of processing units is increased accordingly. In our cloud Batch K-Means, the theoretical optimal number of mappers $M^{*}$ is not proportional to $N$ (see equation \eqref{eq_MStar}), contrary to the DMM/MPI model cost. As a consequence, our cloud version cannot hope to achieve linear scale-up.\\

As a consequence, the scale-up challenge is turned into minimizing growth of wall time as N grows, using $M^{*}(N)$ mappers. Theoretically, our model gives a processing cost ($T_{M}^{comp}$) proportional to $N/M$ and a communication cost ($T_{M}^{comm}$) proportional to $\sqrt{M}$. As a consequence, the algorithm execution total time ($T_{M}^{comp} + T_{M}^{comm}$) is proportional to $\sqrt[1/3]{N}$ and the optimal number of workers $M^{*}$ is proportional to $\sqrt[2/3]{N}$.\\

In the following experiment, the values of $K$ and $d$ are kept constant ($K=1000$ and $d=1000$). For various values of $N$, our implementation is run on different values of $M$ to determine the effective optimal values $M^{*}_{eff}$ for a given $N$.\\

\begin{table}[!h]
    \centering
    \begin{scriptsize}
        \begin{tabular}{|c|c|c|c|c|c|c|c|}
            \hline
             & N & $M^{*}_{eff}$ & $M^{*}$  & Wall Time & Sequential & Effective  & Theoretical \\
             &  &  &  &  & theoretic time & Speedup &  Speedup ($=\frac{M^{*}}{3}$)\\
            \hline
            Exp. 1 & 62500 & 27 & 28 & 264 & 2798 & 10.6 & 9.34\\
            \hline
            Exp. 2 & 125000 & 45 & 45 & 306 & 5597 & 18.29 & 14.84\\
            \hline
            Exp. 3 & 250000 & 78 & 71 & 384 & 11194 & 29.15 & 23.55\\
            \hline
            Exp. 4 & 500000 & 95 & 112 & 521 & 22388 & 43.0 & 37.40\\
            \hline
        \end{tabular}

    \end{scriptsize}
    \caption{Comparison between the effective optimal number of processing units $M^{*}_{eff}$ and the theoretical optimal number of processing units $M^{*}$ for different data set size.}
    \label{table:speedupAchieved1}
\end{table}


As expected, one can see that $M^{*}_{eff}(N)$ and $T_{M}^{comp} + T_{M}^{comm}$ do not grow as fast as $N$. Between the experiment 1 and the experiment 4, $N$ is multiplied by 8. Our theoretical model anticipates that $M^{*}_{eff}$ should grow accordingly by $8^{2/3}=4$. Indeed, $M^{*}_{eff}$ grows from 27 to 95 (that is a 3.51 ratio). In the same way, our model anticipates that the execution wall time should grow by $8^{1/3}=2$. Indeed, the execution wall time grows from 264 seconds to 521 seconds. Figure \ref{fig_KMeansCloudScaleup1} provides the detailed experiment results of the speedup obtained for multiple values of $N$ and $M$.\\

\begin{figure}
    \centering
    \includegraphics[scale=0.55]{../../texfiles/Nips2010Poster/Speedup.pdf}
    \caption{Charts of speedup performance curves for our cloud Batch K-Means implementation with different data set size. For a given size $N$, the speedup grows with the number of processing units until $M^{*}$, then the speedup slowly decreases.}
    \label{fig_KMeansCloudScaleup1}
\end{figure}

For our last experiment, we aim to achieve the nominal highest value possible for speedup. As explained in Subsection \ref{sec_DMM_Comments}, for a fixed number of mappers $M$, the best achievable speedup is obtained by filling the RAM of each machine with data so each machine is in charge of a heavy computation load. While the previous table and Figure \ref{fig_KMeansCloudScaleup1} show how the speedup grows with $N$ (using $M^{*}(N)$ mappers), Figure \ref{fig_KMeansCloudScaleup2} shows how the speedup grows with $M$ (using the highest value of N that do not oversize the RAM of our VM). For this experiment, we set $K=1000$, $d=1000$, and set $N$ in such a way that each mappers is given 50,000 data points $N=M* 50,000$\footnote{The value of $n=50,000$ corresponds to 400 MBytes in RAM, while the RAM of a small role instance is supposed to be 1.75GBytes. In theory, we could have therefore loaded much more our instances. In practice, when we run this experiment in 2010, the VM crashed when we used higher values for $n$}. The results are reported in Figure \ref{fig_KMeansCloudScaleup2}.\\

\begin{figure}
    \centering
    \includegraphics[scale=0.6]{../../texfiles/Nips2010Poster/Speedup2.pdf}
    \caption{Charts of speedup performance curves for our cloud Batch K-Means implementation with different number of processing units. For each value of $M$, the value of $N$ is set accordingly so that the processing units are heavy loaded with data and computations. When the number of processing units grows, the communication costs increase and the spread between the obtained speedup and the theoretical optimal speedup increases.}
    \label{fig_KMeansCloudScaleup2}
\end{figure}

Overall, the obtained performances are satisfactory and the predictive model provides reasonable estimates of the execution wall time and of the optimal number of processing units that need to be used. While there is room for improving our implementation, the latency issues might prevent resorting on a tree like $O(log(M))$ reducer as available in MPI without using direct inter-machines communications. Without native high performances API, communication aspects will probably remain a major concern in Azure implementations.\\

\subsection{Straggler issues}
\label{sec_StragglerIssues}

In this subsection we investigate the variability of CPU performances of the multiple processing units and show that some tasks are processed in amounts of time significantly higher than expected, a phenomenon referred to as stragglers (see e.g. \cite{MapReduce2004} or \cite{StragglerIssuesInMapReduce}).\\

In the first experiment, we run the same task ---that consists in a heavy distance calculations load (each task corresponds to a reassignment phase)--- 100 times. The task is expected to be run in 7 minutes. The results are provided in Figure \ref{fig_StragglerSingleDistribution}.\\

\begin{figure}
    \centering
    \includegraphics[scale=0.6]{../../Pictures/DistributionSingleWorker.png}
    \caption{Distribution of the processing time (in second) for multiple runs of the same computation task for a single VM. As expected, the distribution is concentrated around a specific value.}
    \label{fig_StragglerSingleDistribution}
\end{figure}

In the second experiment, the same task is run 10 times in a row by 85 workers. Each of the 850 records therefore consists in the same computation load, performed on processing units supposedly of the same performance (Azure small role instances). The same experiment has been run on 2 different days, on 2 different hours, on different workers, but the same following patterns have been observed. Figure \ref{fig_StragglerDistribution} provides the empirical distribution of these computation durations.\\

\begin{figure}
    \centering
    \includegraphics[scale=0.6]{../../Pictures/Straggler.png}
    \caption{Distribution of the processing time (in second) for multiple runs of the same computation task for multiple VM. One can note the ``3 modes'' distribution and outliers (tasks run in much more time).}
    \label{fig_StragglerDistribution}
\end{figure}

From this experiment, we can deduce that:

\begin{itemize}
\item \textit{The 3-mode distribution} : 90\% of the tasks are completed between 390 and 500 seconds. For the tasks which have been completed in this interval, we observe 3 different
modes of our empirical distribution.The 3 modes may be due to hardware heterogeneity or multiple VM hosted on the same physical machine.\\

\item \textit{A worker can be affected by temporary slowness} : The three longest runs (823 seconds, 632 seconds, 778 seconds) have been performed by the same VM, which has also performed very well on other iterations: (360 seconds, 364 seconds, ...). This could be explained by very different reasons, such as the fact the physical machine hosting our VM has been hosting temporarily another VM, a temporary downsizing of the size of cache memory for our VM, etc.\\
\end{itemize}

Straggler issues have already been pointed out, for example in the original MapReduce article (\cite{MapReduce2004}) by Google, yet they were observed while running thousands of machines. We show that straggler issues are also observed on a pool of workers as small as 100 VM. \cite{MapReduce2004} describes a monitoring framework to detect tasks taking too much time, and uses backup workers to re-launch tasks that have been detected to be too long. Yet, this approach leads to wait for the standard duration of the task before detecting straggler tasks and launching again the corresponding tasks. This situation severely limits the potential speedup that can be achieved.\\

As pointed out by Graphlab in \cite{Graphlab2010}, the easy access of MapReduce frameworks and the great help it provides to design scalable applications has driven part of the
machine-learning community to think their algorithms to fit in a MapReduce framework. However, in many cases the combination of stragglers and of a synchronous framework like MapReduce prevents users from obtaining good overall speedups.\\


%Instead of trying to optimize $M$ for a given value of (N, D, K) we set the
%number of workers to $M$, and try to provide the best possible speedup, namely $M$. In
%the following experiments, we set K=1000, D=1000, and set N to 50 000*P. Thus,
%each worker will be working on 50 000 points, which represent an important part
%of the RAM of the virtual machine (VM) (around 400 MB), helping to achieve best possible speedups by
%giving heavy computation loads on each worker. The algorithm is run for 10 iterations
%to get stable timing estimates (more iterations would be run on a real K-Means on such a heavy data set).
%
%The following charts report scaleup and speedUp...
%
%Nevertheless, the obtained performances are very satisfactory for a reasonable
%number of mapping processing units (around 60 for this data size). While there
%is room for improving our implementation, the latency issues might prevent
%resorting on a tree like $O(log(M))$ reducer as available in MPI. Without
%native high performances API, communication aspects will probably remain a
%major concern in cloud implementations.

%
%We tested the performances of the proposed implementation, as well as the
%limits of the theoretical cost model on synthetic data generated uniformly in
%the unit hypercube (as pointed out above, timings are independent of the
%actual data values). We report the results of one out of several benchmarks
%that show the same general trends. The dataset consists of $N=500 000$
%observations in dimension $D=1000$ (4 GB). We look for $K=1000$ clusters. The
%algorithm is run for 10 iterations to get stable timing estimates. The
%observed floating point performances of our code on a small instance worker in
%Azure is 669 Mflop/s. The bandwidth of a single threaded worker is about
%8 MB/s. Neglecting loading time and memory issues (a small instance has only 1
%GB of memory), a sequential implementation would use approximately 6 hours and
%13 minutes to run the 10 iterations.
%
%The following table reports the total running time in seconds (including data
%loading) of the proposed implementation for different numbers of mapping PU
%($P$). We report the speedup over the theoretical total running time, the
%efficiency (speedup divided by $P+\sqrt{P}+1$, the total number of PU) and
%the theoretical efficiency predicted by the model.
%\begin{center}\scriptsize
%\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
%P &10&50&60&70&80&90&95&100&110&120&130&140&150&160\\
%\hline
%Time & 2223 & 657 & 574 & 551 & 560 & 525& 521 & 539 & 544 & 544 & 574 & 603 & 605 & 674\\
%\hline
%SpeedUp & 10.0 & 34.1 & 39.0 & 40.6 & 40.0 & 42.6 & 43.0 & 41.5 & 41.2 & 41.2 & 39.0 & 37.1 & 37.0 & 33.2\\
%\hline
%Efficiency & 0.67 & 0.58 & 0.57 & 0.51 & 0.44 & 0.42 & 0.41 & 0.37 & 0.34 & 0.31 & 0.27 & 0.24 & 0.23 & 0.19\\
%\hline
%Theo. Eff. & 0.63 & 0.61 & 0.60 & 0.55 & 0.53 & 0.49 & 0.48 & 0.47 & 0.43 & 0.41 & 0.37 & 0.36 & 0.33 & 0.32\\
%\end{tabular}
%\end{center}
%As expected, the total processing time is minimal for a specific value of $P$
%(here 95), for which one third of the total time is spent in the map phase (as
%predicted by the model). This is lower than the predicted value (170 for the
%chosen constants) as a consequence of the overestimation of the storage efficiency by
%the theoretical model. Further investigations are needed to get a more
%accurate cost model for large $P$. In particular, we need both to take into
%account latency and bandwidth aggregation issues. Indeed, Microsoft reports
%that the total aggregated bandwidth of a PU group tops at 800 MB/s: in our
%model, $B$ is independent of $P$, which underestimates the reduce cost for
%large values of $P$. In addition, we have observed a large latency between a
%write completion to the BlobStorage by a worker and the availability of the
%blob to other workers (up to two minutes in some cases). This latency should
%be modeled and taken into account to predict the optimal $P^*$.
%
%Nevertheless, the obtained performances are very satisfactory for a reasonable
%number of mapping processing units (around 60 for this data size). While there
%is room for improving our implementation, the latency issues might prevent
%resorting on a tree like $O(log(P))$ reducer as available in MPI. Without
%native high performances API, communication aspects will probably remain a
%major concern in cloud implementations.

%\subsection{Price}
%
%Without any commitment and package, 1 hour of CPU on small VM is charged 0.1 dollar and 1000000
%transactions between workers and storage (QueueStorage or BlobStorage) are charged 1 dollar.
%Therefore, one clustering experiment with 10 workers (decreasing wall time from 6 hours to 37 minutes)
%is charged less than 2 dollars, and our biggest experiment running 175 workers is charged less than
%20 dollars. Since our experiments are not one hour long but 10 minutes long at worst, if one can recycle
%the 50 other minutes running other computations, then the cost of our biggest experiment drops to 4 dollars
%on average. For bigger uses, some packages are available to decrease charges.
