\documentclass{esannV2}
\usepackage{graphicx}
\usepackage[utf8]{inputenc}
\usepackage{amssymb,amsmath,array,url}
\usepackage{multicol}

%***********************************************************************
\voffset 0 cm \hoffset 0 cm \addtolength{\textwidth}{0cm}
\addtolength{\textheight}{0cm}\addtolength{\leftmargin}{0cm}


\newcommand{\Rb}[1]{\ensuremath{\mathbb{R}^{#1}}}

\graphicspath{{././Pictures/Logos/ }{././Pictures/MapReduce/}}

\begin{document}
%style file for ESANN manuscripts
\title{Communication and straggler challenges on cloud K-Means}

%***********************************************************************
% AUTHORS INFORMATION AREA
%***********************************************************************
\author{Matthieu Durut$^1$ and Fabrice Rossi$^2$
%
% DO NOT MODIFY THE FOLLOWING '\vspace' ARGUMENT
\vspace{.3cm}\\
%
% Addresses and institutions (remove "1- " in case of a single institution)
1- Lokad, 70, rue Lemercier, 75018 Paris -- France\\
%
% Remove the next three lines in case of a single institution
\vspace{.1cm}\\
2- BILab, T\'el\'ecom ParisTech, LTCI - UMR CNRS 5141 \\
46, rue Barrault, 75013 Paris -- France
}

\maketitle

\begin{abstract}
This paper studies how parallel machine learning algorithms can be implemented
on top of Microsoft Windows Azure cloud computing platform. More specifically,
we exhibit straggler issues even on a small number of workers
(less than 100), and provide communication and synchronization design pattern considerations
for achieving good speedups and scaleups on cloud machine learning applications.
\end{abstract}

\section{Parallel K-Means}
This paper uses the K-Means clustering algorithm as a typical example of
machine learning methods. This choice is mainly motivated by the fact that
apart for the number of iterations to convergence, the processing time of K-Means
is known before running, and only depends on the data
dimensions and on $K$, rather than on the actual data values: timing results
obtained on simulated data then apply to any dataset with the same
dimensions. K-Means is easy to parallelize on shared memory
computers and on local clusters of workstations: numerous publications (see
e.g., \cite{DhillonModha2000}) report linear speedup up to at least 16
processing units (PU, which can be CPU cores or workstations).

Let us recall the principles of Dhillon and Modha's parallel K-Means
\cite{DhillonModha2000}. K-Means consists in alternating two phases (after
proper initialization). In the assignment phase, the algorithm computes the
Euclidean distance between each of the $N$ data points $X_i\in \Rb{D}$ and each
of the $K$ cluster prototypes $m_j\in \Rb{D}$. Each $X_i$ is assigned to its
closest prototype, denoted $m_{k_i}$. In the recalculation phase, each
prototype is recomputed as the average of the data points assigned to it in
the previous phase. Since distance calculations are intrinsically parallel,
it is therefore natural to split the computational load by allocating disjoint
subsets of $N/P$ data points to $P$ PU. When all the PU, called mappers following
\cite{MapReduce2004} terminology, have completed their assignment
phase and computed their local prototypes version, reducers merge the local prototypes
version into a new aggregated version, which is broadcasted back to all the mappers.
In \cite{DhillonModha2000} this is done via a dedicated efficient ``reducing'' function of the MPI library
which uses $\log P$ parallel rounds of communications between
the PU, resulting in a cost of $O(KDlog(P))$.

The cloud implementation studied in this paper is based on the parallel
K-Means described above. The main difficulty consists in implementing
synchronization and communication between the PU that won't affect too much
performances, using the facilities provided by Windows Azure cloud operating system.

\section{Microsoft Windows Azure}
Windows Azure Platform is Microsoft's cloud computing solution, in the form of
Platform as a Service (PaaS). The underlying cloud operating system (Windows
Azure) provides services hosting and scalability. It is composed of a storage
system (that includes Blob Storage and Queue Storage) and of a
processing system (that includes \emph{web roles} and \emph{worker
  roles}). The Azure components we used are described below.

The architecture of an Azure hosted application is based on two components:
\textbf{web roles} and \textbf{worker roles}. Web roles are designed for web
application programming and do not concern the present article. Worker roles
are designed to run general background processing. Each worker role typically
gathers several \emph{cloud services} and uses many \emph{workers} (Azure's
processing units) to execute them. Our prototype uses only one worker role,
several services and tens of PU. The computing power is provided by the
workers, while Azure storage system is used to implement synchronization and
communication between workers. It must be noted indeed that Azure does not
offer currently any standard API for distributed computation, neither a low
level one such as MPI \cite{MPI1996}, nor a more high level one such as Map
Reduce \cite{MapReduce2004} or Dryad \cite{Dryad2007}. Map reduce could be
implemented using Azure components (following the strategy of
\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{Graphlab2010},
those high level API might be inappropriate for iterative machine learning
algorithms such as the K-Means. We rely therefore directly on Azure queues and
blob storage.

\textbf{Azure Queues} provide a message delivery mechanism through distributed
queues. Queues are designed to store a large amount of small job messages.
Using queues to communicate helps building
loosely coupled components and mitigates the impact of individual component
failure. Messages stored in a
queue are guaranteed to be returned at least once, but possibly several times:
this requires one to design idempotent jobs. If a worker fails to complete a job (because it
throws some exception or because the worker dies), the message is requeued after a certain
period of time. Through this process, one can make sure no job is lost because of e.g., a hardware failure.

\textbf{Azure Blob Storage} enables applications to store large objects, up to
50 GB each. Blobs are composed of a
string (that is used as a key to store the value), a value composed of a
binary object, and a timestamp (etag) (that indicates the last write on this
blob). In addition of Get and Put methods, blobs whose key shares a given
prefix can be listed. Optimistic non locking atomic read-modify-write
operations can be implemented using a timestamp matching condition: a write
succeed if and only if the timestamp of the storage matches the one provided
by the write operation.

Our prototype uses
\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}}, an
open-source framework that adds a small abstraction layer to ease Azure
workers startup and life cycle management, and storage access.

\section{Proposed implementation}
Our prototype consists in three cloud services (setup, map and reduce
services), each one matching a specific need. A queue is associated to each
service: it contains messages specifying the storage location of the data
needed for the jobs. Workers regularly ping the queues
to acquire a message. Once it has acquired a message, a worker starts running
the service related to the queue where the message was stored, and the message
becomes invisible till the job is completed or timeouts. Overall, we use
$P+\sqrt{P}+1$ processing units in the services described below.\\

\includegraphics[height=25em]{DistributedKMeans.pdf}

The \textbf{SetUp Service} first generates the original shared
prototypes and pushes them on $\lceil\sqrt{P}\rceil$ different places in the
BlobStorage (this way, we avoid contention while all mappers are trying to read on
these shared prototypes). SetUp Service also generates P split datasets of N/P
points in each and also put them into the BlobStorage. Once completed, it launches
Map Service.

When executing a \textbf{Map Service} job, a worker first downloads the
dataset it is in charged of (once for all). Then the mapper loads one of the copy of the
initial shared prototypes and starts the computation
step of attributing each point in the dataset to the closest prototype. New local prototypes
version is build on the fly as the points are being processed. When completed, the mapper pushes
its local prototypes version into the storage according to the following addressing rule:
iteration/groupId/jobId.

Each \textbf{Reduce Service} worker is in charged of listing and loading as soon as they become available the map results within a given iteration/groupId directory until it gets all the expected results within this directory. The reducer then merges the different prototypes version, and pushes the merged result into the storage in the partial reduce directory.
One last reducer runs the same process on the partial reduce results
directory. Once the $\lceil\sqrt{P}\rceil$ partial reduce results are retrieved, the last
reducer builds a new shared version out of the partial reduce results and pushes a copy of it into each of the  $\lceil\sqrt{P}\rceil$ places in the shared prototypes directory. After several seconds, every mapper
has been pinging and loading one copy of the now available shared result blobs, and the map step can be
run again.

%When all the map jobs have been
%completed, reducers must be notified to aggregate the results. We considered
%two designs for this synchronization process.
%
%A first solution consists in using the atomic read-modify-write operation to
%store a shared counter in the BlobStorage. Each worker increases the counter
%once it has completed its iteration. When the counter reaches P, reduction is
%started. A second possibility uses directly the reduce service. It queries on a
%regular basis the BlobStorage for the results of the Map Service. When it
%finds the required results, the reduction is started.
%
%We chose to use the second solution for several reasons. Firstly, this
%solution is compatible with multiple execution of the same map service job: if
%a map job is run twice, then the result is written twice at the same place and
%it does not affect synchronization. Secondly, it could help reducing
%straggler issues by overlapping latest computations with retrieval of first
%available results. Thirdly, it handles both synchronization and communication
%between mappers and reducers. Fourthly, this design helps a lot while dealing with
%straggler issues, be these jobs being discarded or relaunched. Finally, it proved
%to be much faster when the number of workers reaches between 50 and 100. Indeed, optimistic non locking
%read-modify-write schemes are not adapted to high contention situations: for
%instance, we observed counter updating times up to 20 seconds with 70 workers.
%
%In practice, when all the points of a map service job have been processed
%once, the local prototypes version and their weights are pushed into the
%storage according to the following addressing rule: iteration/groupId/jobId.
%The worker then starts waiting, pinging one fixed copy of current
%iteration shared prototypes until the copy becomes available.



\section{Straggler issues and impact on performance}

We first set an experiment with 100 workers, composed of 85 mappers, 11 reducers
(=$\lceil\sqrt{85}\rceil+1$), and 4 backup workers (in case some workers would
be unavailable). Each mapper performed the same task 10 times, taking about 7 minutes.
No communication between workers were used here, we only record the empirical distribution
of the computation time of the 850 tasks run, as reported in the following graphic :

 \begin{tabular}{cc}
       \includegraphics[height=0.4\linewidth]{Straggler.png} &
       \includegraphics[height=0.4\linewidth]{DistributionSingleWorker.png}
      \end{tabular}\\
%\includegraphics[height=12em]{Straggler.png}

The same experiment has been run on 2 different days, on 2 different hours,
on different workers, and has returned the same pattern.
Therefore, we can deduce 2 points :

\begin{itemize}
\item \textit{The 3 modes distribution} : 90\% of the runs are completed between
390 and 500 seconds. For the runs completed in this interval, we observe 3 different
modes of our empirical distribution. To investigate whether this is due to heterogeneity
in the hardware or to software reasons, we launch a long run on a single machine : 100 iterations
of 7 minutes each, to determine what was the behaviour of a single machine. Therefore, the 3 modes
may be due to hardware heterogeneity or multiple VM hosted on the same physical machine.
\item \textit{A worker can be affected by temporary slowness} : The three
longest runs (823 seconds, 632 seconds, 778 seconds) have been performed by the
same VM, which has also performed very well on other iterations :
(360 seconds, 364 seconds, ...). This could be explained by very different reasons,
such as the fact the physical machine hosting our VM has been hosting temporarily
another VM, a temporary downsizing of the size
of cache memory for our VM, ...
\end{itemize}

Straggler issues have already been pointed out, for example in the original MapReduce
article \cite{MapReduce2004} by Google, yet they were observed while running thousands of
machines. We show that straggler issues are also observed on
as such a small pool of workers as 100 VM. \cite{MapReduce2004} describes a monitoring framework to detect tasks taking too much
time, and use backup workers to relaunch tasks
that has been detected to be too long. Yet, this approach leads to wait for the standard
duration of the task before detecting straggler tasks and launching again the corresponding
jobs. Therefore, for a given number of PU P, we cannot achieve better speed-up than P/2 using
\cite{MapReduce2004} approach on straggler issues.

As pointed out by Graphlab \cite{Graphlab2010}, the ease of access of MapReduce frameworks
and the great help it provides to design scalable applications has driven part of the
Machine Learning community to think their algorithms to fit in a MapReduce framework. GraphLab
shows MapReduce is sometimes not a suitable framework for some specific Machine Learning
algorithm because of its lack of expressivity. The way the original MapReduce framework deals with
staggler issues is very well suited for very large batch jobs, as crawling the web, yet it
might not be the most efficient framework for machine learning on 100 PU.

\section{Communication benchmark}

We fixed K=1000, D = 1000 (so each version of the prototypes is 8MB big) and I =10. For different
values of P, we run our clustering, replacing the processing part by waiting
a fix period of time (15 seconds), so we could record communication time without being affected
by straggler issues, as reported above. The following table provides wall time of the fake clustering
(for 10 iterations), and the amount of time spent in communication (Wall Time - 10*15 seconds).

\begin{center}\scriptsize
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
P &5&10&20&30&40&50&60&70&80&90&100&110&120&130\\
\hline
Wall Time & 287 & 300 & 335 & 359 & 392 & 421& 434 & 468 & 479 & 509 & 533 & 697 & 591 & 620\\
\hline
Communication & 137 & 150 & 185 & 209 & 242 & 271& 284 & 318 & 329 & 359 & 383 & 547 & 441 & 470\\
\end{tabular}
\end{center}

One can see communication is not free, especially for small values of P. Yet, communication costs does not
move much as the number of worker grows, since we design our communication algorithm to limit this. Most of
communication costs are due to aggregated read bandwidth boundaries while all workers are trying to read one copy
of the shared prototypes at the same time, the final reducer write bandwidth boundary (this worker has to push
into the storage $\sqrt{P}$ copies of the shared prototypes in parallel), queues latencies, and some blob temporary unavailability
(when we stress the storage, a given blob already pushed can take as much as 2 minutes 30 before being available to download).

\section{Scale-up results}

In a previous work to be published, we show that for given values of (N, K, D, I),
the implementation proposed above had a computation cost of $\frac{3NDKF}{P}$ per
iteration, and a communication cost of $2\sqrt{P}KDB$ per iteration. We therefore
deduce that the optimal number of workers was $P^*=(3NF/B)^{2/3}$, and that 2 third
of the wall time was spent in communication, and one third in computation, when using
$P^*$ workers.\\

Instead of trying to optimize P for a given value of (N, D, K) we set the
number of workers to P, and try to provide the best possible speedup, namely P. In
the following experiments, we set K=1000, D=1000, and set N to 50 000*P. Thus,
each worker will be working on 50 000 points, which represent an important part
of the RAM of the virtual machine (VM) (around 400 MB), helping to achieve best possible speed-ups by
giving heavy computation loads on each worker. The algorithm is run for 10 iterations
to get stable timing estimates (more iterations would be run on a real K-Means on such a heavy dataset).

The following charts report scaleup and speedUp...

Nevertheless, the obtained performances are very satisfactory for a reasonable
number of mapping processing units (around 60 for this data size). While there
is room for improving our implementation, the latency issues might prevent
resorting on a tree like $O(log(P))$ reducer as available in MPI. Without
native high performances API, communication aspects will probably remain a
major concern in cloud implementations.

\section{Conclusions and future works}



\small
\bibliographystyle{abbrv}
\bibliography{cloud-kmeans}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% LaTeX-command: "pdflatex"
%%% ispell-dictionary: "american"
%%% End:

