\section{Introduction}

The choice issue of synchronism or asynchronism has already been presented in subsection \ref{sec_SynchronismAsynchronism} as an expression of the natural tradeoff between simplicity and efficiency (because of the stragglers mitigation). The resort to asynchronism for cloud machine-learning algorithms therefore seems to be an important mechanism to improve the overall algorithms efficiency when tasks are coarse-grained. In the context of Cloud Computing platforms, the communication costs prompt us to design coarse-grained tasks, and therefore to investigate whether the asynchronism may be appropriate.\\

More generally than the stragglers mitigation, the asynchronism choice seems to improve some algorithms efficiency as it incarnates a strategy of full usage of all the resources available. One can easily notice that distributed algorithms will benefit from bandwidth throughput improvement, CPU cores load increase or apparent latency reductions. As a result, the asynchronism may improve the usage ratio of all the available hardware components and by consequence the algorithm behaviors by allowing overlaps of communication and computation.\\

The interest of asynchronism in scientific computations has been already investigated on multiple platforms: asynchronous algorithms have been analyzed on GPU hardware (see e.g. \cite{AsynchronismGPU}), on heterogeneous clusters (see e.g. \cite{AsynchronousHeterogeneousClusters}) or on a multi-cores machine (see e.g. \cite{AsynchronousMultiThreads}). Many results on the theoretical convergence of asynchronous stochastic gradient optimization algorithms have also been proved, especially by Tsitsiklis (see e.g. \cite{TSI1}) or more recently by Patra in \cite{PatraDALVQ}. To our knowledge, we provide in this chapter the first experimental results of asynchronous gradient-descent algorithms implemented on a Cloud Computing platform.\\

Two main difficulties arise from distributed asynchronous clustering algorithms. The first one is intrinsic to asynchronous algorithms, and mainly comes from the randomness thus introduced: a distributed asynchronous algorithm is not only producing different results than its sequential counterpart, but its returned result is also non-constant, making the experiments hardly reproducible, contrary to Batch K-Means as outlined for example in \cite{BatchVsOnline2}. In addition to this first difficulty, a specific challenge of our clustering problem consists in the non-convexity of our loss function (see equation \ref{eq:EmpiricalDistortion}). This non-convexity leads to a supplementary difficulty: the natural update mechanism, suggested for example by \cite{TSI1} or \cite{PatraDALVQ} and tested in the fist parallelization scheme of chapter \ref{chap:practicalDALVQ} does not provide a speedup.\\

The algorithm used in this chapter therefore relies on an asynchronous version of the latter parallelization schemes of online gradient descent developed in chapter \ref{chap:practicalDALVQ}. It is the adoption of an online paradigm that allows the introduction of the asynchronism. In addition, online gradient descent algorithms are deemed to provide faster rates of convergence than their batch counterparts (see e.g. \cite{BatchVsOnline}). This \textit{a-priori} is confirmed in the end of this chapter: for the clustering context described in section \ref{sec:fightbatch}, our distributed asynchronous Vector Quantization (DAVQ) algorithm significantly outperforms Batch K-Means.\\

This good performance must however be assessed in terms of the development cost of our DAVQ prototype: as introduced in the introduction of this document, the DAVQ project\footnote{\texttt{http://code.google.com/p/clouddalvq/}} is a shared project with Benoit Patra. While we were already used to Azure application development, the development of this project took us months; moreover, while the distributed Batch K-Means experiments have required hardly more than 10.000 hours of CPU, the DAVQ project experiments took us more than 110.000 hours. In addition to this very costly development, the DAVQ algorithm proved to be bound in speedup: beyond a certain limit defined by the clustering context, an excess of workers lead to unstable results (see the case $M=16$ in figure \ref{fig:onelayer} or the case $M=64$ in figure \ref{fig:twolayer}).\\

Let us now briefly describe our DAVQ prototype. Like our distributed Batch K-Means algorithm, it is built on top of Azure. As already explained, our cloud DAVQ implementation makes a great use of the results obtain in Chapter \ref{chap:practicalDALVQ} where parallel implementations of DAVQ have been tested and discussed in general (i.e. independently of the distributed architecture). Especially, the update mechanism based on communication of the displacement terms is implemented.  The DVQ models in chapter \ref{chap:practicalDALVQ} introduced non negligible delays that will picture essential aspects of communication in Cloud Computing platforms. Our DAVQ prototype is an opensource project released under the new BSD license and written in C\#/.NET. The project is also built using the opensource project Lokad.Cloud\footnote{\texttt{http://lokad.github.com/lokad-cloud/}} which provides an O/C (Object to Cloud) mapper and an execution framework that leverages low level technicalities of Windows Azure.\\

This chapter is organized as follows. Section \ref{sec:cloudDALVQImpl} focuses on the software architecture of our DAVQ algorithm and many connections are drawn with Chapter \ref{chap:practicalDALVQ}. The study of the scalability of the DAVQ technique is presented in section \ref{sec:scalabilityCloudDALVQ}. The experiments are similar in spirit to those made on section \ref{sec:parallelGradient}. Precisely, we investigate the scalable gain of time execution brought by DAVQ algorithms compared to a single execution of the VQ method. Finally, the last section \ref{sec:fightbatch} is a benchmark between our DAVQ implementation and the distributed implementation of Batch K-Means developed in chapter \ref{chap:KMeans}.
%
%\section{Overview of Cloud Computing}\label{sec:cloudOverview}
%Cloud Computing is the emerging paradigm of computing as a utility. This term is fuzzy and might refer to very different things. From our point of view the best definition has been provided by Armbrust et al.~in \cite{ARM1}: Cloud Computing \emph{refers to both the applications delivered as services over the Internet and the hardware and systems software in the datacenters that provide those services}. In this chapter we will only consider the pay-as-you-go services with user shared resources sometimes gathered in the terminology Public Cloud as opposed to Private Cloud which stands for an infrastructure dedicated to the IT of a single organization. In the following, a cloud platform (or cloud for short) will denote the sum of the hardware and software of a Public Cloud solution. The main idea under the concept of cloud is the elasticity of (appearing) infinite computing and storage resources quickly available on demand.\\
%
%The phrase ``software as a service'' (SaaS), as explained in Vecchiola et al.~\cite{VEC1} refers to \emph{solutions that are at the top end of the Cloud Computing stack and they provide end users with an integrated service comprising hardware, development platforms, and applications}. For example the popular Google's webmail \texttt{Gmail} is based on a SaaS model. The application is accessible for the user through the internet as opposed to traditional desktop applications (e.g.~\texttt{MS Excel 2010}). At a much low-level computer vision we have the ``infrastructure as a service'' (IaaS) which provides low-level hardware and the users may control a large part of the software environment. However, the automation of resource provisioning is not easy due to few infrastructure abstractions. \texttt{Amazon EC2} is one of the most popular IaaS offer on the market. The third area of ``as a service'' paradigm is platform as a service (PaaS) that can be thought as an intermediate abstraction level between IaaS and SaaS. As defined by Vecchiola et al.~in \cite{VEC1} a PaaS solution provides \emph{an application or development platform in which users can create their own application that will run on the cloud}. A PaaS solution comes with a framework and a set of APIs to build the application for the cloud. It also comes with a portal that will help the developer to run the application on top of a fully abstracted infrastructure. In both PaaS and IaaS approaches the elasticity of resources is guaranteed by a fine use of virtualization tools which help to perform quick replications of virtual machines (abbreviated in VMs, see for instance Craig \cite{CRA1}). This aspect is therefore much more abstracted in a PaaS offer. The main actors proposing PaaS solutions are \texttt{Google AppEngine} and \texttt{Microsoft Windows Azure}.\\
%
%Cloud Computing solutions have a deep impact on economic models. The pay-as-you-go concept brings to a company the opportunity to pay only for what it uses. As an example, it can allow $100$ servers for few hours to respond to a peak traffic on their web application and one small server after that. Indeed, the cost of $100$ hours of CPU with one machine is the same than using $100$ machines for one hour. The Cloud Computing short-term elastic ability of downsizing and upsizing resources available will make the risk of underprovisioning disappear and will reduce the costs due to overprovisioning. Amazon CEO Jeff Bezos in \cite{BEZ1} tells the story of Animoto, a company that used intensively the elasticity of Amazon EC2. Their resource needs increased in three days from $50$ \texttt{Amazon EC2} instances to $3500$. This type of problem is also encountered in the case of the software company Lokad which delivers various forecasting SaaS applications (\texttt{Salescast}, \texttt{Callcast}, etc.) for business intelligence. Lokad's forecasting engine is deployed on top of \texttt{Microsoft Azure} (PaaS) and makes complex machine-learning computations which require a lot of computing resources. Furthermore Lokad is bound by the fact to deliver its forecasts within an hour. Therefore some peak of resources consumption naturally appear in rush hours. The automated provisioning service adapts the resources by deploying hundreds of VMs instances in few minutes.\\
%
%Scientific computing may also make a great use of Cloud Computing paradigm. Indeed, scientists need to perform large scale experiments requiring a large number of computing resources which can be easily provided by Cloud Computing platforms. Usually, they use super computers or clusters to run such research works. However, these solutions are inevitably linked to massive costs of hardware and maintenance. Such approaches are also difficult to setup and operate for a physicist or a mathematician who is not a informatics engineer. The Cloud Computing paradigm offers a new model for the scientist community. Indeed, the storage and computing resources are dynamically provisioned as needed and the experimentation costs could be decreased significantly. \texttt{CloudDALVQ} is an example of a scientific project for large scale experiments based on a PaaS cloud platform. Its goal consists in testing some new quantization algorithms highly scalable.
%
%\section{\texttt{Windows Azure} and \texttt{Lokad.Cloud}}\label{sec:Azure}
%\subsection{Services, queues and workers}
%\texttt{Microsoft Windows Azure} (or simply \texttt{Azure}) is a cloud offer classified as Platform As A Service (PaaS, see Section \ref{sec:cloudOverview} above). It provides elastic computing resources to deploy internet-scale applications. The basic computing units provided by \texttt{Azure} are Virtual Machines hosting instances of roles. There are two types of role. Firstly, the web role acting as a frontend for web applications. Secondly, the worker role used for intensive background computing tasks whose instances can be deployed on many workers. The \texttt{CloudDALVQ} project uses only one web role monitoring the application and one worker role with tens of workers. In order to simplify the discussion, VMs and workers can be assimilated. \texttt{Azure} provides several types of worker instances as shown in Table \ref{table:azureTarifs}. In all our experiments, we used \texttt{Small} computing instances.\\
%
%\begin{table}[!h]
%    \centering
%    \begin{scriptsize}
%        \begin{tabular}{|c||c|c|c|c|c|}
%        \hline
%        \texttt{Instance} & {\textsc{CPU}} & {\textsc {Memory}} & {\textsc {Disk}} & {\textsc {I/O Perf.}} & {\textsc {Cost Per Hour}}\\
%        \hline
%        \hline
%        \texttt{Extra Small} & 1.0 GHz & 768 MB & 20 GB & Low & $ \$0.05$\\
%        \texttt{Small} & 1.6 GHz	& 1.75 GB & 225 GB & Moderate & $\$0.12$\\
%        \texttt{Medium} & 2 x 1.6 GHz & 3.5 GB & 490 GB & High & $\$0.24$\\
%        \texttt{Large}& 4 x 1.6 GHz & 7 GB & 1,000 GB & High & $\$0.48$\\
%        \texttt{Extra Large}& 8 x 1.6 GHz & 14 GB & 2,040 GB & High & $\$0.96$\\
%        \hline
%        \end{tabular}
%    \end{scriptsize}
%    \caption{Compute instance details provided by Microsoft on July 2011.
%    \newline
%    \texttt{http://www.microsoft.com/windowsazure/compute/}.}
%    \label{table:azureTarifs}
%\end{table}
%
%The logic of the worker role is all gathered in the implementation of several \texttt{QueueServices}. These \texttt{QueueServices} are abstractions brought by \texttt{Lokad.Cloud} and a part of a message delivery mechanism using distributed queues. Each \texttt{QueueService} applies its subroutine on multiple contexts, potentially in parallel using multiple VMs. This design can be thought of as a Single Program Multiple Data (see for instance \cite{DAR2}). Each context is described in a job message, and all job messages of a specific \texttt{QueueService} are kept in a dedicated queue. This queue can store a potentially large amount of small messages. A VM runs a given \texttt{QueueService} as follows: it starts with unqueuing one message; it applies the \texttt{QueueService} subroutine using the context given by the message, then it may push job messages for one or multiple other \texttt{QueueServices}; and finally, it deletes the message from the initial queue. Each \texttt{QueueService} can be executed by an arbitrary number of VMs, while each VM can alternatively endorse all the \texttt{QueueServices} logic according to an internal scheduler. Figure \ref{fig:messageProcessing} provides an illustration of the usage of \texttt{QueueServices} for projects built on top of \texttt{Lokad.Cloud}.\\
%
%Using Queues to communicate helps building loosely coupled components and mitigates the impact of individual component failures. Messages stored in a queue are guaranteed to be returned at least once, but possibly several times: this requires one to design idempotent jobs. When a message is unqueued by the so-called \texttt{QueueService} and processed asynchronously by the worker running this \texttt{QueueService}, the message is not deleted but it becomes invisible for other workers. If a worker fails to complete the corresponding job (because it throws some exception or because the worker dies), the message becomes available after a certain period of time. Through this process, one can make sure that no job is lost because of e.g., a hardware failure. This distributed queue architecture enable large scale parallelization scheme (see for instance Li \cite{LIH1}).
%
%\begin{figure}[!h]
%\begin{center}
%\includegraphics[scale = 0.20]{../../Pictures/CloudDALVQ/messageProcessing.pdf}
%\end{center}
%\caption{Illustration of the abstractions of distributed queues (\texttt{QueueServices}) provided by the opensource project \texttt{Lokad.Cloud}.}
%\label{fig:messageProcessing}
%\end{figure}
%
%
%\subsection{\texttt{BlobStorage}}\label{subsec:blobsto}
%\texttt{Azure}'s Blob Storage (\texttt{BlobStorage}) allows applications to store large objects, up to 50 GB each. It has been designed to cope with massively parallel read access. The computing resources are provided by workers while \texttt{BlobStorage} can be used to implement communication between workers. This storage is accessible thought \texttt{http} based-REST API (for more information on REST API we refer the reader to Fielding \cite{FIE1}). The acronym BLOB stands for Binary Large OBjects and sometimes can be viewed as an alternative to relational \texttt{SQL} databases. \texttt{BlobStorage} is based on a key/value system allowing applications to store values by key. Precisely, the key is contained in a string referencing the binary object value. The string contains also a timestamp (etag) that indicates the last write action on this blob. In addition of Get and Put methods, blobs whose key shares a given prefix can be listed. Optimistic non locking atomic read-modify-write operations can be implemented using a timestamp matching condition: a write succeed if and only if the timestamp of the storage matches the one provided by the write operation. In \texttt{CloudDALVQ} project we used the object-to-cloud mapper (O/C) provided by \texttt{Lokad.Cloud} which is a very convenient abstraction layer to communicate with the \texttt{BlobStorage}. From VMs hosted in Microsoft datacenter we measured a rate from 4 Mb/s to 8Ms/s for reading (i.e., to get a blob) while for writing (put a blob) the rate is about 2Mb/s. These figures are really instable and might vary a lot. For further reading in the \texttt{BlobStorage} the reader is referred to to Durut and Rossi \cite[supplementary materials]{DUR2} and to Calder \cite{CAL1}. Some experiments on I/O costs for the \texttt{BlobStorage} have been carried out by \texttt{AzureScope} \cite{EXT1} that produced relatively different results. We are aware that the O/C mapper of \texttt{Lokad.Cloud} may also explain the differences between these results. The Figure \ref{fig:blobStorage} provides an illustration of exchanges that can be made through the \texttt{BlobStorage}.
%
%\begin{figure}[!h]
%\begin{center}
%\includegraphics[scale = 0.25]{../../Pictures/CloudDALVQ/blobStorage.pdf}
%\end{center}
%\caption{Illustration of the \texttt{BlobStorage} allowing communication between workers. The Get and Put methods are supported by \texttt{http} request and is represented with solid black lines (heading to workers for Get method and to storage for Put method).}
%\label{fig:blobStorage}
%\end{figure}

\section{The implementation of cloud DAVQ algorithm}\label{sec:cloudDALVQImpl}

\subsection{Design of the algorithm}

The cloud implementation of our DAVQ algorithm is similar to the cloud implementation of Batch K-Means depicted in chapter \ref{chap:KMeans}. In particular, it is based on two main QueueServices: the ProcessService and the ReduceService. These services are close to the MapService and ReduceService of chapter \ref{chap:KMeans}, but the MapService has been renamed in ProcessService to highlight that this new design does not fit anymore in an iterative MapReduce perspective. The two services are based on the main idea brought in chapter \ref{chap:practicalDALVQ}: averaging concurrent local versions $w^j$'s does not provide speedup for the convergence of the distributed system. A better approach consists in computing the sum of all displacement terms as shown in subsection \ref{subsec:gradients}.\\

The core logic of our DAVQ implementation is embedded in the implementation of the Process-Service. Similarly to the cloud Batch K-Means implementation, we have multiple role instances running in parallel the ProcessService, $M$ instances following notation of chapters \ref{chap:KMeans} and \ref{chap:practicalDALVQ}, each instance performing a VQ execution. In parallel, the displacement terms introduced in subsection \ref{subsec:gradients}, namely the $\Delta^j_{. \rightarrow .}$'s, are sent to the BlobStorage. The ReduceService is hosted by a dedicated worker (a single instance for now). Its task consists in retrieving the multiple displacement terms and computing the shared version of the prototypes, $w^{srd}$. This shared version is the common reference version retrieved by the multiple instances of the ProcessService which use it to update their local version. The ReduceService implements the action described by the second inner braced equations (\ref{eq:correctAveraging}) in subsection \ref{subsec:gradients} Chapter \ref{chap:practicalDALVQ}. The Figure \ref{fig:reduceservice} provides a synthetic overview of the ``reducing task'' assigned to the ReduceService, while Figure \ref{fig:cloudOverview} gives a representation of the communication of the ProcessService instances and the ReduceService instance through the BlobStorage. The implementation of the ProcessService is more complicated than the ReduceService one and is discussed in the next paragraph.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.25]{../../Pictures/CloudDALVQ/cloudImplOverview.pdf}
\end{center}
\caption{Overview of the interaction between ProcessService instances and the instance of ReduceService. All communications are made through the BlobStorage: ReduceService gets the blobs put by the ProcessService instances while they retrieve the computation of the ReduceService.}
\label{fig:cloudOverview}
\end{figure}

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.35]{../../Pictures/CloudDALVQ/ReduceService.pdf}
\end{center}
\caption{Overview of the ReduceService instance processing the ``reducing task''. This QueueService builds the shared version by computing the sum of all the available displacement terms sent by the ProcessService instances in the BlobStorage.}
\label{fig:reduceservice}
\end{figure}

The design of the ProcessService is targeted to benefit from the suppression of the synchronization process. As explained in the introduction of this chapter, in addition to the mitigation of stragglers which is a inter-machine mechanism, the asynchronism allows us to adopt an intra-machine optimization: the overlap of communication and computation inside each processing unit. Indeed, each worker running a process task will compute local VQ iterations while it downloads from the BlobStorage the latest shared version and uploads a displacement term. The design of communication and computation overlap of our ProcessService addresses two difficulties we encountered. The first difficulty is related to a compatibility issue between our prototype, the .NET framework, Azure and the Lokad.Cloud framework. During the first month of our prototype development, the Lokad.Cloud framework was build on top of the .NET framework 3.5; because of compatibility issues, we were not able to resort to the asynchronous I/O of the .NET framework 4.0.\\

The second difficulty we encountered is related to the thread-handling mechanism of Azure. To our surprise, we noticed after many attempts that we could not achieve with an automatic thread management to obtain both satisfactory I/O bandwidth and satisfactory Flops. In many cases, the Flops obtained were close to the Flops obtained without handling parallel communications but the I/O bandwidth was 10 times less than if we were only performing a I/O operation. The exact reason of this disappointing result has not been isolated but probably comes from an unsatisfactory behavior of the thread scheduler inside a VM. To circumvent this difficulty, we have chosen to resort to explicit threads for which we partly control the scheduling. The following paragraph describes how each process task is organized in multiple threads run in parallel.\\ 

The ProcessService performs simultaneously three functions using the following threads: process thread, push thread, and pull thread. The process thread is a CPU-intensive thread while push thread and pull thread are dedicated to communications with the BlobStorage. We put the emphasis on the fact that this parallelization is made inside the VMs, at CPU level. The multi-threaded execution of the ProcessService (in its latest version) is based upon the Task Parallel Library provided in the .NET framework 4.0 (see for instance Freeman \cite{FRE1}). More precisely, the pull thread is responsible for providing the latest shared version to the process thread. The pull thread uses the BlobStorage's timestamps (etags) to detect new shared versions sent by the ReduceService and made available. The pull thread and the process thread are communicating through a classical producer/consumer design pattern, implemented through a read buffer built upon the .NET 4.0 BlockingCollection class which is of great help to the read buffer for managing concurrency issues. The push thread is responsible for pushing to distant blobs the displacement term updates computed by the process thread. The interaction of the push thread and the process thread also uses a Producer/Consumer design pattern with a communication buffer (write buffer). However, in this context, the process thread is the producer and the push thread is the consumer.\\ 

As described above, the automatic scheduling of these three threads led to poor I/O bandwidth that severely deteriorated the communication behavior and therefore the consensus agreement between the multiple processing unit running a ProcessService. To avoid this, we forced the process thread to very short sleep period so it returns control to the thread scheduler thus allowing the two I/O threads to gain more uptime. This method required a fine tuning of the sleep parameters that depends both on the time to communicate a displacement term and on the time to compute a single gradient estimation. We did not manage to find a cleaner method to regulate the threads balance.\\

The blobs are then consumed by the ReduceService: this QueueService is notified by messages in its queue that new updates can be used for the shared version computation. Figure \ref{fig:processService} illustrates the multi-threaded logic of the ProcessService.\\

%The ProcessService performs simultaneously three assignments using the following threads: process thread, push thread, and pull thread. The process thread is a CPU-intensive thread while push thread and pull thread are dedicated to communications with the BlobStorage. We put the emphasis on the fact that this parallelization is made inside the VMs, at CPU level. The multi-threaded execution of the ProcessService (in its latest version) is based upon the Task Parallel Library provided in the .NET framework 4.0 (see for instance Freeman \cite{FRE1}). More precisely, the pull thread is responsible for providing the latest shared version to process thread using a local memory buffer (read buffer). BlobStorage's timestamps (etags) are used to detect new shared versions sent by the ReduceService and available. In other words pull thread and process thread communicate using a Producer/Consumer design pattern: pull thread is the producer whereas process thread is the consumer. For more details on Producer/Consumer design patterns we refer the reader to Grand \cite{GRA2}. The .NET 4.0 BlockingCollection class is of great help to the read buffer for managing concurrency issues. The push thread is responsible for pushing to distant blobs the displacement term updates computed by the process thread. The interaction of the push thread and the process thread also uses a Producer/Consumer design pattern with a communication buffer (write buffer). However, in this context, the process thread is the producer and the push thread is the consumer. The blobs are then consumed by the ReduceService: this QueueService is notified by messages in its Queue that new updates can be used for the shared version computation. Figure \ref{fig:processService} illustrates the multi-threaded logic of the ProcessService.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.35]{../../Pictures/CloudDALVQ/ProcessService.pdf}
\end{center}
\caption{Overview of the ProcessService. Each triangle stands for a specific thread (process thread, push thread and pull thread). Arrows describe read/write actions: the tail of a blue dashed arrow is read by the entity at its head and the entity at the tail of a red solid arrow makes update on the entity that lies at its head. Push thread and pull thread enable communications between process thread and the BlobStorage. Process thread alternatively performs three actions (process action $1,2,3$). Process action 1 replaces the local version of the prototypes by the sum of the latest shared version (kept in the read buffer) and a displacement term. Process action 2 uses data to execute VQ iteration and updates both the local version and the displacement term. Process action 3 moves the displacement term to a dedicated buffer (write buffer) and pushes it content to the BlobStorage.}
\label{fig:processService}
\end{figure}

\subsection{Design of the evaluation process}\label{subsec:evaluation}
In the experiments of section \ref{sec:scalabilityCloudDALVQ} and section \ref{sec:fightbatch}, the performance of clustering algorithms could be measured during their executions. However, the algorithms should not be slowed down by their performance measurements. Consequently, our DAVQ algorithm uses the fact that the BlobStorage has been designed to cope with multiple reading actions. A new QueueService is implemented: the SnapshotService and a dedicated worker is deployed to host it. The SnapshotService keeps making deep copies of the shared version blob ---the snapshots--- and stores them in other persistent blobs. We recall that the blob containing the shared version is constantly modified by the ReduceService. Therefore it is necessary to make copies to track back its evolution. Once the multiple ProcessService instances stop processing VQ iterations, i.e. when the stopping criterion is met, then the evaluation processus starts: all the computing instance that previously ran the ProcessService or the ReduceService now run the EvaluationService (another QueueService); for each snapshot, one of this computing unit  compute the corresponding empirical distortion given by equation (\ref{eq:EmpiricalDistortion}) (or its normalized version (\ref{eq:empiricalDisto})). In order to perform the evaluations of the snapshots the same synthetic data are generated again by using new instances of random generators. Therefore, no data is broadcasted through the network that would have been a massive bottleneck for the evaluations of the algorithms. This task is still highly CPU consuming: indeed, the evaluation of a single snapshot requires comparable delays than all the VQ iterations performed by a ProcessService instance. However, thanks to the elasticity of the VMs allocations, it can be completed with reasonable delays.

\section{Scalability of our cloud DAVQ algorithm}\label{sec:scalabilityCloudDALVQ}

\subsection{Speedup with a 1-layer Reduce}

In this section our objective is to prove that our DAVQ algorithm brings a substantial speedup which corresponds to the gain of wall-clock time execution brought by more computing resources compared to a sequential execution. Consequently the experiments presented in this section are similar to those provided in Chapter \ref{chap:practicalDALVQ}. In this previous chapter we proved that a careful attention should be paid on the parallelization scheme, but without specifications on the architecture. We recall that the experiments of Chapter \ref{chap:practicalDALVQ} simulated a generic distributed architecture. In this section we present some experiments made with our true cloud implementation.\\

The following settings attempt to create a synthetic but realistic situation for our algorithms. We load the local memory of the workers with data generated using the splines mixtures random generators defined in section \ref{sec:dataGenerator}. Using the notation introduced in this section, each worker is endowed with $n = 10000$ vector data that are sampled spline functions with $d=1000$. The benefit of using synthetic data for our implementations has been discussed in subsection \ref{sec:dataGenerator}. The number of clusters is set to $\kappa = 1000$ while there are $G=1500$ centers in the mixture and the number of knots for each spline $\chi$ is set to $100$. The modeling made in chapter \ref{chap:KMeans} has shown that it is not possible to tune the ratio (time to compute a gradient / time to communicate the prototypes or the prototypes update) since both linearly depend on $\kappa$ and $d$. The chosen settings are just set to ensure that the complexity of the clustering job would lead to approximately a 1 hour execution on a sequential machine when using the same learning rate as in the previous chapter: $(\varepsilon_t) = \frac{1}{\sqrt{t}}$ for $t>0$.\\

Similarly to section \ref{sec:parallelGradient}, we investigate the speedup ability of our cloud DAVQ algorithm. Recall that the number of samples in the total data set is equal to $nM$. Thus, this number is varying with $M$ (the number of ProcessService instances). Consequently, evaluations are performed with the normalized quantization criterion given by equation (\ref{eq:empiricalDisto}).\\

The following Figure \ref{fig:onelayer} shows the normalized quantization curves for $M=1,2,4,8,16$. We notice that our cloud DAVQ algorithm has a good scalability property for $M=1,2,4,8$: the algorithm converges more rapidly because it has processed much more points as shown at the end of the section. Therefore, in term of wall clock time, the DAVQ algorithm with $M=2,4,8$ clearly outperforms the single execution of the VQ algorithm ($M=1$). However we remark that bad behavior of our algorithm with $M=16$. This behavior is explained in subsequent subsection and a solution to overcome the problem is presented and evaluated.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/scalabilityOneLayer.pdf}
\end{center}
\caption{Normalized quantization curves with $M=1,2,4,8,16$. Our cloud DAVQ algorithm has good scalability properties up to $M=8$ instances of the ProcessService. Troubles appear with $M=16$ because the ReduceService is overloaded.}
\label{fig:onelayer}
\end{figure}

\subsection{Speedup with a 2-layers Reduce}

The performance curve with $M=16$ of Figure \ref{fig:onelayer}, shows an unsatisfactory situation where the algorithm implementation does not provide better quantization results than the sequential VQ algorithm. Actually, the troubles arise from the fact that the ReduceService is overloaded\footnote{The Lokad.Cloud framework provides a web application that provides monitoring tools, including a rough profiling system and real time analysis of queues length, which helped us to note that the ReduceService became overloaded when $M=16$}. Indeed, too many ProcessService instances communicate in parallel their results (the displacements terms) and fill the queue associated to the ReduceService with new tasks resulting in a ReduceService overload. Once again the elasticity of the cloud enables us to allow more resources and to create a better scheme by distributing the reducing task workload on multiple workers. As we did for Batch K-Means, we now set an intermediate layer in the reducing process by replacing the ReduceService by two new QueueServices: the PartialReduceService and the FinalReduceService. We deploy $\lfloor\sqrt{M}\rfloor$ instances of PartialReduceService that will be responsible for computing the sum of the displacement terms, but only for a certain fraction of ProcessService instances (each PartialReduceService instance is gathering the displacement terms of $\lceil\sqrt{M}\rceil$) ProcessService instances in the same way than for Batch K-Means). Then, the unique instance of FinalReduceService retrieves all intermediate sums of displacement terms made by the PartialReduceService instances and computes the new shared version. Figure \ref{fig:ReduceServiceTwoLayers} is a synthetic representation of the new layers, which are introduced to remove the difficulty brought by the overload of the initial ReduceService. In Figure \ref{fig:twolayer} we plot the performance curves with PartialReduceService instances. We remark that a good speedup is obtained up to $M=32$ ProcessService workers. However the algorithm does not scale-up to $M=64$. The cause of this problem remains under investigation.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/scalabilityTwoLayer.pdf}
\end{center}
\caption{Normalized quantization curves with $M=8,16,32,64$ instances of ProcessService and with an extra layer for the so called ``reducing task''. Our cloud DAVQ algorithm has good scalability properties for the quantization performance up to $M=32$. However, the algorithm behaves badly when the number of computing instances is raised to $M=64$.}
\label{fig:twolayer}
\end{figure}

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.35]{../../Pictures/CloudDALVQ/ReduceServiceTwoLayers.pdf}
\end{center}
\caption{Overview of the reducing procedures with two layers:
\newline
the PartialReduceService and the FinalReduceService.}
\label{fig:ReduceServiceTwoLayers}
\end{figure}

\subsection{Scalability}

Let us recall that the scalability is the ability of the distributed algorithm to process growing volumes of data gracefully. The experiments reported in the Figure \ref{fig:pointsprocessed} investigate this processing ability more precisely, by counting the samples that have been processed for the computation of the shared version. The graph shows the results with one or two layers for the reducing task and with different values of $M$, namely $M=1,2,4,8$ for the first case and $M=8,16,32,64$ for the second one. The curves appear linear, which proves that the algorithms behaves well. The slopes of the multiple curves, characteristic of the processing ability of the algorithm, exhibits a good property: the slope is multiplied by $2$ when the number of processing instances $M$ is multiplied by $2$. This proves that the algorithm has a good processing scalability as we hoped for. This is confirmed by Figure \ref{fig:durutgraph}, showing a linear curve for the number of points processed in a given time span ($3600$ seconds) using different computing instances $M$. We remark that the scalability in terms of data processed is still good up to $M=64$ while, the algorithm does not work well in term of accuracy (i.e. quantization level). Furthermore, the introduction of the second layer does not slow down the execution even for intermediate values of $M$ ($M =8$) as proved by the charts of Figure \ref{fig:pointsprocessed}.

\begin{figure}[p]
\begin{center}
$\begin{array}{c}
\includegraphics[scale = 0.55]{../../Pictures/CloudDALVQ/cloudDALVQPPOneLayer.pdf} \\
\includegraphics[scale = 0.55]{../../Pictures/CloudDALVQ/cloudDALVQPPTwoLayer.pdf} \\
\end{array}$
\end{center}
\caption{These charts plot the number of points processed with the time. The top chart shows the curves associated to $M=1,2,4,8$ with one layer for the reducing task whereas in the bottom a second layer is added and $M=8,16,32,64$.}
\label{fig:pointsprocessed}
\end{figure}


\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/scalabilityTotal.pdf} \\
\end{center}
\caption{This chart plots the number of points processed in a given time span ($3600$ seconds) for different values of $M$ ($M=1,2,4,8,16,32,64$). The reducing task is composed of one layer for $M=1,2,4$ and two layers for $M=8,16,32,64$. We observe a linear scalability in term of point processed by the system up to $M=64$ computing instances.}
\label{fig:durutgraph}
\end{figure}

\section{Competition with Batch K-Means}\label{sec:fightbatch}

The experiments provided in the previous section prove the benefit of using our cloud DAVQ algorithm for large scale quantization tasks compared to the usual VQ algorithm. However such experiments do not prove that our cloud DAVQ algorithm is a good clustering method itself. In this section we present a comparison of our cloud DAVQ algorithm with Batch $K$-Means, which is a standard clustering method. More precisely, we compare our cloud DAVQ algorithm with the Azure implementation of the distributed Batch K-Means developed in chapter \ref{chap:KMeans}. We work with a fixed large synthetic data set and our goal is to provide a summary with a reduced set of prototypes. The two algorithms are compared with the evolution of the quantization error using intermediate values of the prototypes.\\

%CloudBatchKMeans follows Dhillon and Modha's principles for parallel Batch K-Means \cite{DHI1}. Batch K-Means consists in alternating two phases (after proper initialization). In the assignment phase, the algorithm computes the Euclidean distance between each of the data point and each of the $\kappa$ prototypes. Each sample is assigned to its closest prototype. In the recalculation phase, each prototype is recomputed as the average of the data points assigned to it in the previous phase. Distance calculations are intrinsically parallel, both over the data points and the prototypes. It is therefore natural to split the computational load by allocating disjoint subsets of data points to the workers. Each worker, called a mapper following Dean and Ghemawat's terminology \cite{DEA1}, computes a full assignment phase for its fraction of the data set. Next the mapper builds a new prototypes version, where each component is updated as the average of the data points assigned to this component. After that, the workers synchronize and average these prototypes versions. The averaged prototypes are broadcasted to all the mappers, which starts a new iteration on reception. As explained in \cite{DHI1}, a parallel Batch K-Means on the different data subsets and a sequential execution of a Batch K-Means on the whole data set perform the same computations and return the same output.\\

The number of mappers in the cloud Batch $K$-Means is the number of processing units. Therefore it is natural to compare our cloud DAVQ and Batch K-Means with the same data set and the same number of ProcessService instances and mappers $M$. Similarly to section \ref{sec:scalabilityCloudDALVQ} the local memory of a worker is loaded with $n=500000$ vector data whose dimension $d$ is set to $100$. The benefit of using synthetic data for measuring the performance of clustering algorithms has been discussed in subsection \ref{subsec:evaluation}. The number of clusters is set to $\kappa = 5000$ while the number of centers in the mixture $G$ equals $7500$ and the number of knot for each splines $\chi$ equals $100$. We set the learning rate $\varepsilon_t$ (see section \ref{sec:sequentialVQ}) to a constant value $\varepsilon=1.0e^{-2}$.\\

Figure \ref{fig:fightthebatch} shows the curves of clustering performance for the two procedures in competition. We report there three experiments with various sizes of data sets $nM$ ($M=8,16,32$). We see that CloudDALVQ clearly outperforms CloudBatchKmeans. In all cases ($M=8,16,32$) the algorithm takes less time to reach a similar quantization level. The quantization curve of CloudBatchKMeans corresponds to a step function. Indeed, the reference version used for the batch $k$-means evaluations is the version built during the synchronization phase and is modified only there. To conclude, in this situation CloudDALVQ clearly outperforms CloudBatchKMeans. Therefore, this implementation seems to be a competitor to existing solutions for large scale clustering jobs, thanks to its scalability and its mathematical performance.


\begin{figure}[p]
\begin{center}
$\begin{array}{c}
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/fightbatchM8.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/fightbatchM16.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/CloudDALVQ/fightbatchM32.pdf}\\
\end{array}$
\end{center}
\caption{These charts report the competition between CloudDALVQ and Cloud-BatchKMeans. The graphs show the empirical distortion of the algorithm over the time. The empirical distortion is computed using the shared version for CloudDALVQ while it is computed during the synchronization phase (all mappers receiving the same prototypes) for CloudBatchKMeans. In these experiments CloudDALVQ outperforms CloudBatchKMeans: the same quantization level is obtained within a shorter period.}
\label{fig:fightthebatch}
\end{figure} 