%parallelSchemes file

\section{Introduction}\label{sec:intro}

The distributed Batch K-Means is a synchronous distributed algorithm: the reassignment phase, which is the CPU-intensive phase, consists in distance computations that can be distributed over a pool of processing units thanks to its data-level parallelism property. Chapter \ref{chap:KMeans} has highlighted how this algorithm can achieve good overall speedup and scale-up on a Cloud Computing platform but has also shown that its performance still suffers from the stragglers and the high costs of cloud communications. In the two following chapters, we develop a distributed asynchronous clustering algorithm so that the asynchronism will eventually reduce the performance loss incurred by the stragglers and the communications.\\

To do so, let us investigate the parallelization techniques of an algorithm which is a close relative to Batch K-Means. This algorithm is the Vector Quantization (VQ) algorithm, also referred to in the literature as online K-Means. The VQ algorithm is indeed online: its data input is drawn and processed piece-by-piece in a serial-fashion. After the examination of each data piece, a resume of the data ---the prototypes--- is refined accordingly to better reflect the data set. A reminder of the sequential VQ algorithm is given in Section \ref{sec:sequentialVQ}.\\

Because of the online property of the VQ algorithm, the parallelization techniques of the VQ algorithm presented in Section \ref{sec:parallelGradient} do not introduce unavoidable synchronization barriers that would lead to overheads which would be comparable to the distributed Batch K-Means implementation ones. As a consequence, the VQ algorithm may seem to be more adapted to Cloud Computing platforms. Our work on the distributed VQ (DVQ) algorithm is divided into two chapters: the present chapter investigates various parallelization schemes of the VQ algorithm using simple programs simulating a distributed architecture, while Chapter \ref{chap:cloudDALVQ} presents the actual behavior of the chosen DVQ scheme on Azure.\\

As outlined by Pag�s in \cite{PAG1}, the VQ algorithm belongs to the class of stochastic gradient descent algorithms (for more information on stochastic gradient descent procedures we refer the reader to Benveniste et al. in \cite{BEN1}). The DVQ algorithm is based upon the VQ technique: it executes several VQ procedures on different (and possibly distant) processing units while the results are broadcasted and merged through the network. As a consequence, the DVQ algorithm falls within the general framework of parallel gradient descent algorithms. \\

The distributed gradient descent algorithms have been vastly experimented and analyzed (see e.g. Chapter 14 of \cite{BBL}). Distributed and asynchronous stochastic gradient descent procedures for supervised machine-learning problems have been studied theoretically and experimentally by Langford et al. in \cite{SlowLearners} and by Louppe and Geurts in \cite{LOU1}. In these papers the computer programs are based on shared memory architectures. However, our work focuses on the unsupervised learning problem of clustering without any efficient distributed shared memory. The lack of such a shared memory introduces significant time penalties when accessing data, therefore slowing down the exchange of information between the computing entities.\\

In order to avoid this problem, Zinkevich et al.~propose in \cite{LAN2} a parallelized stochastic gradient descent scheme with no communication between processors until the end of the local executions. As mentioned by the authors, this perfectly suits the popular distributed computing framework MapReduce (see, for instance, Dean and Ghemawat in \cite{MapReduce2004}). However, in our quantization context this approach would not work because the loss function, namely the empirical distortion, is not convex\footnote{Even in a convex context, this solution may turn out to be not optimal, as outlined in the so-called ``no-communication'' solution study in \cite{DekelShamir}.}. Therefore a final average of completely independent VQ executions would not lead to satisfactory prototypes, as shown by the results of Section \ref{sec:parallelGradient} below when the delays get large.\\

The non-convexity of the loss function recalled in equation \eqref{eq:EmpiricalDistortion} seems to be a very important point, if not the most important, of the nature of our problem. Indeed, in the convex and smooth case, recent results have presented parallelization schemes that have proved to be asymptotically optimal (see for instance the recent \cite{DekelShamir}). We show in this chapter that comparable distribution schemes do not provide speedup in our non-convex case but we provide improvements through other distribution schemes for which sensitive speedups are obtained.\\

%We suppose having access to a finite dataset containing $n$ $\real^d$-valued samples $\{\zbf_{t}\}_{t=1}^n$ ($d\geq1$, $n\geq1$). The goal consists in finding a ``good'' quantization scheme for this data set by building relevant prototypes $w \in \rdkap$. They will act as a ``summary'' of the data set. The VQ technique defined by equation (\ref{eq:gradientdescent}) builds such prototypes by computing iterations of $\rdkap$-valued vectors $\{w(t)\}_{t=0}^\infty$. Using the material and notation of Section \ref{sec:CLVQ}, in this practical context, the VQ writes
%%\begin{equation}\label{eq:finiteSample}
%%w(t+1)= w(t) - \varepsilon_{t+1}H\left(\zbf_{\{t+1 \text{ mod } n\}},w(t) \right), \quad t \geq 0,
%%\end{equation}
%where $w(0) \in \left(\real^d\right)^\kappa$. The difference with equation (\ref{eq:gradientdescent}) is the \emph{mod} operator, which denotes the remainder of an integer division operation. These iterations make passes (i.e., cycles) over the data set until a stopping criterion is met. The comparison with the theoretical description, where data set is supposed to be infinite and consequently where there is no cycle, have been studied by Bermejo and Cabestany in \cite{BER3}.\\

%Practical settings also impose a new point of view regarding the performance criterion. In what follows, the performance of the clustering is measured using the empirical distortion. This quantity is denoted by $C_{\mu_n}$ in equation (\ref{eq:empirical}), and it will be, as for now, denoted by $C_n$ for simplification. Thus, the empirical quantization performance of the prototypes $w \in \left(\real^d\right)^\kappa$ takes the form
%\begin{equation}\label{eq:disto}
%C_n\left(w\right) = \sum_{t=1}^n{\min_{\ell = 1, \ldots, \kappa} \left\|\zbf_t  - w_\ell\right\|^2}.
%\end{equation}

We will not be concerned with practical optimizations of the several parameters of VQ implementations such as the initialization procedures or the choice of the sequence of steps $\{\varepsilon_t\}_{t=1}^{\infty}$ (introduced in Section \ref{sec:sequentialVQ}). In the present chapter, we assume that a satisfactory sequential VQ implementation has been found. Therefore, our goal is to make this algorithm ``scalable'' (i.e. able to cope with larger data sets) using multiple computing resources without modifying those parameters. Consequently, in all our experiments, the initialization procedure is always the same: each initial prototype is an average of $20$ arbitrary data vectors. For a review of standard initialization procedures, the reader is referred to Peterson et al. in \cite{PET1} where a satisfactory method is also provided for the usual Batch K-Means algorithm. The techniques proposed by Milligan and Isaac in \cite{MIL1}, Bradley and Fayyad in \cite{BRA1}, or Mirkin in \cite{MIR1} also seem to perform well. As for the learning rate, similarly to Langford et al. in \cite{SlowLearners} or \cite{BBL}, we set $\varepsilon_t = 1 / \sqrt{t}$ for all $t\geq1$. A thorough analysis of the theoretical properties of the learning rates can be found in Bottou et al. in \cite{BOT5,BOT6}.\\

The rest of this chapter is organized as follows. Section \ref{sec:sequentialVQ} provides a short introduction to the sequential VQ algorithm. Section \ref{sec:dataGenerator} describes the synthetic functional data used in the experiments of this and the next chapter. Section \ref{sec:parallelGradient} is devoted to analyses and discussions about simple distributed schemes implementing VQ procedures. In particular, we show in this section that the first natural implementation of VQ cannot bring satisfactory results. However, we describe alternative parallelization schemes that, as we had expected, give substantial speedup, i.e. gain of execution time brought by more computing resources compared to a sequential execution.\\

\section{The sequential Vector Quantization algorithm}
\label{sec:sequentialVQ}
Let us recall the exact clustering problem dealt with in Chapter \ref{chap:KMeans} and detailed in equation \eqref{eq:EmpiricalDistortion}: given a data set of $N$ points $\{\zbf_{t}\}_{t=1}^N$ of a $d$ dimensional space, we want to construct $\kappa$ points $\{w_{k}\}_{k=1}^{\kappa}$, referred to in the following as prototypes or centroids, as a resume of the data set using the euclidean distance as a similarity measure. Through this similarity measure, one can define the empirical distortion:

\begin{equation}
\label{eq:EmpiricalDistortion}
C_{N}(w) = \sum_{t=1}^N{\min_{\ell = 1, \ldots, \kappa} \left\|\zbf_t  - w_\ell\right\|^2}, \quad \mbox{$w \in \left(\real^d\right)^\kappa$}.
\end{equation}

Like Batch K-Means, the popular VQ algorithm (see Gersho and Gray in \cite{GER1}) introduces a technique to build prototypes $\{w_{k}\}_{k=1}^{\kappa}$ that provides a satisfactory result as regards the previous criterion.\\

The VQ algorithm defined by equation \eqref{eq:finiteSample} consists in incremental updates of the $\rdkap$-valued prototypes $\{w(t)\}_{t=0}^\infty$. Starting from a random initial $w(0) \in \left(\real^d\right)^\kappa$ and given a series of positive \emph{steps} $(\varepsilon_t)_{t>0}$, the VQ algorithm produces a series of $w(t)$ by updating $w$ at each step with a ``descent term''. Let us first introduce $H(\zbf,w)$ defined by
\begin{equation*}\label{def:defH}
H(\zbf,w) = \left( \left(w_\ell - \zbf\right) \mathds{1}_{\left\{l= \argmin_{i = 1, \ldots, \kappa} \left\|\zbf  - w_i\right\|^2\right\}}\right)_{1 \leq \ell \leq \kappa}.
\end{equation*}

Then, the VQ iterations can be written:

\begin{equation}\label{eq:finiteSample}
w(t+1)= w(t) - \varepsilon_{t+1}H\left(\zbf_{\{t+1 \text{ mod } n\}},w(t) \right), \quad t \geq 0,
\end{equation}

where the mod operator stands for the remainder of an integer division operation. The entire VQ algorithm is described in the logical code of Algorithm \ref{algo_SequentialVQ}. The VQ iterations make passes (i.e. cycles) over the data set until a stopping criterion is met. The comparison with the theoretical description, where the data set is supposed to be infinite and consequently where there is no cycle, has been studied by Bermejo and Cabestany in \cite{BER3}.\\

As pointed out by Bottou and Bengio in \cite{BottouBengioKMeans}, the VQ algorithm defined by the iterations \eqref{eq:finiteSample} can also be viewed as the online version of the widespread Batch K-Means presented in Chapter \ref{chap:KMeans} (which is also referred as Lloyd's method in \cite{Lloyd} for the definition). The VQ algorithm is also known as the Competitive Learning Vector Quantization (especially in the data analysis community), the Kohonen Self Organizing Map algorithm with 0 neighbor (see for instance Kohonen in \cite{KOH1}) or the online K-Means procedure (see MacQueen in \cite{MacQueen} and Bottou and Bengio in \cite{BottouBengioKMeans}) in various fields related to statistics.\\

As already explained in the introduction, it is well known that the VQ algorithm belongs to the class of stochastic gradient descent algorithms. However, the almost sure convergence of the VQ algorithm cannot be obtained by general tools of gradient descent such as the Robbins-Monro method for instance (see Patra in \cite{PatraDALVQ}). Indeed, the main difficulty essentially arises from the lack of convexity and smoothness of the distortion. We refer the reader to Pag�s in \cite{PAG1} for a proof of the almost sure convergence of the VQ procedure.\\

\begin{algorithm}
    \caption{Sequential VQ algorithm}
    \begin{algorithmic}
    \label{algo_SequentialVQ}
        \STATE Select $\kappa$ initial prototypes $(w_{k})_{k=1}^{\kappa}$
        \STATE Set t=0
        \REPEAT
            \FOR{$k=1$ to $\kappa$}
                \STATE compute $||\zbf_{\{t+1 \text{ mod } n\}}-w_{k}||_{2}^{2}$
            \ENDFOR
            \STATE Deduce $H(\zbf_{\{t+1 \text{ mod } n\}},w)$
            \STATE Set $w(t+1)= w(t) - \varepsilon_{t+1}H\left(\zbf_{\{t+1 \text{ mod } n\}},w(t) \right)$
            \STATE increment $t$
        \UNTIL {the stopping criterion is met}
    \end{algorithmic}
\end{algorithm}

\section{Synthetic functional data}\label{sec:dataGenerator}

The various parallelization schemes investigated in this chapter have been tested using simple programs simulating a distributed architecture, and synthetic vector data\footnote{Source code is available at the address \texttt{http://code.google.com/p/clouddalvq/}}. Contrary to Batch K-Means, the parallelization schemes of the VQ algorithm and the sequential VQ implementation do not provide the very same results. As a consequence, the actual data values have an influence on the algorithm duration and performance; contrary to Batch K-Means, we will need to evaluate each parallelization scheme both on a speedup criterion and on an accuracy criterion (using the empirical distortion defined by equation \eqref{eq:EmpiricalDistortion} above). The use of synthetic data serves three purposes: firstly, through this synthetic generation, the data set size can be tuned to resize our clustering problem so that it embodies a challenging problem as far as computations and communications are concerned. Secondly, for a given data set size, we can tune the ``clustering difficulty'' by modifying the complexity of our data, i.e. by modifying the actual data set dimensionality (tuned by $G$ in the following). Finally, the ability to re-generate any part of the data set on any processing unit without downloading it from a storage is of great help when evaluating any quantization results on the cloud as it removes a lot of communication and therefore eases the evaluation process architecture.\\

Many real-world applications produce high dimensional data. In numerous situations, such data can be considered as sampled functions. This is the case in the popular context of time series, meteorological data, spectrometric data etc. We therefore have chosen to resort to $B$-splines to analyze our clustering algorithm implementations. In the present section we explain the construction of the $B$-splines mixtures random generators used in our experiments. Note that the recent researches by Abraham et al. in \cite{ABR} and Rossi et al. in \cite{ROS1} focus on clustering algorithms for $B$-splines functional data.\\

\subsection{$B$-spline functions}
The $B$-spline functions have been thoroughly studied for many decades. They have been analyzed both for their surprising mathematical properties and their ability to create nice shapes. Consequently they are frequently used in industries related to design such as automotive industry, architecture, graphics editor software, etc. In this subsection we only provide a computational definition. For more information on $B$-splines history and mathematical properties, we refer the reader to de Boor in \cite{DEB1}.\\

The term ``$B$-spline'' ---an abbreviation of basis spline---, is ambiguous as its meaning varies depending on the context: in some situations, a $B$-spline refers to one of the $b_{i,n}(.)$ functions defined below. In other situations, a $B$-spline refers to a linear combination of the $b_{i,n}(.)$ functions. In the following, the $b_{i,n}(.)$ functions are referred to as basic $B$-splines, and a $B$-spline is defined as a linear combination of these basic $B$-splines.\\

The family of basic $B$-spline functions of degree $n$ with $\chi$ knots $x_0 \leq x_1 \leq \ldots \leq x_{\chi-1}$ is composed of $\chi-n-1$ piecewise polynomial functions. The symbols $n$ and $\chi$ denote both integers satisfying $\chi \geq n + 2$. They can be defined recursively by the Cox-de Boor formula below (see for instance de Boor in \cite{DEB}). For all $x \in [x_0,x_{\chi-1}]$,
\[b_{i,0}(x) = \begin{cases}1 & \text{ if } x_i \leq x < x_{i+1}\\
0 & \text{ otherwise }
 \end{cases}, \quad i=0, \ldots, \chi-2, \]
and
\[b_{i,n}(x) = \frac{x - x_i}{x_{i+n} - x_i}b_{i,n-1}(x) + \frac{x_{i+n+1} - x}{x_{i+n+1} - x_{i+1}}b_{i+1, n -1}(x), \quad i=0, \ldots, \chi- n -2.\]
Figure \ref{fig:BasicBSplines} is a plot of six cubic basic $B$-splines where $\chi=10$ and $n=3$.

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.6]{../../Pictures/parallelDALVQ/basicBSplines.pdf}
\end{center}
\caption{Plots of the six basic cubic $B$-spline functions with ten uniform knots: $x_0 =0, \ldots, x_9 =9$ ($n =3$, $\chi=10$).}
\label{fig:BasicBSplines}
\end{figure}

As mentioned earlier, a $B$-spline is a vector of the linear span of the basic $B$-splines. Thus, a $B$-spline takes the form, for any $x \in [x_0,x_{\chi-1}]$,
\begin{equation}\label{eq:Bspline}
b(x) = \sum_{i=0}^{\chi-n-2}{p_ib_i(x)},
\end{equation}
where $\{p_i\}_{ i=0}^{\chi -n -2} \in \real^{\chi -n -1}$ are referred to as the control points or de Boor points. Figure \ref{fig:BSplines} shows a cubic (i.e., $n=3$) $B$-spline. The graph illustrates the natural smoothness of such functions. In the sequel, we consider only cubic $B$-splines and uniform knots: $x_0 =0, \ldots, x_{\chi-1} = \chi -1$ with $\chi \geq 6$.

\begin{figure}
\begin{center}
\includegraphics[scale = 0.6]{../../Pictures/ParallelDALVQ/BSplines.pdf}
\end{center}
\caption{Plot of a cubic $B$-spline, a linear combination of the basic $B$-splines plotted in Figure \ref{fig:BasicBSplines}}
\label{fig:BSplines}
\end{figure}

\subsection{$B$-splines mixtures random generators}
Let us now describe the random generators used throughout the experiments described in Chapters \ref{chap:practicalDALVQ} and \ref{chap:cloudDALVQ}. Each of these vector-valued generators is based on $G$ ($G\geq1$) $B$-spline functions which, from now on, will be referred to as the centers of the mixture. We also want these centers ``not to be too close to each other''. To do so, we have chosen our centers with (nearly) orthogonal coefficients $p_i$'s appearing in equation \eqref{eq:Bspline}. In the next paragraph, we explain the construction of such coefficients.\\

For $g=0, \ldots, G-1$ and $i=0, \ldots, \chi- 5$, let the $u_{g,i}$'s be reals, drawn independently from the standard uniform distribution on the open interval $(0,1)$. The vectors $\{\{p_{g,i}\}_{i=0}^{\chi - 5}\}_{g=0}^{G-1}$ are defined as block-wise orthogonal vectors computed by applying a revised Gram-Schmidt procedure on the $G$ vectors $\{u_{g,i}\}_{i=0}^{\chi-5}$ (see for instance Greub in \cite{GRE}). Let us keep in mind that the Gram-Schmidt algorithm is well defined if the number of vectors, here $G$, is lower than or equal to the dimension of the vector space which is equal to $\chi- 4$. In all our experiment settings, we will have $G > \chi - 4$. Therefore, we divide our $G$ vectors $\{\{u_{g,i}\}_{i=0}^{\chi - 4}\}_{g=0}^{G-1}$ into subgroups of $\chi- 4$ vectors. To each group, we apply a block-wise Gram-Schmidt procedure. All the $G$ vectors thus obtained are then normalized to a common value $sc > 0$ and are then referred to the $G$ vectors $\{\{p_{g,i}\}_{i=0}^{\chi - 5}\}_{g=0}^{G-1}$.\\

For any $g\in\{1,\ldots,G-1\}$, let $B_g$ be the $B$-spline defined by equation \eqref{eq:Bspline} with the control points $\{p_{g,i}\}_{i=0}^{\chi - 5}$ chosen as explained in the paragraph above. Let us not forget that in our context, the data cannot be functions but only sampled functions. Consequently, the centers of the distribution are the functions $B_g$'s, only observed through the $d$-dimensional vector ($d \geq1$) $\overline{B}_g$, where
\[\overline{B}_g = \left\{B_g\left(i(\chi-1)/d\right) \right\}_{i=0}^{d-1}.\]

The vectors $\overline{B}_0,\ldots,\overline{B}_{G-1}$ define the centers of the $\real^d$-valued law of probability defined below. As shown in Figure \ref{fig:Centers}, the orthogonal property of the coefficients makes them ``not too close to each other'', as requested.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.55]{../../Pictures/ParallelDALVQ/centers.pdf}
\end{center}
\caption{Plot of four splines centers with orthogonal coefficients:
$\overline{B}_1, \ldots, \overline{B}_4$ where $G = 1500$, $d=1500$, $\chi = 50$, $sc =10$.}
\label{fig:Centers}
\end{figure}

We are in a position to define the distribution simulated by our $\real^d$-valued random generator. Let $\textbf{N}$ be a uniform random variable over the set of integers $\{0, \ldots , G-1\}$ and $\varepsilon$ a $\real^d$-valued Gaussian random variable with $0$ mean and $\sigma^2 I_d$ for covariance matrix, where $\sigma >0$ and $I_d$ stands for the $d\times d$ identity matrix. Our random generators simulate the law of the $d$-dimensional random variable $\textbf{Z}$, defined by
\begin{equation}\label{eq:randomVariable}
\textbf{Z} = \overline{B}_{\textbf{N}} + \varepsilon.
\end{equation}
Figure \ref{fig:data} shows two independent realizations of the random variable $\textbf{Z}$ defined by equation \eqref{eq:randomVariable}, the sampled functional data used in our experiments.
\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.55]{../../Pictures/ParallelDALVQ/data.pdf}
\end{center}
\caption{Plot of two independent realizations of the random variable $\textbf{Z}$ defined by equation \eqref{eq:randomVariable}: $\overline{B_1}, \ldots, \overline{B_4}$ where $G = 1500$, $d=1500$, $\chi = 50$, $sc =10$.}
\label{fig:data}
\end{figure}


\section[VQ parallelization scheme]{Discussing parallelization schemes of the VQ algorithm}\label{sec:parallelGradient}
This section is devoted to the analysis of some practical parallelization schemes for the DVQ algorithm. Indeed, the practical parallelization of the VQ procedure is not straightforward. For example, let us remark that, if two different prototypes versions $w^1$ and $w^2$ have been computed by two concurrent VQ executions, there is no guarantee that the quantization scheme provided by the convex combination $\alpha w_1 + (1- \alpha) w_2$ ($\alpha \in (0,1)$) has good performance regarding the non-convex empirical distortion criterion. Therefore, careful attention should be paid to the parallelization scheme of any attempt of DVQ implementation.\\

Following the notation of Subsection \ref{sec_KMeans_SMP_Algo}, we suppose that we own a (potentially) large data set split among the local memory of $M$ computing units. This split data set is represented by the sequences $S^{i}=\{\zbf_{t}^{i}\}_{t=1}^{n}$, $i\in\{1,\ldots,M\}$, the global data set being composed of $N = nM$ vectors. In addition to its part of the split data set $S^{i}$, the processing unit $i$ owns a specific version of the prototypes $w^{i}$.\\

From the $M$ prototypes versions $w^{i}$, $i\in\{1,\ldots,M\}$, a shared version $w^{srd}$ of the prototypes is built, following a merging logic which varies over the different parallelization schemes described below. This shared version corresponds to the result that the algorithm would return to the user if it was asked to. It is strongly related to the ``consensus version'' described for example in \cite{PatraDALVQ}. The shared version is the prototypes version used to evaluate the actual accuracy of our various implementations. More specifically, we use a normalized empirical distortion to evaluate the vector $w^{srd}$. This empirical distortion is defined for all $w \in \rdkap$ by
\begin{equation}\label{eq:empiricalDisto}
C_{n,M}(w) = \frac{1}{nM} \sum_{i=1}^M \sum_{t=1}^n{\min_{\ell = 1, \ldots, \kappa} \left\|\zbf^i_t  - w_\ell\right\|^2}.\\
\end{equation}\\

We have already highlighted in Chapter \ref{chap:KMeans} the importance for distributed Batch K-Means of making sure that the machines RAM are heavily loaded. In the same manner, the RAM of each processing unit is heavily loaded in our distributed VQ experiments. As a consequence, the number of points in the total data set ($nM$) is made dependant of the number of computing unit $M$. Strictly speaking, the values of $C_{n,M}(w)$ obtained for different values of $M$ and fixed values of $n$ and $w$ are therefore not equal. Yet, they are all very close to the generalization error and can be compared without trouble.\\

All the following parallelization schemes are evaluated with a simulated distributed architecture (actually performed sequentially on a single-core CPU). All the experiments in this section are carried out with the following parameters: $n = 1000$, $d = 100$, $\kappa = 100$ and with the synthetic data sampled by the random generators introduced above. The generators are tuned with the following settings:  $G = 150$, $\chi = 100$, $sc =100$ and $\sigma = 5.0$. As it is detailed in Subsection \ref{subsec:evaluation}, the evaluation of a prototypes version is often as long as the entire quantization algorithm. As a consequence, reporting the evolution of the quantization error over time or over the iterations is much more costly than running the algorithm (as for a given VQ execution, the evaluation of the quantization error is run multiple times to plot its evolution). The chosen settings reflect this fact and are kept small to lower the evaluation time performed on our single-core CPU. Our evaluation settings are modified in Chapter \ref{chap:cloudDALVQ} to induce much larger computational challenges, as we will leverage on hundreds of processing units for the quantization algorithms and for the evaluation of these algorithms.\\

In order to bring substantial speedup, three schemes for VQ parallelization are proposed below. We start by the investigation of the most natural parallelization scheme, where the versions resulting of the distributed VQ executions are averaged on a regular basis. This updating rule is used for example in \cite{TSI1}, \cite{PatraDALVQ} or \cite{DekelShamir}\footnote{In \cite{DekelShamir}, the gradients are averaged but the prototypes versions. However, this strategy results in the very same prototypes version update rule}. We have tracked the evolution of the empirical distortion with the iterations. This evolution shows that no speedup can be brought using this scheme. Thus, a new model is proposed to overcome this unsatisfactory situation. The main idea beneath this new scheme is the following one: the agreement between the computing units is not performed using an average anymore, but it uses instead the cumulated sum of the so-called ``displacement terms''. Finally, the last model to be considered is an improvement of the second model, where delays are introduced to prepare the analysis of asynchronism used in Chapter \ref{chap:cloudDALVQ}.\\

\subsection{A first parallel implementation}

Our investigation therefore starts with the following simple parallelization scheme of the VQ technique. All the versions are set equal at time $t=0$, $w^1(0) = \ldots =w^M(0)$. For all $i \in \{1, \ldots,M\}$ and all $t \geq0$, we have the following iterations:
\begin{equation}\label{eq:averaging}
\begin{cases}
w^i_{temp} = w^i(t) - \varepsilon_{t+1} H\left(\zbf^i_{\{t+1\text{ mod }n\}},w^i(t)\right)& \\ %\label{eq:processing}\\
w^i(t+1) = w^i_{temp} & \text{ if }  t \text{ mod } \tau  \neq 0 \text{ or } t=0,\\
    \begin{cases}
        w^{srd} =  \frac{1}{M}\sum_{j=1}^M{w^j_{temp}} & \\
        w^i(t+1) =  w^{srd}&\\
    \end{cases}& \text{ if } t \text{ mod } \tau =0 \text{ and } t \geq \tau.
\end{cases}
\end{equation}

This algorithm is composed of two phases. The processing phase is given by the first equation of \eqref{eq:averaging}. It corresponds to a local execution of the VQ procedure. The result of these computations is referenced by the variables $w^i_{temp}$. The averaging phase, defined by the braced inner equations, describes the merging strategy chosen for this parallelization scheme. Note that the averaging phase is executed only whenever $\tau$ points have been processed by each concurrent processor. The larger the integer $\tau$ is, the more independent the concurrent processors are left for their execution. During the averaging phase a common shared version $w^{srd}$ is computed as the average of all the $w^i$, $i\in\{1,\ldots,M\}$ . Once $w^{srd}$ has been computed, all the local versions, namely the $w^i$, are set equal to $w^{srd}$. If the averaging phase does not occur, then the local versions are naturally set equal to the result of the local computations. See Figure \ref{fig:illustrationParallel} for a graphical illustration of the iterations \eqref{eq:averaging}.\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.22]{../../Pictures/ParallelDALVQ/parallel.pdf}
\end{center}
\caption{Illustration of the parallelization scheme of VQ procedures described by equations \eqref{eq:averaging}. The prototypes versions are synchronized every $\tau$ points.}
\label{fig:illustrationParallel}
\end{figure}

This parallelization scheme fits the theoretical Distributed Asynchronous Learning Vector Quantization (DALVQ) model presented by Patra in \cite{PatraDALVQ}. In particular, the iterations \eqref{eq:averaging} define a DALVQ procedure without asynchronism and for which the averaging phases are performed on a regular basis. A straightforward examination shows that both sets of assumptions \textbf{$\left(AsY_1\right)$} and \textbf{$\left(AsY_2\right)$} presented in \cite{PatraDALVQ} are satisfied. As a consequence, asymptotic consensus and convergence towards critical points are guaranteed.\\

The results are gathered with different values of $\tau$ ($\tau\in\{1,10,100\}$) in the charts displayed in Figure \ref{fig:parallel}. For each chart, we plot the performance curves when the distributed architecture has a different number of computing instances: $M\in\{1, 2, 10 \}$. The curve of reference is the sequential execution of the VQ, that is $M=1$. We can notice that for every value of $\tau\in\{1,10,100\}$, multiple resources do not bring speedup for convergence. Even if more data are processed, there is no increase in the convergence speed. To conclude, no gain in terms of wall time can be brought using this parallel scheme. In the next subsection we investigate the cause of these non-satisfactory results and propose a solution to overcome this problem.\\

\begin{figure}[p]
\begin{center}
$\begin{array}{c}
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/pa1.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/pa10.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/pa100.pdf}\\
\end{array}$
\end{center}
\caption{Charts of performance curves for iterations \eqref{eq:averaging} with different numbers of computing entities: $M=1,2,10$. The three charts correspond to different values of $\tau$ which is the integer that characterizes the frequency of the averaging phase ($\tau=1,10,100$).}
\label{fig:parallel}
\end{figure}

\subsection{Towards a better parallelization scheme}\label{subsec:gradients}
Let us start the investigation of the parallel scheme given by iterations \eqref{eq:averaging} by rewriting the sequential VQ iterations \eqref{eq:finiteSample} using the notation introduced in Section \ref{sec:intro}. For $t \geq \tau$ it holds
\begin{equation}\label{eq:iteration}
w(t+1) = w(t - \tau + 1)- \sum_{t'=t-\tau+1}^{t} \varepsilon_{t'+1}H\left(\zbf_{\{t'+1\text{ mod }n \}},w(t')\right).
\end{equation}

For iterations \eqref{eq:averaging}, consider a time slot $t \geq0$ where an averaging phase occurs, that is, $t \text{ mod } \tau =0$ and $t > 0$. Then, for all $i\in\{1, \ldots,M\}$,
\begin{equation}\label{eq:iteration2}
w^i(t+1) = w^i(t - \tau + 1) - \sum_{t'= t- \tau +1}^{t}\varepsilon_{t'+1}\left(\frac{1}{M}\sum_{j=1}^M H\left(\zbf^j_{t'+1},w^j(t')\right)\right).
\end{equation}

To the empirical distortion defined by equation \eqref{eq:empiricalDisto} is associated its theoretical counterpart the distortion function C (see for example \cite{Pollard} or \cite{BiauKMeans}). In the first situation of iterations \eqref{eq:iteration}, for a sample $\zbf$ and $t' >0$, $H(\zbf,w(t'))$ is an observation and an estimator of the gradient $\nabla C(w(t'))$ (see e.g. \cite{PatraDALVQ}). In the second situation of iterations \eqref{eq:iteration2}, let us assume that the multiple versions are close to each other. This means that $w^j(t')\thickapprox w^i(t')$, for all $(i,j) \in \{1,\ldots,M\}^2$. Thus, the average
\[\frac{1}{M} \sum_{j=1}^M H\left(\zbf^j_{\{t'+1\text{ mod }n \}},w^j(t')\right)\]
can also be viewed as an estimation of the gradient $\nabla C(w^i(t'))$, where $i \in \{1,\ldots,M\}$.\\

In both situations, the descent terms following the step $\varepsilon_{t' +1}$ in equations \eqref{eq:iteration} and \eqref{eq:iteration2} are estimators of gradients. Consequently, the two algorithms can be thought of as stochastic gradient descent procedures with different estimators. However, they are driven by the same learning rate which is given by the sequence of steps $\{\varepsilon_{t}\}_{t=1}^{\infty}$. The convergence speed of a non-fixed step gradient descent procedure is essentially driven by the decreasing speed of the sequence of steps (see for instance Kushner et al. in \cite{KUS1} or Benveniste et al. in \cite{BEN1}). The choice of this sequence is subject to an exploration/convergence trade-off. In the case of the sequential VQ iterations \eqref{eq:finiteSample} the time slot $t$ and the current count of samples processed are equal. Therefore, for the same time slot $t$, the distributed scheme of iterations \eqref{eq:averaging} has visited much more data than the sequential VQ algorithm to build its descent terms. Yet, since the two algorithms share the same learning rate, they are expected to get comparable convergence speeds, which is confirmed by the charts in Figure \ref{fig:parallel}.\\

We now introduce for all $j \in \{1, \ldots, M\}$ and $t_2 \geq t_1 \geq 0$ the term $\Delta^{j}_{t_1 \rightarrow t_2}$ ---referred to in the following as the ``displacement term''--- which corresponds to the variation of the prototypes computed by processor $j$ during the time interval $(t_1, t_2)$,

\begin{equation*}\label{eq:displacement}
\Delta^j_{t_1\rightarrow t_2} = \sum_{t'=t_1 + 1}^{t_2} \varepsilon_{t'+1}H\left(\zbf^j_{\{t'+1\text{ mod }n \}},w^j(t')\right).
\end{equation*}
Using the notation above, the parallel scheme given by iterations \eqref{eq:averaging} writes, for all $t > 0$ where $t \text{ mod } \tau =0$ and all $i \in \{1, \ldots,M\}$,

\begin{equation}\label{eq:iteration3}
w^i(t+1) = w^i(t - \tau + 1)-\frac{1}{M}\sum_{j=1}^M \Delta^j_{t -\tau \rightarrow t}.
\end{equation}

In the previous paragraphs we have explained that the sequential VQ and the distributed scheme above share the same convergence speed with respect to the iterations $t\geq0$. Therefore, if one considers the evolution of the learning rate with respect to the number of samples processed, then the distributed scheme has a much lower learning rate, thereby favoring exploration to the detriment of convergence. As explained in the introduction to the chapter, we assume that the learning rate provided by the sequence $\{\varepsilon_t\}_{t=1}^{\infty}$ is supposed to be satisfactory for the sequential VQ. In this work, we seek for a parallel scheme that has, in comparison to a sequential VQ, the same learning rate evolution in terms of processed samples and whose convergence speed with respect to iterations is accelerated. Notice that these iterations correspond to the true wall time measured by an exterior observer. With this aim in view, we propose the new system of equations \eqref{eq:correctAveraging}.\\

At time $t=0$ all versions are equal, $w^1(0) = \ldots =w^M(0)=w^{srd}$. For all $i \in \{1, \ldots,M\}$ and all $t \geq0$, the new parallel scheme is given by
\begin{equation}\label{eq:correctAveraging}
\begin{cases}
w^i_{temp} = w^i(t) - \varepsilon_{t+1} H\left(\zbf^i_{\{t+1\text{ mod }n\}},w^i(t)\right)& \\ %\label{eq:processing}\\
w^i(t+1) = w^i_{temp} & \text{ if }  t \text{ mod } \tau  \neq 0 \text{ or } t=0,\\
    \begin{cases}
        w^{srd} = w^{srd} - \sum_{j=1}^M{\Delta^j_{t- \tau \rightarrow t} } & \\
        w^i(t+1) =  w^{srd} &\\
    \end{cases}& \text{ if } t \text{ mod } \tau =0 \text{ and } t \geq \tau.
\end{cases}
\end{equation}

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.20]{../../Pictures/ParallelDALVQ/gradientParallel.pdf}
\end{center}
\caption{Illustration of the parallelization scheme of VQ procedures described by equations \eqref{eq:correctAveraging}.}
\label{fig:illustrationGradient}
\end{figure}

Roughly speaking, these iterations add up the displacement terms $\Delta^j_{. \rightarrow .}$ rather than use the average as shown in equation \eqref{eq:iteration3}. The computation of $w^{srd}$ described in equations \eqref{eq:correctAveraging} is now called ``reducing phase'' instead of ``averaging phase''. It can be easily implemented, but at the price of keeping supplementary information in the local memory of the computing units. More precisely, iterations \eqref{eq:correctAveraging} require to have, for processor $j$ at any time $t\geq0$, the displacement term from $\lfloor t/\tau\rfloor\tau + 1$  (last reducing phase) to the current time $t$. With the notation above, this displacement term writes $\Delta^j_{\lfloor t/\tau\rfloor\tau \rightarrow t}$. Figure \ref{fig:illustrationGradient} provides a synthetic representation of these iterations.\\

Contrary to the previous parallelization scheme, the iterations defined in equation \eqref{eq:correctAveraging} do not fit into the theoretical Distributed Asynchronous Learning Vector Quantization (DALVQ) model presented by Patra in \cite{PatraDALVQ}. To our knowledge, no theoretical results provide such iterations with the convergence towards critical points. However, the results of the simulations in the charts displayed in Figure \ref{fig:correctedParallel} show effective convergence and that substantial speedups are obtained with more distributed resources. The acceleration is greater when the reducing phase is frequent. Indeed, if $\tau$ is large then more autonomy has been granted to the concurrent executions and they could be attracted to different regions that would slow down consensus and convergence.

\begin{figure}[p]
\begin{center}
$\begin{array}{c}
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/correctedpa1.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/correctedpa10.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/correctedpa100.pdf} \\
\end{array}$
\end{center}
\caption{Charts of performance curves for iterations \eqref{eq:correctAveraging} with different numbers of computing entities, $M=1,2,10$. The three charts correspond to different values of $\tau$ ($\tau=1,10,100$).}
\label{fig:correctedParallel}
\end{figure}

\subsection{A parallelization scheme with communication delays.}
In the previous subsections we have focused on the speedup that can be brought by parallel computing resources. However, the previous parallelization schemes did not deal with the computation of shared versions. In our context, no efficient shared-memory is available. The shared version will be visible for other processors only through network connections. The resulting communication costs need to be taken into account in our model. With this aim in view, we introduce delays.\\

The effect of delays for parallel stochastic gradient descent has already been studied for instance by Langford et al. in \cite{SlowLearners} or by Dekel et al. in \cite{DekelShamir}. However, these authors consider only the case where the computing architecture is endowed with an efficient shared memory. In this context, as outlined in \cite{SlowLearners}, computing the gradient is much more time-consuming than computing the update (which includes communication latencies). On the contrary, in the Cloud Computing environment, the gradient computation in a single point is much faster than the actual update, because of the communications. In this subsection, we improve the model of iterations \eqref{eq:correctAveraging} with non negligible communication costs, resulting in the following more realistic iterations \eqref{eq:delayedAveraging}.\\

For each time $t \geq0$, let $\tau^{i}(t)$ be the latest time before t when the unit $i$ finished to send its updates and received the shared version. At time $t=0$ we have $w^1(0) = \ldots =w^M(0)=w^{srd}$, and for all $i \in \{1, \ldots,M\}$ and all $t \geq0$,

%\begin{equation}\label{eq:delayedAveraging}
%\begin{cases}
%w^i_{temp} = w^i(t) - \varepsilon_{t+1} H\left(\zbf^i_{\{t+1\text{ mod }n\}},w^i(t)\right)& \\ %\label{eq:processing}\\
%w^i(t+1) = w^{i}_{temp} & \text{ if }  t \neq \tau^{i}(t) \\
%w^i(t+1) = w^{srd}(\tau^{i}(t-1)) - \Delta^i_{\tau^{i}(t-1) \rightarrow t}& \text{ if }  t = \tau^{i}(t) \\
%w^{srd}(t+1) = w^{srd}(t) - \displaystyle\sum_{j, t = \tau^{j}(t)}{\Delta^j_{\tau^{j}(\tau^{j}(t-1) -1) \rightarrow \tau^{j}(t-1)}}& \\
%\end{cases}
%\end{equation}\\

\begin{equation}\label{eq:delayedAveraging}
\begin{cases}
w^i_{temp} = w^i(t) - \varepsilon_{t+1} H\left(\zbf^i_{\{t+1\text{ mod }n\}},w^i(t)\right)& \\ %\label{eq:processing}\\
w^i(t+1) = w^i_{temp} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;  \text{ if }  t \text{ mod } \tau  \neq 0 \text{ or } t=0,& \\
    \begin{cases}
        w^{srd} =   w^{srd} - \sum_{j=1}^M{\Delta^j_{t-2\tau \rightarrow t -\tau} } & \text{ if } t \text{ mod } \tau =0 \text{ and } t \geq 2\tau,\\
        w^i(t+1) =  w^{srd} - \Delta^i_{t -\tau \rightarrow t}&  \text{ if } t \text{ mod } \tau =0 \text{ and } t \geq \tau.\\
    \end{cases}
\end{cases}
\end{equation}\\

The main difference with iterations \eqref{eq:correctAveraging} is the introduction of stale information: a dedicated process permanently modifies the shared version with the latest updates received from the processing units. This shared version is made available for every processing unit as a kind of consensus that emerges from the aggregation of the prototypes versions. Because of the communication delays now modeled, each processing unit can access to all the information of its own VQ procedure, but can only access stale gradient results from the other processing units through the shared version. This model easily extends to asynchronous schemes, as the aggregation process does not require the displacement terms from all of the processing units to proceed and can provide a partially updated shared version. This model is therefore close to the cloud asynchronous version developed in Chapter \ref{chap:cloudDALVQ}. Figure \ref{fig:delayedAveraging} gives a synthetic view of the iterations \eqref{eq:delayedAveraging}.\\

The performance curves resulting from the simulations of this distributed scheme are displayed in Figure \ref{fig:delayedParallel}. Remark that in our experiments small delays ($\tau=1$ or $10$) do not have any severe impact, since the results are similar to the non-delayed model reported in Figure \ref{fig:correctedParallel}. However, large delays $\tau =100$ can have a significant impact and prevent the distributed scheme from bringing speedup to convergence with more computing resources.

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.215]{../../Pictures/ParallelDALVQ/delayedGradientParallel.pdf}
\end{center}
\caption{Illustration of the parallelization scheme described by equations \eqref{eq:delayedAveraging}. The reducing phase is only drawn for processor $1$ where $t=2\tau$ and processor $4$ where $t=4\tau$.}
\label{fig:delayedAveraging}
\end{figure}

\begin{figure}[p]
\begin{center}
$\begin{array}{c}
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/delayedGradient1.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/delayedGradient10.pdf} \\
\includegraphics[scale = 0.5]{../../Pictures/ParallelDALVQ/delayedGradient100.pdf} \\
\end{array}$
\end{center}
\caption{Charts of performance curves for iterations \eqref{eq:delayedAveraging} with different numbers of computing entities, $M=1,2,10$. The three charts correspond to different values of $\tau$ ($\tau=1,10,100$).}
\label{fig:delayedParallel}
\end{figure}

\subsection{Comments}

The results presented in this chapter are rather insensitive to the experiment settings that we have chosen. In particular, the main conclusion is that averaging the results of multiple prototypes versions has not led to any speedup in our context ---probably because of the non-convexity of our empirical distortion--- but that the alternative parallelization scheme introduced in Subsection \ref{subsec:gradients} provides such a speedup. This conclusion has proved to be verified in numerous experiment settings on which we have not reported. More specifically, this conclusion is true for multiple choices of the decreasing learning rate $(\varepsilon_t)_{t>0}$ , for multiple choices of $G$, $d$, $\chi$, $\kappa$, $sc$. In the same way, the introduction of small randomness and asynchronism between the communication windows in model \eqref{eq:delayedAveraging} does not affect the conclusion.\\

Conversely, the choice of $\tau$ is of prime importance. While in the first two parallelization schemes, this value shows little impact on the speedup achieved, it becomes a decisive factor in the scheme presented by equations \eqref{eq:delayedAveraging}. Indeed, the use of stale information (or information with delays) leads to less effective schemes: such a conclusion has already been stated, for example in \cite{BBL} or in \cite{DekelShamir}. More specifically, the degree of delay that is endurable without suffering noticeable performance loss must be set with regards to both the time to compute a gradient on a single point and the number of points to be processed for the scheme to approach convergence. As an example, the delays that will be observed in the following chapter are much higher than $\tau=100$ but are compensated with much larger computation requirements before convergence.\\    