\section{Introduction}

The SaaS cloud solutions presented in Chapter \ref{chap:cloudpres} have already proved to be a successful economic and technological model, as demonstrated for example by Amazon or Gmail. In parallel, the IaaS cloud solutions have also proved to be good candidates for many customers, as suggested by the long list of some of Amazon Web Service customers\footnote{http://aws.amazon.com/solutions/case-studies/} or by their business volume (see \cite{AmazonBusiness}). In contrast, the PaaS solutions are younger and address a different market.\\

The PaaS cloud solutions are deemed to be easy to manipulate. Indeed, many tools are provided to improve the developer efficiency. In addition to these tools, many abstractions and primitives are already available to prevent the PaaS customers from re-implementing the same engineered solutions multiple times. For example, the Azure Queues provide an asynchronous mechanism that helps to build loosely coupled components at a low development cost. However, the development of cloud applications requires specific considerations when the application is expected to have a certain level of complexity or to achieve a certain scale-up. The recent interest for PaaS solutions has therefore driven some research about design patterns for improved cloud application development (see e.g. \cite{Helland} or \cite{CloudDP}).\\

In addition to the general Cloud Computing design patterns, the question of design patterns in the context of CPU-intensive cloud applications arises. To our knowledge, the Azure platform has not been originally designed specifically to host intensive computations. While the overall system has been designed to provide very satisfactory scale-up (in terms of the number of available computing units or of the aggregated bandwidth bounds) for many cloud applications, the Azure PaaS system does not provide all the specific software primitives of CPU-intensive applications would require. In this chapter, we investigate the design of large-scale applications on Azure and provide some cloud design patterns that have appeared as mandatory when working on these applications.\\

Since Azure has not been initially designed to be a scientific computing platform, no computation framework (with the exception of Windows Azure HPC presented in the introduction of Chapter \ref{chap:azurepres}) is presently available for Azure: Azure provides neither a ``low-level'' framework such as MPI, nor a ``high-level'' framework such as MapReduce. While the Azure team has reported that they are working on implementing the MapReduce framework for Azure, such a framework is not available yet. In this situation, one can only resort to the abstractions and the primitives which are already available (see for example Subsection \ref{sec_SynchroPrimitives}) to build CPU-intensive applications. These abstractions and primitives can turn out to be inadequate for CPU-intensive application development.\\

During my PhD work, I have been involved in four CPU-intensive applications. Two of them are the cloud Batch K-Means prototype and the cloud Vector Quantization prototype\footnote{Both of these prototypes are available at http://code.google.com/p/clouddalvq/} that are described in detail in Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ}. In addition to these two prototypes, I have worked on two applications run by Lokad that are not open-source. The first one is the Lokad time series forecasting engine: it is the scientific core of Lokad applications which is hosted on Azure and provides an API that is in charge of building and returning the actual forecasts. This engine is used in production and benefits from the scale-up elasticity and the robustness provided by Azure. The second application is Lokad benchmarking engine, which replicates the Lokad forecasting engine but which is used as an internal tool to monitor, profile and evaluate our forecasting engine in order to improve the accuracy of the forecasts, increase the scale-up potential and reduce the computation wall time (see definition in section \ref{sec_KMeans_SMP_Algo}). This application does not require the same robustness as the Lokad forecasting engine, but faces many challenges, such as a complex reporting process for which the parallelization is both mandatory and difficult.\\

In the light of the experience obtained while working on these four applications, this chapter describes engineering considerations about the software development of cloud applications with a specific focus on CPU-intensive issues when it is relevant. It is organized as follows. Section \ref{sec_DPCommunications} describes how the communications can be implemented between the different processing units of a given worker role. Section \ref{sec_DPArchitecture} presents recurring elements of cloud application architecture design. Section \ref{sec_DPScaling} presents some elements of the design required to provide the Azure applications with scalability. Section \ref{sec_DPAdditional} provides additional elements, especially the idempotence constraint. Section \ref{sec_Counters} presents a synchronization primitive that happened to be necessary in many situations we encountered.\\

\section{Communications}
\label{sec_DPCommunications}
\subsection{Lack of MPI}

The Message Passing Interface (MPI) is an API standard that defines the syntax and semantics of libraries that provide a wide range of routines to automatically harness direct inter-machine communications in a distributed architecture. More specifically, the various MPI implementations provide an abstraction layer that disburdens the application developer from manual management of inter-machine communications. In Section \ref{Distributed K-Means Algorithm Section}, we highlight the importance of MPI for the distributed Batch K-Means on Distributed Memory Multi-processors (DMM) architectures. We show how significant the impact of the actual implementation of MPI primitives is on the speedup that can be achieved by the distributed Batch K-Means on DMM architectures: this is the tree-like topology of the communication patterns of MPI primitives that results in a $O(log(M))$ cost for averaging the prototypes versions where $M$ is the number of processing units.\\

Direct inter-machine communications are also available in Azure. As explained in Section Azure Compute (Section \ref{sec_AzureCompute}), an internal endpoint is exposed by each processing unit of Azure so that each of the processing units could talk directly to the others using a low-latency, high-bandwidth TCP/IP port. The bandwidth of these direct communications is very sensitive to multiple factors such as the number of units that are communicating at the same time (because of aggregated bandwidth bounds), the fact the resource manager may allocate other VM instances from other deployments (applications) on the same physical hardware, the fact these other VM may also be I/O intensive, etc. Yet, the average behavior of such direct communications is very good: \cite{Early_Observations_On_Azure} and \cite{AzureScope} report direct inter-machine communication bandwidth from 10 MBytes/sec to 120 MBytes/sec with median measurement bandwidth of 90 MBytes/sec.\\

While these direct inter-machine communications are available on Azure, no framework has already been released on Azure to provide higher level primitives such as the merging or the broadcasting primitives provided by MPI.\\

A key feature of the Azure VM system is that the cloud client cannot get any direct topology information about the VM he temporarily owns. On the contrary, the VM are totally abstracted to disburden the cloud client from these considerations. Such a design does not prevent from building a MPI-like API but makes it more difficult. Several works have ported MPI on Cloud Computing platform such as EC2 (we refer the reader for example to \cite{CloudMarinescu} or \cite{MpiEc2}) but no MPI implementation is available on Azure.\\

Because of the importance of bandwidth in the speedup performance of many distributed machine-learning algorithms such as Batch K-Means, and because the direct inter-machine bandwidth is much higher than the bandwidth between the storage and the processing units (see Subsection \ref{sec_AzureBenchmark}), MPI would be a very useful framework for Azure, as far as intensive computing is concerned.\\

\subsection{Azure Storage and the shared memory abstraction}

The Azure Storage service provides a shared memory abstraction in the form of the BlobStorage (and of the TableStorage, but because of their similarity we only mention BlobStorage in this subsection). The different processors of a Symmetric Multi-Processors (SMP) system (see Section \ref{Distributed K-Means Algorithm Section}) can access the shared memory implemented in RAM to read or write data and therefore use this shared memory as a means of communication between the different processors. In the same way, all the different computing units of an Azure application can access any of the blobs stored into the BlobStorage.\\

There is however a critical difference between the use of a shared memory in a SMP system and the use of the BlobStorage as a shared memory in a cloud environment: it is the latency and the bandwidth. Indeed, in the SMP case, each processor is very close to the shared memory and efficient means of communication such as buses are used to convey the data between the shared memory and the processors. In contrast, the BlobStorage data are stored on different physical machines from those hosting the processing VM. Therefore, the communication between the computing units and the BlobStorage are conveyed through TCP/IP connections between the distant machines. The resulting bandwidth is much lower than the bandwidth of a real shared memory: around 10MBytes/sec for reads and 3 MBytes/sec for writes (see Subsection \ref{sec_AzureBenchmark}). In return, the communication through the BlobStorage provides persistency: if a worker fails and a task is restarted, many partial results can be retrieved through the storage, therefore mitigating the delay for job completion induced by this failure.\\

\subsection{Workers Communication}

The previous subsections have presented two approaches for the communications: the first one ---a direct inter-machine mechanism through IP communications--- is efficient but made difficult by the lack of any abstraction primitive as MPI would provide: re-implementing MPI-like primitives in an environment where the computing units (which are VM) are likely to fail, to be rebooted or to be moved from a physical device to another one, would be a demanding task that would be similar to implementing a peer-to-peer network. The second approach, which consists in communications through the BlobStorage, is designed to be both simple and scalable, but is inefficient (because of the low bandwidth between the workers and the storage).\\

Both approaches seem valid but lead to different designs, challenges and performances. We have made the choice of resorting to the BlobStorage and to the QueueStorage as communication means between workers because of the two following reasons. Firstly, beyond the pros and cons of each approach, the communication through storage seems to be more suited to the Azure development philosophy. Indeed, this is the way communications are made in every Microsoft code sample provided in tutorials, and it is also suggested by the Azure QueueStorage, which has been specifically designed for this approach. Secondly, because of its simplicity over the ``peer-to-peer'' communication model, the communication through the storage model is the one that has been chosen by Lokad for both the forecasting engine and the benchmarking engine. Resorting to the same mechanism for our clustering prototypes was therefore an industrial constraint.\\
%
%Azure does not offer currently any standard API for distributed computation, neither a low
%level one such as MPI \cite{MPI1996}, nor a more high level one such as Map
%Reduce \cite{MapReduce2004} or Dryad \cite{Dryad2007}. MapReduce could be
%implemented using Azure components (following the strategy of
%\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{Graphlab2010},
%those high level API might be inappropriate for iterative machine learning
%algorithms such as the k-means or learning vector quantization because of MapReduce overhead on synchronisation and waiting process.
%
%We therefore need to explicitly design our communication process between Azure workers, process that can realized by 2 means :
%
%\begin{enumerate}
%\item Workers can communicate each other through direct IP communications, that is to say re-implement a peer-to-peer network inside Azure cloud. TO DEVELOP
%\item Azure storage system can be used to implement synchronization and
%communication between workers : each worker is pinging (or "listening") a specific queue, and it gets some message from this queue holding
% some blobname in BlobStorage where it can retrieved the data it needs.
%\end{enumerate}
%
%While the first solution comes with higher performance in terms of bandwidth, as reported in AzureScope or ... (CITATION NEEDED), it implies a very tough work which would be like implementing MPI on Azure, which was outside the scope of this thesis. Moreover, the "storage approach" for communication is much more in agreement with Azure's abstractions and design thinking (CITATION NEEDED).
%Thus, we will rely therefore directly on Azure queues and blob storage to build all our communication systems.

\subsection{Azure AppFabric Caching service}
\label{sec_AzureCaching}
Caching is a strategy that consists in storing data in the RAM of a machine instead of storing it on a hard-drive. Thanks to this strategy, the data can be accessed more quickly, because one does not need to pay for the cost of hard-drive accesses. The caching mechanism is therefore known to improve application performance.\\

Some cloud providers provide a caching service, such as Google App Engine, AppScale and Amazon Web Services (through Amazon Elasticache). In the same way, the Azure AppFabric Caching service (Azure Caching for short) provides a caching solution for the Azure applications. This system became available only after a large part of our clustering algorithm implementations had been developed and benchmarked. As a consequence, it has not been used in the cloud algorithms presented in Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ}. We believe the Caching service may yet lead to significant throughput increase, especially during the prototype versions communication between the processing units (see e.g. Chapter \ref{chap:KMeans}).\\

\section{Applications Architecture}
\label{sec_DPArchitecture}
\subsection{Jobs are split into tasks stored in queues}

Following the terminology of the original MapReduce research paper (\cite{MapReduce2004}), the total execution of an algorithm is referred to as a job. A given job is divided in multiple tasks that stand for the elementary blocks of logic that are executed. Each task is run by a single worker, and a single worker can only process one task at a time. During the duration of an entire job, each processing unit is expected to run successively one or several tasks.\\

If the number of tasks of a given job is lower than the number of processing units, or if one or several workers are temporarily isolated from the rest of the network, some workers may process no task during a given job execution.\\

The multiple tasks are described by messages stored in queues. More specifically, to each task is associated a message queued in the QueueStorage. When a processing unit is available, it pings a queue and dequeues a message that describes a specific task. Once the task is completed, the corresponding processing unit pings a queue to get a new message and then processes the new corresponding task. The number of tasks does not need to be defined when the job is started. On the contrary, a common Azure application design is that many tasks, once completed, produce one or several new tasks in return .\\

Among all the tasks that are processed during a given job, many of them refer to the same logical operation applied on multiple distinct data chunks of the same type, a situation often described in the literature as a data-level parallelism. To reflect this data parallelism, a frequently chosen design consists in gathering in the same queue only the messages whose corresponding tasks refer to the same logical operation applied on distinct data chunks. In such a design, to each queue is associated a QueueService which holds the logical operations to be applied on each item of the corresponding queue.\\

\subsection{Azure does not provide affinity between workers and storage}
\label{sec_WorkersAffinity}
As outlined in the original MapReduce research paper (\cite{MapReduce2004}), \textit{``Network bandwidth is a relatively scarce resource in [our] computing environment''}. The MapReduce framework has therefore adopted a design where the data are not sent to the processing units, but the code to be executed is sent by the framework environment on the machines actually storing the data to avoid network contention (see also Subsection \ref{sec_MapreduceConsiderations}). This strategy is often referred to as data/CPU affinity.\\

Presently, there are no mechanisms in Azure to provide such an affinity between the workers and the storage. Indeed, data are accessed through the BlobStorage and the TableStorage but the machines that actually store the data (and run the stream layer described in Subsection \ref{sec_StreamLayer}) are totally abstracted. As a result, we cannot run any tasks on the machines where the corresponding piece of data are stored. Each worker processing data will therefore need to first download the data it needs to process, contrary to Google's MapReduce, with which the tasks are scheduled and assigned on each worker by the framework in such a way that data downloading is minimized.\\

%Such a behavior is intended by Microsoft to simplify cloud application development: one does not need to design its application to take data location into account, Azure providing a satisfactory scale-up and a sufficient bandwidth in many conditions. Besides, the lack of data/CPU affinity allows the user to consider the mappers as equivalent, which also simplifies a lot the tasks scheduling (any processing unit can process any task indifferently). This design choice is therefore about simplicity over performance.\\

\subsection{Workers are at first task agnostic and stateless}
\label{sec_WorkersStateless}

When a worker role (see Subsection \ref{sec_AzureCompute}) is deployed, a certain number of processing units (the role instances) which are controlled by the customer are assigned to the worker role to run the underlying code instructions. In addition to the event of a role instance failure, the Windows Azure Fabric can arbitrarily and silently move a VM from a physical machine to another one. Because of these events, any role instance needs to be resilient to reboot or displace, and therefore must be stateless, i.e. it must not store, between two tasks, data (in RAM or in the local hard-drive) that are essential to successfully complete the worker role logical operations.\\

Let us describe three properties shared by all the role instances of a given worker role. Firstly ---as just explained--- the role instances are stateless. Secondly, no instance role stores locally a part of the data to be processed (see Subsection \ref{sec_WorkersAffinity}). Finally, all the role instances have the same size (small instances/medium instances/big instances, etc.). As a consequence of these three properties, all the role instances are exchangeable and the mapping between the tasks and the role instances can be done arbitrarily without undergoing any performance drop.\\

This behavior has been observed in the development of Lokad forecasting engine or Lokad benchmarking engine. However, we have found out that in the context of our clustering algorithms a significant reduction of role instances I/O could be obtained when slightly relaxing the stateless hypothesis and partially dropping the previous commutability characteristic.\\

Indeed, many distributed machine-learning algorithms require that the total data set should be split among the multiple processing units so that each processing unit loads into its own local memory a part of the total data set, and that multiple computations are made on this partial data set. In this situation, reloading the partial data set for each computation to be done leads to a very large I/O overhead cost with the storage. This overhead cost can be significantly reduced by making the processing units keep the partial data set within their local memory until the algorithm is completed. When adopting this design, we reintroduce a data/CPU locality by assigning to each processing unit all the computations related to a specific partial data set. This design, close to the memoization technique, is used in the algorithms presented in Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ}. More specifically, each processing unit is assigned to the processing tasks of a specific partial data set. When the partial data set is needed, the processing unit checks if the data set is already available in the local memory. If the data set is not available, it is downloaded from the storage and kept into memory for the subsequent uses.\\

\section{Scaling and performance}
\label{sec_DPScaling}
\subsection{Scaling up or down is a developer initiative}

Through the management API or the Azure account portal, the Azure account manager can modify the number of available workers. Contrary to many other hardware architectures, the application scaling-up-or-down is therefore left to the developer's choice.\\

In some cases, the estimation of an adequate number of processing units may be deduced by the initial data set to be processed. As an example, Chapter \ref{chap:KMeans} provides an algorithm for which the knowledge of the data set size and of the framework performance (CPU+bandwidth) is sufficient to determine an adapted number of processing units. This is also the case of the Lokad forecasting engine in which the knowledge of the data set size (number of time series and length of each time series) is sufficient to determine how many workers are necessary to return the forecasts in one hour.\\

In other cases, the application scaling requirement may be more difficult to anticipate. In these cases, the auto-scaling system may rely on the approximative estimation of the queue size described in Subsection \ref{sec_QueueSize}. More specifically, an adaptive resizing strategy may be adopted in which the number of processing units is increased as long as the number of the processing messages in the queues keeps growing, and in which the number of processing units is decreased when the number of queues messages is decreasing. Because of the friction cost of worker resizing (see Subsection \ref{sec_WorkersCount}), this resizing strategy is adapted only on the applications that are run during hours on end.\\

Depending on the granularity level set by the developer, the downsizing of the workers pool may be delayed until the algorithm is completed. Indeed, the Azure Management API does not provide any way to choose the workers to be shut down when downsizing the workers pool. If the granularity level is too coarse, the chance of shutting down a worker comes at a cost that may deter the user from downsizing the workers pool before all the computations are completed.\\

\subsection{The exact number of available workers is uncertain}
\label{sec_WorkersCount}

Several reasons may affect the number of available processing units. Firstly, following a resizing request, the re-dimensioning  of role instances follows a general pattern in which a large majority of workers are allocated within 5 to 10 minutes (see for example \cite{Early_Observations_On_Azure}), but the last workers to be instantiated may take as much as 30 minutes before becoming available (in the worst encountered case). Secondly, a worker can be shut down by the Azure Fabric, the hosting VM can be moved on a different physical machine, or the physical machine can die because of a hardware failure. Finally, each processing unit may become temporarily unavailable (for example because of a temporary connectivity loss). Therefore, the number of available processing units may vary over time.\\

As a consequence, no algorithm should rely on an exact number of available processing units. If this number exceeds the quantity of workers expected by the algorithm, this excess of computing power often results in a waste: the additional workers are not of any help (this is the case for example in the algorithm presented in Chapter \ref{chap:KMeans}, in which the initial data set is split into $P$ data chunks processed by $P$ workers, and additional workers have no impact on the algorithm speedup). In contrast, the lack of any expected processing unit often results in dramatic performance reduction: in the event of a synchronization barrier, the lack of a single processing unit may result in a completion taking twice the expected time or no completion at all depending on the implementation.\\

Because of this cost asymmetry, a frequent design is to request a number of processing units slightly higher than the number of processing units that will be used by the distributed algorithm.\\

\subsection{The choice of Synchronism versus Asynchronism is about simplicity over performance}
\label{sec_SynchronismAsynchronism}

While migrating Lokad forecasting engine and Lokad benchmarking engine to the cloud, we have observed a recurring pattern in which a part of the application could be schematized as a sequential set of instructions ---referred to as$I_1,I_2,I_3, etc.$--- applied independently on multiple data chunks, referred to as $D_1,D_2,D_3, etc.$. A specific task is associated to each pair of instruction and data chunk.\\

In this situation, no logical constraint requires the instruction $I_j$ (for $j>1$) to wait for the instruction $I_{j-1}$ to be applied on every data chunk before $I_j$ is applied on any chunk. Yet, we have observed that in this situation, adding artificial synchronization barriers to ensure that the requirement described above is guaranteed vastly contributes to simplify the application design and debugging. Provided the number of tasks for each instruction significantly exceed the number of processing units, the relative overhead of these artificial synchronization barriers is kept small.\\

On the contrary, when the tasks that can be processed in parallel are tailored to match the number of available processing units, the overhead of synchronization induced by the stragglers (see Subsection \ref{sec_MapreduceConsiderations} or \cite{LooseSynchronization}) may be high, as described in our cloud-distributed Batch K-Means chapter (see Chapter \ref{chap:KMeans}). In this situation, the overhead is actually so big that it leads us to redesign the algorithm to remove this synchronization barrier (see Chapters \ref{chap:practicalDALVQ} and \ref{chap:cloudDALVQ}).\\

\subsection{Task granularity balances I/O costs with scalability}

The choice of task granularity is left to the developer. It results in a tradeoff between scalability and efficiency because of overhead costs. Indeed, the coarser the granularity, the less the processing units will pay for I/O with the storage and for task acquisition through the queue-pinging process. But the coarser the granularity, the less additional processing units may contribute to the speedup improvement and the more sensitive the overall algorithm will be to stragglers.\\

The design of the clustering algorithms of Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ} has made the choice of very coarse-grained granularity: each of the processing unit is expected to process only one task for the total algorithm duration, to minimize the heavy storage I/O related to the data set download (see Subsection \ref{sec_WorkersStateless}). In contrast, the Lokad forecasting engine and the Lokad benchmarking engine have been tuned to use a much finer granularity. In both cases, a general design pattern has been implicitly followed: the granularity is minimized under the condition that the I/O overhead induced by the granularity choice does not exceed a few percents of the total algorithm duration.\\

\section{Additional design patterns}
\label{sec_DPAdditional}

\subsection{Idempotence}
\label{sec_idempotence}

Multiple definitions of idempotence have been stated, but we refer the reader to the definition of Helland in \cite{Helland}: \textit{``idempotence guarantees that the processing of retried messages is harmless''}. The idempotence question arises in many cloud application situations when the messages queued can be processed multiple times.\\

Let us consider the cloud context of message processing: a processing unit consumes a message from a queue and then updates some data stored in a durable way (for example in the BlobStorage). The message is first consumed by the worker, communication is then made between the worker and the BlobStorage, and data are updated in the BlobStorage. After the updating of the data, the worker may die before it tells the queue that the message has been successfully processed and should be deleted. Queues have been designed to react to machine failures in an ``at-least-once'' way: in the event of a machine failure, the message is re-queued after some timespan and another worker will process the same message, resulting in a second updating operation being performed on the same data stored in the BlobStorage.\\

Technically, this issue derives from the fact there is no direct mechanism to bind the message consumption and deletion with the message processing made by the worker so that one of these two events cannot happen without the other. The absence of this coupling leads to failure windows between the end of the data update and the message deletion in which the message is delivered and processed more than once. This scenario is illustrated in Figure \ref{fig:idempotence}.\\

\begin{figure}
\begin{center}
\includegraphics[height=16em]{../../Schemes/idempotencyIssues2.pdf}
\end{center}
\caption{Multiple scenarii of message processing that impact the BlobStorage}
\label{fig:idempotence}
\end{figure}

Because of this ``at-least-once'' design, what Helland refers to as the recipient entity (which is the BlobStorage entity that may be subject to multiple updates) must be designed so that the repeated processing of a single message does not lead to a state for this entity which would be different from the one which could have been expected with a single processing. In practice, lots of message processing of our cloud applications are intrinsically idempotent. In particular, provided the processing is a deterministic function of the input message and that neither the entities that may be used nor the internal worker state may be modified, the message processing is idempotent. Among the tasks that suit this pattern, the most widespread case is the design of a work-flow in which each message processing is a deterministic function of its input and each entity stored in the permanent storage is \textit{read-only}, i.e. it is not modified once it has been created and written into the storage for the first time. In this case, when re-processing a message, the output is necessarily the same as the one obtained the first time, and the same blob is rewritten with the same value at the same place.\\

An important case that does not suit this pattern is the message processing that consists in a blob update, and in particular counter increments and decrements. In this case, a solution consists in the design of the recipient entity (i.e. the counter) so that the counter remembers the previous increments and decrements already made and that
the update primitives are idempotent. The particular case of counter increments and decrements is explored in Section \ref{sec_Counters}.\\

\subsection{Queues size usage}
\label{sec_QueueSize}
The Azure QueueStorage API provides a method to get a rough estimation of the number of messages actually stored in a given queue, but no exact count is provided by the API. As a result, the API cannot be used as a synchronization barrier to start a process when $x$ messages have been processed in a specific queue. In contrast, the approximative estimation of a queue size may be used to estimate the load of the corresponding QueueService.\\

%\subsection{Pinging queues as a tradeoff between cost and simplicity}
%
%Workers will be pinging queues to detect if there are some messages to process. Since even QueueStorage operations are charged, we can wonder whether pinging queues can be avoided, and whether it is affordable. Indeed, we could open a TCP connection on each worker, set-up a listener so each worker is listening for notification. Each time a notification is sent, the listener tells the worker to ping the queue and get the message, or the notification is the message by itself and worker is directly told to process the message. In the first case, we do not ping queues unless a message needs to be processed. In the second case, we do not ping queues at all. We have been choosing to not implement listeners patterns and TCP connections, for simplicity reasons. Pinging a queue every 100 ms for a worker means a 315 millions request per worker per year, charged as 3000 dollars (see \cite{Azure_Pricing}). We have been adopting a back-off strategy : each time we ping an empty queue, we double time before pinging again the queue CHECK THIS IN LOKAD.CLOUD. IN THE EXPERIMENTATION PART, GIVE HOW MANY TIMES THE QUEUE IS PINGED



\subsection{Atomicity in the BlobStorage}
\label{blobStorage_atomicity}

As already stated in Subsection \ref{partition_name}, the Azure BlobStorage does not provide any atomicity for transactions implying multiple blobs. Indeed, as outlined for example in \cite{Helland}, \textit{``a scale-agnostic programming abstraction must have the notion of entity as the boundary of atomicity''}.\\

Despite this framework impossibility, it is still possible to design applications so that the transactions implying multiple objects (for example class objects) can be done atomically. The first solution consists in storing the multiple objects in a single entity (i.e. in a single blob, as far as the BlobStorage is concerned). This solution leads to some overhead cost: the read (resp. the update) of any of the objects stored in an entity requires the whole entity to be downloaded (resp. downloaded then re-uploaded). When the design of the application results in a situation in which the read/update of one of the objects stored in an entity and the reads/updates of the other objects stored in the same entity always happen simultaneously, this overhead is negligible.\\

A second solution consists in resorting to an indirection through a kind of pointers. Let us examine the case of two objects stored in two different entities and that may be both involved in a single update operation requiring atomicity. In this context, the storage locations of the two entities may be unknown by the processing units and only accessed through a third entity that stores location of the two previous entities; this third entity being referred to in the following as the pointer entity. When a given processing unit needs to update one entity (resp. the two entities), it accesses the pointer entity, downloads the entity (resp. the two entities), locally updates it (resp. them), and re-uploads it (resp. them) into the storage in a new location (resp. two new locations). Then it updates the pointer entity so that it stores the updated location (resp. locations). In the event of a concurrent update of one or the two entities through the same mechanism, the Read-Modify-Write (RMW) primitive described in Chapter \ref{chap:azurepres} guarantees that one update will succeed while the other will fail and will be restarted on the updated pointer entity and on the updated entity (resp. entities). Due to the indirection cost introduced by the pointer entity, this second solution use cases appear to be rare.\\

In the algorithms presented in Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ}, the prototype versions and their corresponding weights are stored together in a single entity, following the first solution. On the contrary, sometimes the atomicity is not mandatory, as shown through the example detailed in Subsection \ref{sec_BitTreeCounter}.\\

\subsection{Lokad-Cloud}
\label{sec_LokadCloud}

\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}} is an open-source framework developed by Lokad that adds a small abstraction layer on top of the Azure APIs to ease the Azure workers startup and life cycle management, and the storage accesses. It has been awarded the 2010 Microsoft Windows Award. A slightly customized Lokad.Cloud framework has been used to build our cloud prototypes described in Chapters \ref{chap:KMeans} and \ref{chap:cloudDALVQ}.

\section{The counter primitive}
\label{sec_Counters}

\subsection{Motivation}

While Azure is deemed to provide many primitives and abstractions that ease the application development, it does not presently provide a synchronization barrier primitive that is lifted once a given set of tasks has been completed. To implement this synchronization barrier we choose to resort to a concurrent counter stored in the BlobStorage\footnote{As explained in Subsection \ref{sec_AzureCaching}, this counter would probably vastly benefit from being stored in the Azure Caching service using the same implementation strategy than the one described below, but this service was released too late for our implementations.} . This counter is initially set to the number of tasks that need to be processed and is decremented every time a task is completed. When this counter hits $0$, all the tasks have been completed and the synchronization barrier can be lifted.\\

The design of synchronization barriers and counters in a distributed world has been studied in many contexts. In particular, special attention is paid to avoid memory contention in order to achieve overall slow latency and high throughput. To alleviate the memory contention, a classical pattern is to resort to a combining tree structure of counters that distributes the data requests on multiple pieces of data instead of a single one. This pattern, referred to as \textit{software combining} in \cite{AMP} has been used for more than a quarter of a century (see e.g. \cite{HotSpot}, \cite{Goodman89efficientsynchronization}, \cite{Herlihy95scalableconcurrent} or \cite{AMP}).\\

\subsection{Sharded Counters}

Before presenting the adopted solution, let us describe a well-known distributed counters used in many cloud applications: the sharding counters (\cite{ShardingCounters}). The sharding counters consist in a very simple set of distributed counters designed to limit the memory contention and therefore achieve a much higher update rate. Instead of using a single counter that is incremented or decremented, the sharding counter owns $L$ different counters referred to as shards. To increment or decrement the counter, a shard is chosen at random and this shard is incremented or decremented accordingly.\\

Sharding counters are targeted to address the use-case in which the actual value of the counter may be slightly outdated or non-consistent when it is read, and the write operations (increment or decrement) are much more frequent than the read operations. Indeed, since the counter is split into $L$ shards, the read operation requires to read the actual $L$ values and to add them up. A well-known example of sharding counters use-case is the monitoring of online users for a given application to adapt the quantity of servers with the traffic-load. In this scenario, the traffic-load may be approximated with several percent of error margin without damaging the servers dimensioning logic.\\

In the context described in the previous subsection, the counter is read after each decrement to check if the value 0 has been hit. The sharding counters abstraction would therefore lead in our situation to unsatisfactory read latencies.\\

%\subsection{BitTreeCounter}
%\label{sec_BitTreeCounter}
%Based on the observation that sharded counters were not suitable to our purpose and that no other cloud counter abstraction was available, we have developed a new implementation of combining tree counters, referred to in the following as BitTreeCounter. In addition to the large throughput and the low latency required for many distributed counters, we also require our BitTreeCounter to be adapted to the idempotence phenomenon described in Subsection \ref{sec_idempotence}: the same task processed several times should result in a single decrement of our counter.\\
%
%This additional condition requires that tasks should be given an Id (to fix ideas, the tasks number is set to $N$ and task Ids set in the range 0 to $N-1$), and that either the counter or the processing units using the counter should own the list of the tasks already completed at least once (and therefore for which the counter has already been decremented). For an obvious reason of design simplicity, this information is remembered on the counter side.\\
%
%This BitTreeCounter is designed to handle efficient optimistic read-modify-write concurrent operations (see Subsection \ref{sec_SynchroPrimitives}) by implementing a set of dependent counters to avoid collisions and remove the contention issues while dealing with these optimistic read-modify-write operations. More specifically, the BitTreeCounter is a tree of nodes, where each node has a counter and is stored inside the BlobStorage in a distinct entity. The counter stored in each node is composed of an integer storing the actual counter value, and of a bit array that stores which task completion has led to the decrement. Thus, if a decrement request corresponding to a specific task completion has already been registered, the integer is not decremented again.\\
%
%Each node, with the exception of the leaves, has $C$ successors. The tree depth $H$ is set such as $H = \lceil\frac{log(N)}{log(C)}\rceil$. The counter of each node is initially set to its maximal value $C$. When a node is decremented and hits the 0 value, its predecessor is decremented. If the root node hits 0, all the $N$ have been completed and the synchronization barrier is lifted.\\
%
%Note that our BitTreeCounter may happen to be in a incoherent state. Indeed, since the multiple nodes are stored in distinct entities and since the second solution presented in Subsection \ref{blobStorage_atomicity} has not been implemented, a given counter node may be completed while the counter of the node predecessor has failed to be updated. The choice not to resort to the second solution of Subsection \ref{blobStorage_atomicity} is about efficiency over coherence. In our situation, this potential incoherent state does not result in any problematic behavior: since the counter decrement is part of each task, the failure to complete the total decrement process results in a task failure; the task will be re-processed and the counter will then be updated to recover its correct state.
%
%\begin{figure}
%\begin{center}
%\includegraphics[height=17em]{../../Pictures/Cloud/BitTreeCounter2.pdf}
%\end{center}
%\caption{Distribution scheme of our BitTreeCounter. Each node holds a bit array listing which tasks have already been completed. In the current situation, all the tasks attached to counters at node $N$ and $P$ have already been completed, and counter $E$ has been updated accordingly. A single task is missing for the $O$ counter to be completed too. When this task will be completed, counter $E$ will be updated accordingly. This event will result in the subsequent update of counter $B$. When counter $A$ is set to $\{0.0.0\}$, then all the tasks have been completed at least once and the synchronization barrier can be lifted.}
%\label{fig:bitTreeCounter}
%\end{figure}


\subsection{BitTreeCounter}
\label{sec_BitTreeCounter}

Based on the observation that sharded counters were not suitable to our purpose and that no other cloud counter abstraction was available, we have developed a new implementation of combining tree counters, referred to in the following as BitTreeCounter. The BitTreeCounter is designed to be a synchronization barrier primitive: the counter must hit 0 only when all the tasks of a given set are completed.

Our BitTreeCounter is designed to handle efficient optimistic read-modify-write concurrent operations (see Subsection \ref{sec_SynchroPrimitives}) by implementing a set of dependent counters to avoid collisions and remove the contention issues while dealing with these optimistic read-modify-write operations. \\

In addition to the large throughput and the low latency required for many distributed counters, we also require our BitTreeCounter to be adapted to the idempotence phenomenon described in Subsection \ref{sec_idempotence}: the same task processed several times should result in a single decrement of our counter. This additional condition requires that tasks should be given an Id (to fix ideas, the tasks number is set to $N$ and task Ids set in the range 0 to $N-1$), and that the counter should own the list of the tasks already completed at least once (and therefore for which the counter has already been decremented).\\

Let us now describe the actual implementation of our BitTreeCounter. Our counter is designed so a single method can be used by the counter user: it is the decrement method that takes in input the task Id that is completed, and return a boolean to tell if all the tasks have been completed and therefore if the synchronization barrier is lifted. Internally, the BitTreeCounter is a tree of nodes, where each node is a counter and is stored inside the BlobStorage in a distinct entity.\\ 

Each node is composed of an integer storing the actual node value, and of a bit array that stores which task completion has led to the decrement. Thus, if a decrement request corresponding to a specific task completion has already been registered, the integer is not decremented again.\\

Each node, with the exception of the leaves, has $C$ successors. The tree depth $H$ is set such as $H = \lceil\frac{log(N)}{log(C)}\rceil$. The counter of each node is initially set to its maximal value $C$. When a node is decremented and hits the 0 value, its predecessor is decremented. If the root node hits 0, all the $N$ have been completed and the synchronization barrier is lifted.\\

Note that our BitTreeCounter may happen to be in a incoherent state. Indeed, since the multiple nodes are stored in distinct entities and since the second solution presented in Subsection \ref{blobStorage_atomicity} has not been implemented, a given counter node may be completed while the counter of the node predecessor has failed to be updated. The choice not to resort to the second solution of Subsection \ref{blobStorage_atomicity} is about efficiency over coherence. In our situation, this potential incoherent state does not result in any problematic behavior: since the counter decrement is part of each task, the failure to complete the total decrement process results in a task failure; the task will be re-processed and the counter will then be updated to recover its correct state.

\begin{figure}
\begin{center}
\includegraphics[height=17em]{../../Pictures/Cloud/BitTreeCounter2.pdf}
\end{center}
\caption{Distribution scheme of our BitTreeCounter. Each node holds a bit array listing which tasks have already been completed. In the current situation, all the tasks attached to counters at node $N$ and $P$ have already been completed, and counter $E$ has been updated accordingly. A single task is missing for the $O$ counter to be completed too. When this task will be completed, counter $E$ will be updated accordingly. This event will result in the subsequent update of counter $B$. When counter $A$ is set to $\{0.0.0\}$, then all the tasks have been completed at least once and the synchronization barrier can be lifted.}
\label{fig:bitTreeCounter}
\end{figure}



%
%
%
%Simple sharded thread-safe idempotent tree counter shared among workers
%through the blobStorage. BitTree counter provides an abstraction to hide complexity
%related to performance handling.
%
%The usecase for this counter is synchronisation barrier between multiple
%tasks to be processed. Class is designed to avoid bandwidth contention, minimize
%overall latency of decrementation, and ensure idempotency (a job marked as completed
%multiple times should not hurt logic). The counter is incremented once of a quantity
%called "capacity", and then decremented "capacity" times. Each time the counter is
%decremented, we need to check if the internal value hit 0 to remove synchronization
%barrier. Thus, simple sharded counters would not have been efficient enough : because
%it would have required we read all the counters each time we decrement one of the counters.
%
%This bitTreeCounter is designed to handle efficient optimistic read-modify-write concurrent
%operations by implementing a tree topology of dependant counters to avoid collisions and remove
%all the contention issues while dealing with these optimistic read-modify-write operations.
%
%Typical use of this counter is to create synchronisation barrier for jobs among a same service :
%we first define the number of messages(=jobs) to be executed and set all the bit of the counters to true.
%Then we run the jobs and at the end of each job, we let the counter know that
%the job i has been completed at least once. The bitTreeCounter then set the bit index i to false. When all
%the bit have been marked as false, the bitTreeCounter returns true once and only once.
%The idempotency policy handles 2 scenarios :
%1) a job is run twice and we don't want the counter to be decremented twice. (Idempotency)
%2) a job is run twice and we don't want the counter to hit -1 after all the decrementation
% (would trigger next service twice and therefore double the workload).
%
%For n decrements, we have a tree depth of about log(n)/log(20). When a counter hits 0, it decrements the counter above him.
%When the only counter at depth 0 hits 0, everything is completed.
%
%It is not possible to get an exact count of how many messages are stored in some queue, we only can get an approximative count of queued messages when asking to the queue its metadata. When a service needs to process P messages, then need to push a message in another queue to launch a new service, we need to get an exact count of remaining messages to process. To achieve this, we are using blobcounters, blobs storing an integer and decremented by each worker using etag properties when the worker is done with a message. Since the worker can fail at any moment, and a failure means the message is requeued after some delay and is going to be processed again, we decrement the counter in the end, when the job is finished and the message is going to be deleted. Thus, the only way a single message can lead to decrement twice the counter is to make the worker fail exactly when it is noticing the queue to delete the message. Theoretically, the counter could therefore be decremented twice (with a very low probability). One way to deal with this issue would be to use an idempotent counter :
%    \begin{enumerate}
%    \item we push P messages in the queue. Each message has a unique Id between 1 and P
%    \item we push a binary counter of P bits set to 1111..1111 in the blobStorage
%    \item each time the job stored in message i is completed, we update using etag the counter and apply to the counter a \& operation 111110111111 where 0 is at index i. This way we made a counter where decrements will be idempotent.
%    \end{enumerate}
%    Even if getting number of messages stored in a queue is approximate, Azure is providing a transactional count of how many times a message is dequeued. FINISH HERE
%
%



