\section{Introduction}

For the last twenty years, the technological improvements in computing hardware have led to a situation where a gigantic amount of data can be gathered, stored and accessed. The amount of processing power required to explore and use these data largely exceeds the capacity of a single retail computer (referred to in the following as commodity hardware). Therefore, several computing systems have been designed to tackle this issue. In the 80's, more powerful systems known as supercomputers were created by improving custom hardware: this solution is referred to as scaling-up. The present trend is on the contrary to run these broad computations on a large set of commodity CPU working together, a solution referred to as scaling-out. In other words, processing more data nowadays often consists in throwing more commodity hardware at the problem.\\

The parallelization of computation on a large amount of machines gives rise to many problems, such as communications between computers, access to shared resources, or workload balance. The past two decades have been constantly providing engineered solutions to these issues, some of them known as Grid Computing and Peer-to-Peer architectures. These solutions have led to well-known software applications or results that have a deep impact on our everyday life, as is the case for Napster, Google Search, social networks like Facebook, etc.\\

While these considerations were challenging only few people several years ago, the number of software companies involved in large-scale computations is growing quickly. This situation has led to the creation of a new economic market of storage and computation facilities. Some very large software actors have decided to provide these facilities as a commercial service, allowing new players to outsource their computing solution, making computation and storage a facility as electricity already is. These new commercial services are referred to as Cloud Computing.\\

The fast-growing interest in Cloud Computing over the past few years has led to a fuzzy and continuously evolving situation: many have heard of it, but few people actually agree on a specific definition. More importantly, even fewer understand how it can benefit them. From our point of view, the best definition of Cloud Computing has been provided by Armbrust et al. in \cite{AboveTheCloud}: Cloud Computing "\emph{refers to both the applications delivered as services over the Internet and the hardware and systems software in the data centers that provide those services}".\\

From a consumer's point of view, Cloud Computing allows any user to rent a large number of computing instances in several minutes to perform data/compute intensive jobs or build web applications. Since such instances can be dynamically provisioned up or down, it allows users to meet specific scale-up constraints, i.e. to be able to be enlarged to accommodate growing amount of work. For example, this scaling elasticity lets consumers face weekly data consumption peaks. The storage and computing capacities are provided as a service in a pay-as-you-go way. Cloud Computing therefore disburdens users from the hardware investment.\\

Many Cloud Computing solutions have already been developed. There are two categories of cloud solutions. The first category includes all the big Cloud Computing providers, such as Google, Microsoft or Amazon. They often provide the cheapest prices of Cloud Computing solutions, through well-designed but fixed frameworks. The second category gathers all the small Cloud Computing offers. Contrary to the former ones, the small actors provide their customers with custom Cloud Computing solutions. While the storage or CPU costs of these companies usually cannot compete with the big providers' prices, their economic advantage lies in all the specific services provided to their customers with. In particular, the small Cloud Computing providers sometimes embody the ability for companies' top management to challenge or replace some tasks previously reserved to their Information Technology (IT) department, for example in the case of a conflict between the top management and the IT. As a consequence, the Cloud Computing adoption may sometimes be the subject of internal political dealings. The small and custom Cloud Computing offers are therefore of prime interest for many potential customers as they can adapt to specific needs. However, the present chapter does not further detail these offers, as it aims to focus on the most appropriate cloud solutions for intensive computations.\\

A survey of Cloud Computing realized in 2011 by TNS (\cite{CloudSurvey}) provides first feedbacks of early cloud solution adopters. The survey was conducted on 3645 IT decision makers in eight different countries. The study states that 33\% of companies adopted cloud primarily to access information from any device rather than to cut costs. Indeed, while a large part of these companies saved money thanks to cloud adoption (82\%), the savings are on average small (more than half of the companies report less than \$20,000 savings). Beyond the reduction of the costs, Cloud Computing can contribute to improve most of the quality of services. 93 \% of all the companies interviewed have noted some improvements in their services since their cloud adoption. Among these improvements, the hardware administration and maintenance release is a major element.\\

Cloud Computing is also a subject of interest for academic researchers. The use of Cloud Computing can provide them with access to large-scale experiment platforms without getting hardware access grant or hardware investments from their administration. All the scientific intensive computing tasks do not fit the Cloud Computing constraints, especially because of the communication throughput and latency constraints. Yet, provided the tasks fit into the scope of Cloud Computing, even scientists with limited experience with parallel systems are provided with the resources of a large distributed system.\\

One can foresee that the importance of Cloud Computing in the coming years will keep growing, as many companies have expressed their will to resort to these cloud platform solutions. This dynamic is strengthened by the large research and development investments of the major cloud providers. These providers keep investing a lot of work and money to improve their services and tackle economical, ecological and performance challenges. For example, in 2011 Microsoft planned to spend 90 percent of its \$9.6 billion research and development budget on cloud strategy (according to an interview given by the Microsoft International President Jean-Philippe Courtois to Bloomberg (\cite{CloudInvestment})).\\

This chapter is a short overview of the current state of Cloud Computing (in early 2012). It is organized as follows. The second and following section gives a short introduction to the origins of Cloud Computing. The third section is devoted to a more in-depth definition of Cloud Computing. In particular, we compare Cloud Computing and Grid Computing, and give some insights of the different abstraction levels it targets. Cloud storage design is investigated in a fourth section, while the fifth section describes the main cloud execution frameworks.
%
%\subsection{Cloud Objectives}
%
%While Cloud Computing offers differ in their form, as detailed in section \ref{xaas}, they share the same concerns and objectives. This section lists the different challenges cloud address. Some of this points have already been discussed, and others will be discussed in following sections, \\
%
%\begin{itemize}
%\item \textbf{Scaling} is the ability of the system to support huge data sets (multiple terabytes) and very high request rates. Cloud systems achieve scaling by scale-out using large numbers of commodity servers (on the contrary to scale-up which is achieved by using more powerful hardware). An effective scale-out system balance load across servers. The choice between scaling-up and scaling-out is discussed in section \ref{cloudEmergence}.
%\item \textbf{Elasticity} is the ability to dynamically dimension and redimension the number of machines used to adjust to the present computation and storage needs. Elasticity is a side-effect of resources mutualization, as explained in section \ref{gridvsCloud}.
%\item \textbf{Availability} guarantees that every request receives a response. While total availability cannot be guaranteed on large systems, most Cloud Providers have been committing themselves into Service Level Agreement (SLA) to guarantee a minimum rate of effective availability. This availability is made harder as the cloud is made of cheap commodity hardware.
%\item \textbf{Consistency} is a set of properties (strong consistency, eventually consistency, ...) related to the way data are accessed and modified by all the computing instances. It is defined and discussed in section \ref{nosqlParagraph}.
%\item \textbf{Cost-effectiveness} is the ability to execute and complete data/compute intensive tasks on the cloud at a lower price than it would have cost on other hardware. It is a mandatory condition for computing consumers to switch on Cloud Computing offers. Pricing is briefly discussed in section \ref{cloudOffers}.
%\item \textbf{Hardware Administration free}:
%\item \textbf{Low latency responsiveness} describes the time data/compute requests take to complete and return answer to the consumer. It can concern latency between compute instances and storage, between consumers and storage, or between consumers and compute instances. To lower latency between consumers and the data centers that host storage and compute instances, cloud providers have been positioning their data centers all over the world, so that a request could be treated quickly, no matter its geographical origin.
%\item \textbf{Reliable Persistency} guarantees that data stored in the cloud remains available in the long term, independently of hardware failures. Persistency implies that each data chunk is replicated multiple times, which tends to complicate consistency and low latency responsiveness.
%\item \textbf{Security} is a major concern for a lot of potential cloud customers. More specifically, a result of Cloud Computing outsourcing is that the data privacy and accessibility management are also outsourced. While the cloud provider is in most cases technically more qualified than its customers to address security issues, the cloud provider is also subject to foreign laws. Despite the efforts of cloud providers to clarify the legal situation, it still remains fuzzy.
%\end{itemize}

%\item \textbf{Consistency/availability, throughput and latency tradeoff}:

\section{Origins of Cloud Computing}

\subsection{HPC and commodity hardware computing}

A large part of High Performance Computing (HPC) researches and experiments is implemented on custom HPC hardware, sometimes called supercomputers. While the exact beginning of custom HPC platforms is rather uncertain, it goes back at least to the 1960s and the work of Seymour Cray. These computing platforms have been successfully used in many research fields such as weather forecasting, aerodynamic research, nuclear explosion simulations, molecular dynamics simulation, drug discovery, etc.\\

During the 80's and 90's, the amount of personal computers all around the world has increased very significantly. The resulting scale economies have led to a situation where it is nowadays often more cost-effective to stack cheap retail CPU than buying expensive dedicated hardware. In parallel with HPC development on supercomputers, research has therefore been made to distribute computations on commodity hardware.\\

This situation is summarized by Drepper in \cite{Drepper}: \textit{"It is important to understand commodity hardware because specialized hardware is in retreat. Scaling these days is most often achieved horizontally instead of vertically, meaning today it is more cost-effective to use many smaller, connected commodity computers instead of a few really large and exceptionally fast (and expensive) systems"}.\\

To distribute computations on commodity hardware, initial solutions have been provided to network together distinct commodity computers. For example, the Parallel Virtual Machine (PVM) framework originally written in 1989 is a software system that enables a collection of potentially heterogeneous computers to simulate a single large computer. A second major framework is the Message Passing Interface (MPI) framework, whose version 1.0 was released in 1994. This second framework is more explicitly described in Chapter \ref{chap:KMeans}. The hardware systems composed of multiple commodity computers and networked through software as PVM or MPI are often called Beowulf clusters.\\

A halfway solution between supercomputers and Beowulf clusters was introduced several years ago. This solution mixes the cost efficiency of commodity CPU with the throughput of dedicated hardware such as custom communication bus and ethernet cards. This solution is therefore composed of commodity CPU and RAM memory, along with custom and dedicated power supply devices, motherships, and inter-machines connections. All the different components are fastened on racks and/or blades, then gathered in rack cabinets.\\

Grid Computing is a general term referring to hardware and software configurations to distribute computations on commodity CPU, whether in Beowulf clusters or in custom racks and cabinets. The following subsection describes more precisely the Grid Computing frameworks.\\

\subsection{Grid Computing}
\label{grid}

Grid Computing (or grid for short) is a distributed computing system composed of many distinct networked computers (sometimes referred to as nodes). These computers are usually commercial standard hardware gathered into a grid platform. The different components are by design loosely coupled, which means each computer has little knowledge of the existence and identity of the others. A dedicated low-level software layer called middleware is used to monitor and load-balance tasks between the nodes.\\

Contrary to supercomputers, the network that connects the different devices of a Grid can be rather slow. In the case of supercomputers, all the processors are geographically very close and communication is endorsed through an efficient local computer bus or a dedicated High Performance Network. In the Grid Computing case, communication goes through a Local Area Network (LAN) or a Wide Area Network (WAN), which cannot compete with the throughput of a local bus. This communication constraint on Grid Computing means that Grid is very well-suited to large batch jobs where there are intensive computational requirements and small communication needs between machines.\\

Grid Computing architectures have been implemented in many different ways, some of them being simple gathering of physical devices where administrators and end users are one and the same people. In other cases, the system is a more complex entity composed of distinct collaborative organizations, referred to as administrative domains. In the latter case, the middleware manages the whole system administration, grants users access to the resources, and monitors all the participants to prevent malfunctioning or malicious code from affecting the results. Depending on the context, the term Grid Computing might refer only to this latter case. One of the first middleware handling of the multiple administrative domains is Condor (\cite{Condor}).\\

Grids have been successfully set-up and used in many production fields. For example, it is used by Pratt \& Whitney, by American Express, in banks for Value At Risk estimation, pricing or hedging. It is also used by the National Aeronautics and Space Administration (NASA) in various projects such as the Information Power Grid (IPG) project. Grids have also been provided for academic researches, as it is the case for the Large Hadron Collider (LHC) in the form of the LHC Computing Grid, or for Grid 5000 (see \cite{grid5000}).\\

One of the most well-known forms of Grid Computing appeared in the 2000s, resulting from two major events of the 90s: the personal computer democratization and the Internet explosion. This new form of Grid Computing is known as the Peer-to-peer architecture, which is a Grid Computing platform composed of a collaborative set of machines dispatched worldwide and called peers. This architecture removes the traditional client/server model: there is no central administration and peers are equally privileged. They share a portion of their own resources (hard-drive, network bandwidth, CPU, etc.) to achieve the tasks required. Peer-to-peer model has been popularized by very large-scale distributed storage system, resulting in peer-to-peer file-sharing applications like Napster, Kazaa or Freenet (\cite{freenet}). Since then, Peer-to-peer architectures have also been successfully applied on very intensive computational tasks. Some of the largest projects are listed by the Berkeley Open Infrastructure for Network Computing (see \cite{BOINC}) like Seti@Home or Folding@Home (see \cite{folding@home}).\\

\subsection{Emergence of Cloud Computing}
\label{cloudEmergence}

During the 2000s, the Internet boom urged some behemoth like Google, Facebook, Microsoft, Amazon, etc. to experience computation and storage challenges on data sets of a scale hardly ever met. In addition to the scaling-out of their systems, they faced two additional challenges. Firstly, they needed to create abstractions on storage and distributed computations to ease the use of these systems and allow non-distributed computing experts to run large-scale computations without getting involved in the current parallelization process. Secondly, they were concerned with minimizing the operating cost per machine in their data centers.\\

Such constraints meant partially dropping the previous mainstream data storage system (this situation is detailed in Section \ref{nosqlParagraph}) along with developing new software and hardware frameworks to easily express distributed computation requests. These companies continued the work on commodity hardware and merged various innovative technologies into coherent technology stacks, on which a short overview is given in Subsection \ref{technoStack}. They built large data centers hosting tens of thousands of commodity machines and used the technology stacks they improved to meet their internal computations and storage needs.\\

Initially, the infrastructures and data centers as well as the new storage systems and the software frameworks for distributed computations were designed to be tools exclusively dedicated to the companies internal needs. Progressively, some of these tools have been transformed into public commercial products, remotely accessed through Application Programming Interface (API) as paying services. The companies that made these commercial offers became Cloud Computing providers.\\

\section{Cloud design and performance targets}

\subsection{Differences between Cloud Computing and Grid Computing}
\label{gridvsCloud}

A classical distinguishing criterion between Cloud Computing and Grid Computing is the uniqueness or multiplicity of administrative domains. Clouds are supposed to be rather homogeneous servers located in data centers controlled by a single organization, while grids are supposed to be a federation of distinct collaborative organizations sharing (potentially heterogeneous) resources. A consequence of the multiple administrative domains in Grid Computing is that grids need to handle the additional constraints of a federated environment, e.g. checking the credentials and the agents' identities, but also monitoring the system to detect malicious uses. These additional constraints, together with the potential lower bandwidth between two distinct administrative domains increase the communication burdens and therefore tend to require coarser granularity for the tasks distributed on the Grid than those distributed on the cloud.\\

A second aspect of Cloud Computing is the intensive use of Virtual Machines (VM). A VM is the software implementation of a computer that executes programs as if it was a physical machine. VM implementations were already available several years ago, but they were uniquely software based and therefore suffered from a significant impact on their overall performance. Nowadays the VM implementations are both hardware and software based, reducing the loss due to virtualization in a significant way. While the original motivation for virtual machines was the desire to run multiple operating systems in parallel, Cloud Computing is intensively using VM to build a higher degree of abstraction of the software environment from its hardware. The use of VM provides Cloud Computing with desirable features such as application provisioning and maintenance tools, high availability and easier disaster recovery.\\

A third difference between grid and cloud is ownership. In most Grid Computing examples, consumers as a group own the resources they use. In the cloud case, resources are totally outsourced to the cloud provider and do not need delegation of authentication or authority: cloud consumers are paying for resources in a pay-as-you-go manner. Rather than accessing resources directly, Cloud Computing consumers access them through a kind of service they pay for. By adopting cloud, computing consumers do not need to invest in and own hardware and infrastructure (data center, power supply, cooling system, etc.) as they would probably need in the Grid Computing case.\\

A side effect of this ownership shift is multi-tenancy and elasticity. Elasticity is the ability for the cloud consumer to provision up or down the number of computing instances as well as the size of the data storage system. Since different resources are gathered into a specific cloud company actor instead of being held by each customer, the resources are mutualized. Such a mutualization of resources allows elasticity, in the same way the mutualization of the risks allows insurance mechanisms. This dynamic reconfiguration of the resources allows each consumer to adjust its consumption to its variable load, therefore enabling an optimum resource utilization. For instance, in most Cloud Computing systems, the number of computing instances can be resized in few minutes.\\

One last significant difference between cloud and grid is hardware administration and maintenance. Cloud consumers do not care about how things are running at the system end. They merely express their requests for resources and the Cloud Computing service then maps these requests to real physical resources. It therefore disburdens customers of a part of system administration. In the same way electricity became available to consumers in the 1930's when a national electricity grid was implemented, Cloud Computing is an attempt to commoditize IT. The administration part left to the cloud consumer depends on the type of cloud offer which will be chosen (Infractruture as a Service, Platform as a Service, etc.) as detailed in the following subsection.\\

From a computing consumer's point of view, Cloud Computing is therefore a Grid Computing with a single administrative domain, where the management of physical resources has a clear outsourced centralized ownership and hardware is accessed indirectly through abstractions, services and virtual machines. There are much fewer commitments for the consumer since no initial investments are required anymore\footnote{With the notable exception of the cost of porting the existing applications on the cloud.}.\\

\subsection{Everything-as-a-Service (XAAS)}
\label{xaas}

A Cloud Computing environment is a three-actor-world. It is composed of the cloud provider which supplies hardware ownership and administration, the cloud consumer using this abstracted hardware to run applications (the cloud consumer often being a software company), and the application consumer, which is using the applications run on the cloud. The three actors are related to each other as follows: the cloud consumer is the customer of the cloud provider, and the application consumer is the customer of the cloud consumer.\\

It is up to the cloud consumer to choose which part of the application building he wants to handle by himself and which part will be built using the cloud provider pre-made libraries and services. This decision mostly depends on the cloud consumer's will to get involved in low-level software considerations. More specifically, there is a tradeoff for the cloud consumer between a higher control on the hardware and the systems he consumes, and a higher level of programming abstractions where developing applications is made easier.\\

To address the different choices of the cloud consumers, each cloud system has developed distinct strategies and API, as reported in \cite{BenchmarkingCloudServices}. A mainstream classification of Cloud Computing offers is therefore based on the programming abstraction level proposed by the cloud providers. Let us provide a short description of these different abstraction levels.\\

\begin{itemize}
\item \textit{Infrastructure as a Service} (IaaS) is the lowest cloud abstraction level. Resources (computation, storage, and network) are exposed as a capability. Cloud consumers are relieved of the burden caused by owning, managing or controlling the underlying hardware. Therefore they do not access directly the physical machines, but indirectly through VM.
    %The resources available to an application run by a VM are strictly bounded by the resources allocated to this VM.
    IaaS consumers are given almost full control on the VM they rent: they can choose a pre-configured Operating System (OS) image or a custom machine image containing their own applications, libraries, and configuration settings. IaaS consumers can also choose the different Internet ports through which the VM can be accessed, etc. A VM in IaaS can run any application built by the cloud consumer or anyone else. The main example of IaaS offers is Amazon Elastic Cloud Compute (EC2).\\

\item \textit{Platform as a Service} (PaaS) is a higher cloud abstraction level than IaaS, designed to ease applications building and hosting. Contrary to IaaS on which the OS and the configuration settings can be chosen and any application can be run, PaaS provides the cloud consumer with much less freedom. This consumer is given a fixed and constrained execution environment run on a specialized OS, and he is only in charge of defining the code to be run within this framework. The PaaS environment manages code deployment, hosting and execution by itself. PaaS offers may also provide additional development facilities, such as application versioning and instrumentation, application testing, system monitoring, etc. In the case of PaaS, storage is mostly provided in the form of a remote abstracted shared storage service. Prime examples of PaaS are Amazon MapReduce with Simple Storage Service (S3), Microsoft Azure, BungeeLabs, etc.\\

\item \textit{Software as a Service} (SaaS) is the highest cloud abstraction level where applications are exposed as a service running on a cloud infrastructure. Contrary to the customer of IaaS or PaaS, the SaaS consumer is not involved in any software development. This customer is only an application consumer communicating through the Internet with the application which is run on the cloud. Some examples are GMail, Salesforce.com, Zoho, etc.\\
\end{itemize}

These distinct abstraction level technologies have different positioning on some tradeoffs:\\

\begin{itemize}
\item \textit{abstraction/control tradeoff}: PaaS provides higher abstractions to develop applications than IaaS, but the applications built on top of PaaS need to conform and fit with the framework provided. As outlined in Subsection \ref{MapReduceCriticism}, some processes hardly fit with a rigid framework like MapReduce. On the other hand, IaaS will provide cloud consumers with a higher control on the machines but with fewer off-the-shelf abstractions in return: the same distributed computation challenges will probably need to be repeatedly solved.\\
\item \textit{scalability/development cost tradeoff}: Applications run on a PaaS environment prove to be easily scalable. Yet, this scalability comes as a side-product of a strong design of the application so that it fits in with the framework provided. On the contrary, applications run on an IaaS environment provides more flexibility in the way the application can be built, but the scalability remains the customer's responsibility.\\
\end{itemize}

\subsection{Technology stacks}
\label{technoStack}

Cloud Computing frameworks are different hardware and software technologies merged together into technology stacks to handle thousands of machines. This subsection highlights some of these hardware and software technology stacks developed by first row cloud actors.\\

\subsubsection{Hardware technologies}

In this subsection we describe how some of the cloud challenges are taken up, namely the minimization of the operating cost per machine and the improvement of bandwidth throughput between machines. The cost issue is managed by lowering energy consumption and improving automation (we refer the reader to \cite{CloudAutomation} for an illuminating introduction to automation challenges in the cloud). The bandwidth throughputs are optimized with specialized hardware improvement and network topology refinements.\\

According to Jon Koomey (Consulting Professor at Stanford) on his blog, \textit{``Cloud Computing is (with few exceptions) significantly more energy efficient than using in-house data centers''}. Yet, more than $40\% $ of the amortized cost of a data center is still energy consumption, according to \cite{CloudCost}. This energy consumption is optimized through special care of the cooling system and the minimization of power distribution loss (which respectively account for 33\% and 8\% of the total energy consumption of data centers, according to \cite{CloudCost}).  While early days data centers were built without paying much attention to those constraints, the most recently built data centers are located in places where power and cooling are cheap (for example on a river in Oregon for a Google data center, or in the Swedish town of Lulea for a Facebook data center).\\

Since automation is a mandatory requirement of scale, a lot of work has been done to improve it in data centers. According to \cite{CloudCost}, a typical ratio of IT staff members to servers is 1:100 in an efficient firm and 1:1000 in an efficient data center. This is partially due to the technologies developed by hosting companies (like Rackspace or OVH) or backup companies (like Mozy, funded in 2005). The virtualization techniques of machines ---e.g. VM-ware or Microsoft Hyper-V--- increase administrators' productivity by letting them handle much more machines. In addition, the number of CPU and the number of cores per CPU has been recently increased. This led to a situation in which the administrators are handling much more machines and in which each machine holds more cores. Machines are replaced when the physical failing machines in a rack or in a container hit a given ratio. Other techniques of automation can be found in \cite{CloudAutomation} or \cite{lenk2009witcaamotcl}.\\

A key factor in cloud performance bandwidth (whether intra or inter data centers bandwidth) is the spatial disposition of the hardware in the data center, as well as the network topology between the different nodes dedicated to data storage and computing. While a lot of research has been made on this subject, major cloud companies have not been disclosing much about this major technological challenge and their own implementation. According to \cite{CloudCost}, Microsoft plans to upgrade some of its data centers following the networking architecture studied in \cite{DatacentersDesign}.

\subsubsection{Software technologies}

Several software technology stacks have been developed to make the access to CPU and storage easier. They split into three abstraction level : the storage level, the execution level, and the DSL (domain-specific language) level.\\

\begin{itemize}
\item The storage level is responsible for preserving the data into the physical storage. It handles all the issues of distributed storage management, of data contention and provides the data with a structured access. It can be SQL or No-SQL storage, as described in Section \ref{nosqlParagraph}. This storage can also be used as a means of communication between machines. Other types of storage are available in IaaS solutions such as Amazon Elastic Block Store that provides storage volumes that can be attached as a device to a running Amazon EC2 instance and that persist independently from the life of this instance.\\
\item The execution level defines the general execution framework. It specifies how the machines are supposed to organize themselves, it monitors the workers and restarts the failing machines. Depending on the type of Cloud Computing offers (IaaS, PaaS, etc.), the execution level can provide each computing device with anything a standard OS could achieve, or it can give a much more constrainted execution framework where the execution flow must be designed in a specific way (the most famous example of such a constrained execution framework is MapReduce, described in Subsection \ref{MapReduceSection}).\\
\item The DSL level is a higher level execution framework than the execution level. It often provides a declarative language instead of the procedural approach provided by the execution level, therefore saving the users from specifying how the work is supposed to be parallelized. It provides automated and under the hood parallelization, scheduling and inter-machines communications. Therefore, the DSL level helps users focus only on defining the result they want, and not on how they want it to be computed. DSL implementations are often built in a SQL-like manner, allowing the people who are familiar with this syntax to run those requests in a similar way on an underlying No-SQL storage. Besides, since it provides a run-time framework to execute those requests, it spares the users from compiling their software each time they modify or add a request. Historically, they have been added years after the introduction of the execution level to provide non developing specialists with a means to easily use the execution level. PigLatin, Sawzall, DryadLinq are well-known examples of DSL implementations.\\
\end{itemize}

The storage level can be used independently from the two other levels as an outsourced storage service. The execution level can also be used independently from the storage level and the DSL level, provided no persistence is required. However, most cloud applications are built using both storage and execution level environments. The DSL level is an optional layer provided to ease the usage of the storage and execution levels. Theoretically, each project at a given level could be substitutable to any other project at the same level so we could for example plug and use any storage framework with any execution framework. Yet, in practice, some components are designed to work only with other components (as it is the case for the original MapReduce framework which requires some knowledge about where the data are stored for example). Therefore, there are four main frameworks that provide a full technology stack holding the three abstraction levels.\\

The first framework has been presented by Google. It is the oldest one. On the storage level, it is composed of Google File System (\cite{GoogleFileSystem}) and BigTable (\cite{bigtable}) (the latter being built on top of Google File System). The execution level is represented by the original MapReduce version (described in Subsection \ref{MapReduceSection}): Google MapReduce (\cite{MapReduce2004}). After the release of MapReduce, Google engineered a dedicated language, on the DSL level, built on top of MapReduce that provides the storage and the computation framework a higher level access: Sawzall (\cite{sawzall}).\\

The second technology stack is the Apache Hadoop project (\cite{Hadoop}). This is one of the most famous open-source frameworks for distributed computing. It is composed of several widespread subprojects. The storage level includes HDFS, Cassandra, etc. The execution level includes Hadoop MapReduce, but can also be hand-tailored with the help of distributed coordination tools such as Hadoop Zookeeper. The DSL level includes Hive and Pig. Pig is a compiler that processes Pig Latin (\cite{piglatin}) code to produce sequences of MapReduce programs that can be run for example on Hadoop MapReduce. Pig Latin is a data processing language that is halfway between the high-level declarative style of SQL and the lower procedural style of MapReduce. Pig Latin is used for example at Yahoo! (see for example \cite{piglatin}) to reduce development and execution time of data analysis requests.\\

The third technology stack is composed of Amazon Web Services (AWS) and AWS-compatible open-source components. Its most popular storage components are Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), Amazon SimpleDB, Amazon ElastiCache or Amazon Relational Database Service (Amazon RDS). AWS execution level is mainly composed of Amazon Elastic Cloud Compute (Amazon EC2) and Amazon Elastic MapReduce (which uses a hosted Hadoop framework running on top of Amazon EC2 and Amazon S3). AWS is one of the first large commercial Cloud Computing offers: it was initially launched in July 2002, and was valued at as a 1 billion dollar business in 2011 (see \cite{AmazonBusiness}). In parallel, Eucalyptus provides an open-source stack that exports a user-facing interface that is compatible with the Amazon EC2 and S3 services.\\

The fourth framework has been developed by Microsoft. The storage level is embodied by distinct independent projects: Azure Storage, Cosmos or SQL-Server. Cosmos is a Microsoft internal project that has been used by Search teams and BING. Azure Storage is a different project, initially built using a part of the Cosmos system (according to \cite{AzureStorage}). Azure Storage  is meant to be the storage level standard layer for applications run on Azure. The execution level being held by Dryad (\cite{Dryad2007}) or by Azure Compute (or small frameworks like Lokad-Cloud (\cite{LokadCloud}) (on top of Azure Compute or Lokad-CQRS (\cite{LokadCQRS})). The DSL level is represented through two different projects: DryadLINQ (\cite{DryadLINQ}) and Scope (\cite{SCOPE}).\\

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.6]{../../Pictures/Cloud/msparallel.jpg}
\end{center}
\caption{Illustration of the Google, Hadoop and Microsoft technology stacks for cloud applications building. For each stack, the DSL Level, the Execution Level and the Storage Level are detailed. In this scheme, SQL is also described as a stack in which the three levels are merged together and cannot be used independently.}
\label{fig:StorageExecutionAndDsLevel}
\end{figure}

Figure \ref{fig:StorageExecutionAndDsLevel} (on the next page) is taken from a blog post of Mihai Budiu. It summarizes the technology stacks of Google, Hadoop and Microsoft.

\section{Cloud Storage level}
\label{nosqlParagraph}

Persistence is a characteristic that ensures data outlive the process that has created them. A data storage is a system that provides data persistence and is a fundamental element in most application developments. In the case of distributed computing, data storage can also be used for a different purpose as an inter-machines means of communication.\\

While working on a cloud platform, no hardware shared memory is available since the machines are physically distant. In such a situation, inter-machines communications can be achieved through direct IP communications or through a distributed data storage system. While the direct IP communications have been much favored in several systems such as most Peer-to-Peer systems and are still largely used in Amazon EC2, the use of a data storage system as a means of communication is widely used in numerous Cloud Computing platforms. Since the cloud storage is abstracted and accessed ``as a service'', it is easier to communicate through a data storage than to physically access a virtualized machine. The cloud storage has also been designed to easily scale and address satisfactory communication throughput.\\

In this section we describe the challenges involved by a large-scale distributed data storage system. We introduce in Subsection \ref{acid} the Relational DataBase Management System (RDBMS), which was the undisputed data storage system several years ago. Then we explain (in Subsection \ref{Cap_theorem_section}) why some of the guarantees of RDBMS impose constraints which are hard to meet in a distributed world. Finally, we introduce the reader in Subsection \ref{sec_taxonomy} to some new data storage systems, known as No-SQL storages, which relax some of the guarantees of the relational database systems to achieve better scaling performance.

\subsection{Relational storage and ACID properties}
\label{acid}

Relational databases are structured through the relations between data entities. They are inspired by relational algebra and tuple calculus theories, while they do not respect all of their constraints. Relational databases are well understood and offer great guarantees for reliably storing and retrieving data in a robust manner. In addition to these guarantees, the relational model proved to be well-suited to express real-world business data interdependencies. RDBMS are therefore widely used in business applications.\\

A Structured Query Language (SQL) is provided together with the RDBMS to make the use of data easier. SQL is a declarative programming language (with some procedural elements) intended to manage data queries. Declarative languages express \textit{what} computations should return rather than explicitly defining \textit{how} the computations should be done. In the case of relational storages, queries are expressed in SQL to define requests to the desired data, while the underlying RDBMS performs the actual operations to answer the requests. Listing \ref{sql_listing} gives an example of a SQL query.\\

\begin{lstlisting}[caption=a SQL query sample, label = sql_listing]
SELECT T1.variable1, T1.variable2
FROM Table1 T1, Table2 T2
WHERE T1.variable1 = T2.variable3
ORDER BY T1.name
\end{lstlisting}

A single ``business'' operation applied on data stored in a database is called a transaction. A single transaction can affect several pieces of data at once. The guarantees provided by RDBMS can be understood by observing how transactions are applied on the database. The following paragraphs introduce the main relational storage guarantees by describing how they are translated in terms of transactions management.\\

\begin{itemize}
\item \textit{Consistency} is a property related to the way data are modified during transactions. There are multiple definitions of consistency, resulting in different consistency levels and depending on whether the storage is distributed or not. Initially, consistency was a property that guaranteed that some user-specified invariants on data were never broken. For example, such an invariant could consist in having the sum of some variables equal the sum of other variables. With the rise of distributed storages, the definitions of consistency have been multiplied, and consistent distributed storage (also called strongly consistent storages) often refers to the fact that each update is applied to all the relevant nodes at the same logical time. A consequence of this definition is that if a piece of data is replicated on different nodes, the data piece will always be the same on each node, for each given instant. A more in-depth definition of consistency can be found in \cite{CAP_Theorem}.\\

\item \textit{Atomicity} guarantees each transaction is applied as if it were instantaneous. More specifically, a transaction composed of $n$ database elementary operations is atomic if all the operations occur or none occur (if one or several operations fail) and if all the processing units either see the database in the state it was before the beginning of the transaction or after the end of the transaction. A storage has the atomicity property if any transaction can be guaranteed to be atomic. This property guarantees that any transaction applied on a database does not corrupt it.\\

\item \textit{Isolation} guarantees that each transaction should happen independently from other transactions that occur at the same time.\\

\item \textit{Durability} enforces that data are persisted permanently and that transactions remain permanent, even in the presence of a system failure.\\
\end{itemize}

These four important guarantees are gathered under the acronym ACID, which stands for Atomicity, Consistency, Isolation and Durability. These properties are guaranteed by every relational storage. In the case of a distributed relational storage, some other very mandatory properties can be desired, such as Partition tolerance and Availability.\\

\begin{itemize}
\item A network is \textit{partitioned} when all the messages sent from nodes in one component of the partition to nodes in another component are lost. Distributed storage is said to be partition tolerant when it continues to run in the presence of a partition failure.\\

\item A distributed storage guarantees \textit{Availability} if every request received by a non-failing node in the system returns a response.\\
\end{itemize}
\subsection{CAP Theorem and the No-SQL positioning}
\label{Cap_theorem_section}

The CAP theorem was first expressed by Eric Brewer at the 2000 Symposium on Principles of Distributed Computing (PODC). The CAP theorem was then proved in 2002 by Seth Gilbert and Nancy Lynch in \cite{CAP_Theorem}. This theorem is a theoretical impossibility result, which states that a distributed storage cannot achieve both Availability, Consistency, and Partition Tolerance.\\

More specifically, this theorem is a theoretical translation of the well-known practical challenges of applying ACID transactions on data split on multiple machines. The ACID constraints require a lot of inter-machines communications to complete a single transaction (see for example \cite{TwoPhaseCommitProtocol}) and lead to serious lock contentions that deeply decrease the transactional throughput.\\

Moreover, guaranteeing that the data stored in cheap and unreliable hardware are highly available implies multiple replications of each single piece of data. Since data servers are expected to fail from time to time, the data are duplicated several times (Azure storage is reported to hold for each data 6 replicas: 3 replicas in a first data center and 3 others in a backup data center (\cite{AzureStorage})), so that a data server can die without any data loss. To guarantee strong consistency while keeping multiple versions of any data pieces in different locations causes large overheads, as reported in \cite{CLAP}.\\

Scaling traditional ACID-compliant relational databases to distributed shared-nothing architectures with cheap machines therefore turned out to be very difficult. Data storage systems developed in the 2000s known as NoSQL storage (for Not Only SQL) such as Cassandra, HBase, BigTable, SimpleDB, etc. therefore dropped some of the ACID guarantees to achieve better scaling and lower latency. As stated by Daniel Abadi and Alexander Thomson in their blog, \textit{``NoSQL really means NoACID''} (see \cite{NoSQLmeansNoAcid}).\\

Some of these systems (like BigTable or Azure BlobStorage) decided to provide weakened atomicity and isolation guarantees for transactions. Indeed, transactions that modify more than one piece of data are not guaranteed to be either all done or none. Therefore, no logical transaction implying several pieces of data can be guaranteed to be atomic at the storage level. It is up to the storage user to ensure logical transactionality, and that user-specified invariants are not modified.\\

Other cloud storage systems like Cassandra have been dropping strong consistency between replicas to provide only a weakened consistency guarantee called eventual consistency. We refer the reader to \cite{Eventually_Consistent} for an introduction to eventual consistency, while an introduction to BASE (Basically Available, Soft state, Eventually consistent) can be found in \cite{BASE}. While the eventual consistency is satisfactory enough for a given number of well-known web applications such as Facebook, the loss of strong consistency is a very difficult issue for many other real world applications, which makes application development much harder. This is a key aspect of No-SQL storage: the responsibilities are partly shifted and the scalability comes at the expense of a stronger developer burden. The consistency of various cloud storage systems has been investigated in \cite{CloudConsistency}.\\

%Recently, the relational database model features, known as ACID, have been challenged. Brewer's CAP theorem (Consistency, Availability and Partition tolerance) \cite{CAP_Theorem} states that distributed storage won't be able to propose ACID properties in an acceptable way. Abadi recently explained some defaults of CAP theorem \cite{CLAP_Presentation}. During this time, web services delivered by very large companies like Google, Amazon, Yahoo!, Microsoft... have been partially abandoning relational database and ACID properties, starting the NoSQL movement (NoSQL standing for Not Only SQL). Even some SQL-related people recently expressed their interest for NoSQL database, like Ayende (NHibernate contributor) did  \cite{Slaying_Relational_Dragons}.


\subsection{Cloud Storage Taxonomy}
\label{sec_taxonomy}
Along with the calling into question of ACID properties, the relational data model which is related to them has also been challenged. No-SQL storages therefore provide different data models, often noted to provide a more fuzzy division between the database model and the application. In this paragraph we describe the most used database models. A more comprehensive list of these models can be found in \cite{NoSQL}.\\

Document-oriented databases assume data are organized around the concept of documents. Each document holds data and is persisted through a standard data format such as XML, YAML, JSON, etc. or even through binary or .pdf files. Contrary to RDBMS where the data scheme is fixed, document-oriented databases do not impose that each document should own the same fields as the others. Thus, new information can be added on a document without having to fill in the same information for all the other available documents. Each document can be accessed through a unique key, which is often a string. CouchDB, MongoDB, SimpleDB are well-known document-oriented databases.\\

Graph databases are structured on the standard graph theory concepts of nodes and edges. The information is stored on nodes, each node representing a different entity. Each entity is entitled to a set of properties describing the entity. Edges connect nodes to nodes or nodes to properties and describe the relationships between the nodes and the properties. Because of this graph structure, data and more specifically nodes are given an adjacency notion which is index-free and only driven by the nature of the data. Horton, FlockDB, AllegroGraph are graph databases.\\

Key-value pairs databases are schema-free databases. The data is stored on a material device in the form of a serialized programming language datatype, typically a class object. Any type can be persisted and data are accessed through a key, which can be a string or a hash of the data content. Apache Cassandra, Dynamo, Velocity, BigTable, Azure BlobStorage are key-value pairs storages.\\

%We have been noticing that system un-availability is not only theoretical. When accessing to the storage, we can get notified an Azure internal Exception. Therefore, we have to implement retry policies so we can try again to run a read or write operation. Since storage is designed to be accessed by internal workers but also by outside (we can think for example of a picture or a video stored and devoted to be downloaded a lot of time), it makes no sense to guarantee 100\% availability since Internet connection does not give such guarantee.\\
%
%\begin{itemize}
%\item \textbf{Block commits} : Data bigger than 4MBytes must be split into blocks before being uploaded. Once every block has been uploaded into Azure, the "transaction" is commited, and data is finally wrote on Azure side. This 2-phase upload (partial uploads, then integral commit) is TO BE FINISHED
%\item \textbf{Etag} : As reported in the BlobStorage and TableStorage technical papers \cite{Azure_Storage_Resources}, Azure provides a kind of timestamp, called etag. When updating some data, we first download the data, update it, and upload it with the etag provided. When replacing the data, Azure checks that etag of data being replaced matches the etag provided by the upload. If it does not, the overwrite is rejected, an exception is thrown, so the user can retry. This mechanism helps to deal with multiple workers trying to modify a data at the same time, but is very probably also used inside Azure to ensure consistency between the different replica of a same data.
%\item \textbf{Very few overwrites} Most applications do not need to overwrite data. Some typical usage could be for example a big video upload once, and hundreds or thousands of read request on this data (Facebook usecase). Clusterings we run on Azure required very few overwrite (mostly for counters), as it will be shown.
%\end{itemize}



%The phrase ``software as a service'' (SaaS), as explained in Vecchiola et al. \cite{VEC1} refers to \emph{solutions that are at the top end of the Cloud computing stack and they provide end users with an integrated service comprising hardware, development platforms, and applications}. For example the popular webmail GMAIL is based on a SaaS model. The application is accessible for the user through the Internet as opposed to a traditional desktop application (e.g. Microsoft Excel 2010). At a much lower level we have the ``infrastructure as a service'' (IaaS) which provides low level hardware and the users may control a large part of the software environment. However, the creation of automatic provisioning of resources is not easy due to few infrastructure abstractions. Amazon EC2 is one of the most popular IaaS on the market. The third area of ``as a service'' paradigm is platform as a service (PaaS) that can be though as an intermediate abstraction level between IaaS and SaaS. As defined in Vecchiola et al in \cite{VEC1} a PaaS solution provide \emph{an application or development platform in which users can create their own application that will run on the cloud}. A PaaS solution comes with a framework and a set of API to build the application for the cloud. It also comes with a portal that will help the developer to run the application on the top of a fully abstracted infrastructure. In both PaaS and IaaS approaches the elasticity of resources is guarantee by a fine use of visualizations tools which help to perform quickly replications of virtual machines. This aspect is therefore much more abstracted in a PaaS offer. The main actors proposing PaaS solution are Google AppEngine and Microsoft Azure.\\
%
%Cloud Computing solutions have a deep impact on economic models. The pay-as-you-go concept brings the possibility to a company to pay only for what it uses. As an example it can allow $100$ servers for few hours to respond a peak traffic on their web application and one small server after that. Indeed, the cost of $100$ hours of CPU with one machine is the same that using $100$ machines for one hour. Cloud computing short-term (in minutes) elasticity ability of downsizing and upsizing resources available will make the risk of underprovisioning disappear and will reduce the costs due to overprovisioning. Amazon CEO Jeff Bezos in \cite{BEZ1} tells the story of Animoto, a company that used intensively the elasticity of Amazon EC2. Their resource needs pass in three days from $50$ Amazon EC2 instances to $3500$. The software company Lokad propose various SaaS application (Salescast, Callcast, etc...) which use data from their customer (mostly enterprizes) and deliver figures for business intelligence (mostly forecasts and related criterion). The Lokad forecasting engine, deployed in Microsoft Azure PaaS, make complex computations and then requires a lot of CPU. Lokad is bound by the fact to deliver its forecasts within an hour therefore some peak of consumption naturally appear making the automatic short term scaling from $2$ to $500$ VM instances in few minutes.\\
%
%The potential field of application of Cloud Computing is not restricted to IT business. The scientific computing needs to perform large scale experiments requiring large number of computing resources. Usually, scientists used super computers or clusters to run those research works. However, they were inevitably linked to massive costs of hardware and maintenance. Such approach are also difficult to setup and operate to a non expert user so as a physician or mathematician. A new model is brought through Cloud Computing to the scientist community. The storage and computing resources are dynamically provisioned as needed and the experimenting costs would be cut off for non profitable organization such as academic research institutions.\\

\section{Cloud Execution Level}

We examine in this section some of the most widespread execution frameworks. While these frameworks are not only cloud-based, they are often an essential part of scientific cloud applications. The section begins with the most well-known framework: MapReduce.

\subsection{MapReduce}
\label{MapReduceSection}

\subsubsection{General context}

MapReduce has two different meanings. It first describes a general framework concept explained in this section. It also defines the original implementation of this abstract framework, developed by Google (\cite{MapReduce2004}). Other implementations of the concept have then been set-up, like Hadoop (\cite{Hadoop}), or CloudMapReduce (\cite{CloudMapReduce2010}). Unless explicitly stated otherwise, the term MapReduce refers in this section to the framework abstraction concept.\\

MapReduce is a framework that eases distributed computation runs and deployments. It was born from the realization that \textit{``The issues of how to parallelize the computation, distribute the data, and handle failures conspire to obscure the original simple computation with large amounts of complex code to deal with these issues''}, as stated by the original MapReduce paper (\cite{MapReduce2004}). MapReduce is a scale-agnostic programming abstraction (see \cite{Helland}) originally designed to prevent programmers from solving these issues repeatedly and to provide access to a huge amount of computation resources to programmers without any experience with parallel and distributed systems. The resort to distributed computations and to these computation resources lead to significant speedup, i.e. the time to run distributed computations is significantly reduced compared to the sequential computation.\\

MapReduce is a gathering of ideas which have already been discussed in the computer-science literature for a long time (even for decades for some ideas) and it is not the first framework to have implemented some of these ideas. MapReduce is still a well-known abstraction concept, thanks to the large success of its first implementation: Google MapReduce. This framework implementation has proved to be very easy to use for ``programmers without any experience with parallel and distributed systems'' ---referred to in the following as non-experts--- and achieved very good overall scalability as well as a strong resilience to failures.\\

In the following paragraphs, we provide a very short overview of this framework. We refer the reader to the Hadoop definitive guide (\cite{Hadoop}) for in-depth details of the most famous open-source implementation and to \cite{MapReduceForTextMining} and \cite{MapReduceForMachineLearning} for examples of machine-learning problems that can be addressed through a MapReduce framework.\\

\subsubsection{Programming Model}

MapReduce provides an execution environment to process a two-stage execution (divided into a Map stage and a Reduce stage). A typical job in MapReduce is composed of many Map tasks and Reduce tasks. In the Map stage, a user-specified computation ---referred to as Map--- is applied independently and in parallel on many data chunks. Each Map task points to a specific data chunk to which the Map computation will be applied on. Once a Map task is completed, intermediate results are produced and stored on hard-drives. Once all the Map tasks have been successfully completed, an aggregation operation is started, referred to as Reduce. Depending on the size and complexity of all the intermediate results produced by the Map tasks, the reduce stage is divided into one or many reduce tasks.\\

The Map and the Reduce logic are defined by the user. This user submits the MapReduce job through the MapReduce interface, specifying the actual Map and Reduce computations, in addition to the specification of the data that must be processed. The framework provides an environment that handles all the execution orchestration and monitoring. In particular, MapReduce automatically maps the job into a set of sub-tasks, sends the Map and Reduce instructions to the right processing units, makes sure all the computations are successfully run, and restarts or reschedules tasks that have failed.\\


%: the Map tasks and the Reduce tasks. MapReduce automatically maps the job into a set of sub-tasks, schedules the tasks across a large set of machines and monitors the execution of the different tasks.\\
%
%The MapReduce framework takes a set of input key/value pairs and returns a set of output key/value pairs. The framework handles everything by itself except 2 functions that must be user-defined : Map and Reduce, as inspired by the \textit{map} and \textit{reduce} functions in functional languages like Lisp.\\
%
%Map is a function which is applied independently and in parallel to each data chunk. It takes an input key/value pair and produces a set of intermediate key/value pairs.\\
%
%Reduce is an aggregation function which combines map results. It takes as input a key of intermediate key/value pairs and a set of values for that key. It merges (or reduces) them into to form a smaller set of values associated to that key (typically, it reduces the set of values into 1 or 0). The input intermediate values are supplied through an iterator so the reducer could handle data that do not fit into its own memory. Typically, Reduce jobs are much smaller than Map jobs in term of time of computation, but not necessarily in term of I/O.\\

%\includegraphics[height=1.0\linewidth]{../../Pictures/mapreduce.pdf}

\subsubsection{Technical considerations}
\label{sec_MapreduceConsiderations}

The primary targets of MapReduce are heavy computations or computations on large data sets. The leading idea of MapReduce is that it is not an experimental framework but a production tool. As a direct consequence, MapReduce provides a simple and robust abstraction that can easily be handled by a wide range of non-experts.\\

The simplicity of MapReduce comes from two different design choices. Firstly, by making a clear distinction between \textit{what} needs to be computed (user-defined) and \textit{how} the computations are actually made (handled automatically by the framework), MapReduce disburdens the users from difficult parallel computations considerations. More specifically, the users need not dive into general parallel applications design, neither do they need to have any knowledge about the hardware actually running the computations: MapReduce indeed provides a scale-agnostic (\cite{Helland}) interface.\\

Secondly, the framework helps the users to figure out what the parallel algorithm is actually running. As stated in \cite{MapReduceForTextMining}, \textit{``concurrent programs are notoriously difficult to reason about''}. Because of its elementary design, a MapReduce execution has Leslie Lamport's sequential consistency property\footnote{Provided that the map and reduce operators are deterministic functions of their input values.}, which means the MapReduce result is the same as the one that would have been produced by some reordered sequential execution of the program. This property helps to understand the MapReduce execution as it can therefore be rethought as a sequential run speeded-up.\\

As previously stated, the MapReduce framework also provides a strong resilience to failures. A reliable commodity hardware usually has a Mean-Time Between Failures (MTBF) of 3 years. In a typical 10,000 server cluster, this implies that ten servers are failing every day. Thus, failures in large-scale data centers are a frequent event and a framework like MapReduce is designed to be resilient to single point of failures\footnote{With the noticeable exception of the single orchestration machine referred to as master.}. Such a resilience is hard to achieve since the computing units are dying silently, i.e. without notifying of their failure. MapReduce holds a complex monitoring system to ping all the processing units and to ensure all the tasks will be successfully completed, as detailed in \cite{MapReduce2004}.\\

The initial problems MapReduce has been devised to deal with are related to text mining. These problems involve gigantic amounts of data that do not fit in the memory of the processing units dedicated to the computation, as explained in \cite{MapReduce2008}. In such situations, providing the processing units with a continuous flow of data without starving these units out would require an aggregated bandwidth very hard to achieve. MapReduce has provided a new solution to this problem by co-locating the data storage and the processing system. Instead of pushing the data chunks to the processing units in charge of them, MapReduce pushes the tasks to the processing units whose storage holds the data chunks to be processed. Such a design requires a distributed file system that can locate where the data are stored, but removes a lot of stress on communication devices.\\

\subsubsection{MapReduce performance}\label{sec_MapReducePerformance}

Since its first release, MapReduce has proved to be very well-suited to multiple large computation problems on machine clusters. An embarrassingly parallel problem is one for which little effort is required to separate the problem into a number of parallel (and often independent) tasks. These problems require few or zero inter-machines communications and therefore allow to achieve speedup close to the optimal. On such problems, MapReduce is performing very well.\\

A MapReduce mechanism that can significantly impact MapReduce performance is the synchronization process: the Reduce stage cannot begin before all the Map tasks have been completed. This design is very sensitive to stragglers. As defined in \cite{MapReduce2004}, a straggler is a \textit{``machine that takes an unusually long time to complete one of the last few map or reduce tasks''}. Because of the synchronization process in the end of the Map stage, the overall computation is significantly slowed down by the slowest machine in the pool. In the case of a very large execution involving thousands of processing units, the worst straggler will behave much worse than the median behaviour, and significant delays can be noted. MapReduce provides a backup execution logic that duplicates the remaining in-progress tasks when almost all the map tasks have already been completed. Such a duplication mechanism is reported to significantly reduce the time to complete large MapReduce operations in \cite{MapReduce2004}.\\

MapReduce has proved to be very efficient in the case of embarrassingly parallel algorithms. For example, the Hadoop implementation has been successfully used by Yahoo! to win the terasort contest (see \cite{TeraSortWinnerYahoo}). The Google implementation is also reported to be widely used by Google's internal teams (see e.g. \cite{MapReduce2008}).\\

\subsubsection{Recent criticisms of the MapReduce framework for machine-learning}
\label{MapReduceCriticism}

In addition to the quality of the implementations of Google's and Hapache's MapReduce, a key factor in MapReduce success and adoption by the community is how easy it is to use. Because MapReduce only supports tasks that can be expressed in a Map/Reduce logic, it is rather straightforward to parallelize a task using this framework: either the task is intrinsically in a Map/Reduce form and the expression of the task within the framework is obvious, or the task cannot be expressed within MapReduce.\\

The simplicity to use it and to figure out how to adapt a given sequential algorithm into the framework has driven the interest of statistical and machine-learning researchers and practitioners that had no knowledge in distributed computations. Such a success has paved the way for numerous research works to modify and adapt algorithms originally not suited to the MapReduce framework and to suggest MapReduce implementations of the modified algorithms. Yet, since the MapReduce is a rigid framework, numerous cases have proved to be difficult or inefficient to adapt to the framework:\\

\begin{itemize}
\item The MapReduce framework is in itself related to functional programming. The Map procedure is applied to each data chunk independently. Therefore, the MapReduce framework is not suited to algorithms where the application of the Map procedure to some data chunks need the results of the same procedure to other data chunks as a prerequisite. In other words, the MapReduce framework is not suited when the computations between the different pieces of data are not independent and impose a specific chronology.\\
\item MapReduce is designed to provide a single execution of the map and of the reduce steps and does not directly provide iterative calls. It is therefore not directly suited for the numerous machine-learning problems implying iterative processing (Expectation-Maximisation (EM), Belief Propagation, etc.). The implementation of these algorithms in a MapReduce framework means the user has to engineer a solution that organizes results retrieval and scheduling of the multiple iterations so that each map iteration is launched after the reduce phase of the previous iteration is completed and so each map iteration is fed with results provided by the reduce phase of the previous iteration.\\
\item Both the thinking of the abstract framework and of google's practical implementation have been originally designed to address production needs and robustness. As a result, the primary concern of the framework is to handle failures and to guarantee the computation results. The MapReduce efficiency is therefore partly lowered by these reliability constraints. For example, the serialization on hard-disks of computation results turns out to be rather costly in some cases.\\
\item MapReduce is not suited to asynchronous algorithms.\\
\end{itemize}

MapReduce is the reference framework for distributed computations. Its parallelization scheme is the natural choice because of its simplicity. But because of its limitations,
some have started to question the relevance of MapReduce in the case of fine-granularity tasks in machine-learning. As outlined in the alternative GraphLab framework introduction paper (\cite{Graphlab2010}) : \textit{``The overhead associated with the fault-tolerant, disk-centric approach is unnecessarily costly when applied to the typical cluster and multi-core settings encountered in ML research''}. This position was also shared by numerous researchers (including John Langford) during the NIPS workshop LCCC debate in December 2010.\\

The questioning of the MapReduce framework has led to richer distributed frameworks where more control and freedom are left to the framework user, at the price of more complexity for this user. Among these frameworks, GraphLab and Dryad are introduced in the following paragraphs.\\

\subsection{GraphLab}

GraphLab is a new framework directly designed to address parallelization challenges of machine-learning algorithms. GraphLab was initially developed after it has been observed that MapReduce was insufficiently expressive for some machine-learning algorithms, while low-level APIs (like MPI) usage led to too much development overhead.\\

The GraphLab representation model is much more complex and richer than the MapReduce one. It consists in a directed graph representing the data and computational dependencies and in a set of update functions specifying the local computations to be run. By adopting a graph model, GraphLab supports structured data dependencies and is therefore suited to computation schemes where update scopes overlap and some computations depends on other computation results. More specifically, the original GraphLab introduction paper states that it is targeted to \textit{``compactly express asynchronous iterative algorithms with sparse computational dependencies''}.\\

Like MapReduce, GraphLab provides a higher abstraction level than MPI and prevents users from managing synchronization, data races and deadlocks challenges by themselves. It also provides ad-hoc tasks schedulers.\\

GraphLab is therefore a machine-learning parallelization framework which provides more expressiveness than MapReduce, at the expense of a more complex usage. To our knowledge, the GraphLab framework is only currently available on shared-memory multiprocessors setting (only tested on single machines up to 16 cores), making GraphLab unavailable or ineffective for distributed cloud computations.\\

\subsection{Dryad and DryadLINQ}
\label{cloud_dryad}

Dryad (\cite{Dryad2007}) is a general-purpose distributed computation framework designed to scale on any single administrative domain, be it a multi-core single computer or a whole data center with thousands of computing instances. Conversely, Dryad is not targeted to multiple administrative domain environments and does not handle all the authentication and access grants required for most Grid Computing environments.\\

Like GraphLab, Dryad exposes a much more expressive computing framework than MapReduce in the form of a Direct Acyclic Graph (DAG). In this graph, each vertex is a computation task and edges are data and intermediate results channels. Dryad infrastructure maps the logical DAG of computation tasks onto actual physical resources without requiring the developer to get involved in this mapping process. In addition, Dryad handles scheduling across resources, failure recoveries, communication concurrencies, etc.\\

Whereas with MapReduce the developer is given few implementation choices except on how to fit its computation task into the rigid framework of MapReduce, Dryad provides the developer with much more control through the expressiveness of the DAG. This control gain can be explained by a different approach of the frameworks: while MapReduce was focused on accessibility to provide the widest class of developers with computation power, Dryad is more concerned about expressiveness and performance.\\

In addition to a richer expressiveness of the execution flow, Dryad provides a higher control than MapReduce on communication flow and tasks scheduling. This optional higher control endows the programmers with a better understandings of the underlying hardware performance and topology or of the computation tasks to improve the overall performance. For example, Dryad provides different communication media between computing nodes: direct communication through TCP pipes, or indirect communication through files written to hard-drive or shared-memory buffers. Whereas the default Dryad settings is to use files, the developer can choose to resort to a different medium, e.g. in the case of a single multi-core computer.\\

As a side result of this higher control and expressiveness, Dryad is reported to be more demanding on the developer's knowledge and requires several weeks to be mastered. To make Dryad more user-friendly, Microsoft has also been developing an additional framework built on top of Dryad and mimicking the LINQ library interface: this additional framework is called DryadLINQ (\cite{DryadLINQ}). Both Dryad and DryadLINQ have been made available on the Microsoft research website.

%\subsection{Event centric frameworks}
%
%\subsection{HPC on cloud}
%
%We refer to \cite{BridgingtheGap}, \cite{BenchmarkingAmazonEC2}, \cite{HighPerformanceComputingWithClouds}, \cite{TOP500},  \cite{BenchmarkingCloudServices},...

%\section{Azure Storage}
%
%\subsection{ACID framework}
%
%\subsection{Relational Database}
%
%\subsection{Emergence of the No-SQL movement}
%\label{nosqlParagraph}
%
%Recently, the relational database model features, known as ACID, have been challenged. Brewer's CAP theorem (Consistency, Availability and Partition tolerance) \cite{CAP_Theorem} states that distributed storage won't be able to propose ACID properties in an acceptable way. Abadi recently explained some defaults of CAP theorem \cite{CLAP_Presentation}. During this time, web services delivered by very large companies like Google, Amazon, Yahoo, Microsoft... have been partially abandoning relational database and ACID properties, starting the NoSQL movement (NoSQL standing for Not Only SQL). Even some SQL-related people recently expressed their interest for NoSQL database, like Ayende (NHibernate contributor) did  \cite{Slaying_Relational_Dragons}. Most NoSQL data storages are censored here \cite{NoSQL}. The properties of these new data storage systems have been described as BASE (Basically available, Soft State, Eventually consistant) \cite{BASE}.\\
%
%One of the key aspect of distributed data storage is data replication (Azure holds for each data 6 replicas : 3 replicas in a first data center and 3 others in a backup data center REFERENCE NEEDED). Since data servers are expected to fail from time to time, data are duplicated several times, so a data server can die without any data loss. This replication process (even more if replication is not stored on the same physical support) has a deep impact on consistency and performance as reported in \cite{CLAP_Presentation}. Some storage like Cassandra have been abandoning consistency guarantees for eventually consistency \cite{Eventually_Consistent}.\\
%
%Azure does not provide (at least yet) consistency or availability guarantees. Since the Azure code is not disclosed, we cannot certify any of these properties. We have been noticing that system un-availability is not only theoretical. When accessing to the storage, we can get notified an Azure internal Exception. Therefore, we have to implement retry policies so we can try again to run a read or write operation. Since storage is designed to be accessed by internal workers but also by outside (we can think for example of a picture or a video stored and devoted to be downloaded a lot of time), it makes no sense to guarantee 100\% availability since Internet connection does not give such guarantee.\\
%
%To our knowledge, Azure has not been guaranteeing consistency too, even if they say they do. Yet, we have not been able to experience any consistency issue. Three points here :
%\begin{itemize}
%\item \textbf{Block commits} : Data bigger than 4MBytes must be split into blocks before being uploaded. Once every block has been uploaded into Azure, the "transaction" is commited, and data is finally wrote on Azure side. This 2-phase upload (partial uploads, then integral commit) is TO BE FINISHED
%\item \textbf{Etag} : As reported in the BlobStorage and TableStorage technical papers \cite{Azure_Storage_Resources}, Azure provides a kind of timestamp, called etag. When updating some data, we first download the data, update it, and upload it with the etag provided. When replacing the data, Azure checks that etag of data being replaced matches the etag provided by the upload. If it does not, the overwrite is rejected, an exception is thrown, so the user can retry. This mechanism helps to deal with multiple workers trying to modify a data at the same time, but is very probably also used inside Azure to ensure consistency between the different replica of a same data.
%\item \textbf{Very few overwrites} Most applications do not need to overwrite data. Some typical usage could be for example a big video upload once, and hundreds or thousands of read request on this data (Facebook usecase). Clusterings we run on Azure required very few overwrite (mostly for counters), as it will be shown.
%\end{itemize}
%
%\subsection{Azure BlobStorage}
%
%Azure team has been communicating few on their underlying system architecture that handles the storage and how they achieve scalability.
%Most of information presented here relies on the Windows Azure Storage Team blog (http://blogs.msdn.com/b/windowsazurestorage/) and are not guaranteed.\\
%
%Azure Blob Storage enables applications to store large objects, up to
%50 GB each.  It supports a massively scalable blob system, where hot blobs will
%be served from many servers to scale read access. Blobs are composed of a
%string (that is used as a key to store the value), a value composed of a
%binary object, and a timestamp (etag) (that indicates the last write on this
%blob). In addition of Get and Put methods, blobs whose key shares a given
%prefix can be listed. Optimistic non locking atomic read-modify-write
%operations can be implemented using a timestamp matching condition: a write
%succeed if and only if the timestamp of the storage matches the one provided
%by the write operation.
%
%\subsubsection{Blob Storage Data Model}
%
%All the objects stored in the Blobstorage are stored through a key/value pair system. Each object is stored and accessed through the key, which is a string composed of 2 parts : container name and blob name. The container name is scoped by the account, which means you can freely name a container no matter how other customers build their container names.
%To build a container name string, you need to comply to several rules edicted by Azure :
%
%\begin{itemize}
%\item Container names must start with a letter or number, and can contain only letters, numbers, and the dash (-) character.
%\item Every dash (-) character must be immediately preceded and followed by a letter or number. There should be no consecutive dash characters
%\item All letters in a container name must be lowercase (This rule is often broken by mistake while using Camel Case)
%\item Container names must be from 3 through 63 characters long.
%\end{itemize}
%
%BlobName is a second string that is unique for any blob stored inside the container. ContainerName and BlobName are separated by the character"/".\\
%Here is some example of a valid key : \\
%
%some-container/blobname1\\
%
%Caution : using an incorrect blob name will cause a very unhelpful error message : IndexOutOfRange Exception.
%
%\subsubsection{Type of blobs}
%
%//To be rewritten :
%%
%%There are two types of blobs supported:
%%
%%    Block Blobs – targeted at streaming workloads.
%%        Each blob consists of a sequence/list of blocks.
%%        Max block blob size is 200GB
%%        Commit-based Update Semantics – Modifying a block blob is a two-phase update process.   It first consists of uploading blocks as uncommitted blocks for a blob.  Then after they are all uploaded, the blocks to add/change/remove are committed via a PutBlockList to create the updated blob.   Therefore, updating a block blob is a two-phase update process where you upload all changes, and then commit them atomically.
%%        Range reads can be from any byte offset in the blob.
%%    Page Blobs – targeted at random write workloads.
%%        Each blob consists of an array/index of pages.
%%        Max page blob size is 1TB
%%        Immediate Update Semantics – As soon as a write request for a sequential set of pages succeeds in the blob service, the write has committed, and success is returned back to the client.  The update is immediate, so there is no commit step as there is for block blobs.
%%        Range reads can be done from any byte offset in the blob.
%
%\subsubsection{Partition}
%
%To load balance the objects access, Azure is using a partitioning system. While a same partition key can be shared among multiple entities in TableStorage, the partition key is unique for each blob in BlobStorage and defined by ContainerName + BlobName. Azure is automatically load balancing the blobs across their servers based upon these partitions.\\
%
%For the TableStorage, all objects with the same partition key value are grouped into the same partition and are accessed from the same partition server. Thus, atomic operations can be run across objects in the same partition (because they share the same physical machine). Yet, since all objects stored in a partition are managed by the same partition server, scalability in this partition will be much harder to achieve.\\
%
%For the BlobStorage, since each object is stored in a different partition key, access to different blobs (even with the same ContainerName) can be load balanced across as many servers as needed in order to scale access.
%
%\subsubsection{Architecture and scalability}
%
%The storage access architecture is composed of the 3 following layers:
%
%\begin{enumerate}
%\item \textbf{The Front-End layer} is composed of multiple servers processing the incoming requests and routing them to the layer below : the partition layer. Each front-end server holds in cache the mapping between partitions and partition servers so it can routes request to the correct partition server.
%\item \textbf{The Partition layer} is composed of multiple servers managing the different partitions. One partition server can manage one or multiple partitions, but a given partition is managed by only one partition server. For the BlobStorage, this is obvious since the partition key is down to the blob name. A partition master monitors each partition server's load and determines load balancing.
%\item \textbf{The Distributed File System (DFS) layer} is responsible of persisting data on disk and replicating the data across multiple servers to avoid data loss. All partition servers can access any of the DFS servers.
%\end{enumerate}
%
%
%%Now that we have given a high level description of storage accounts, storage abstractions and how they are grouped into partitions, we want to talk about the scalability targets for storage accounts, objects and their partitions.
%%
%%The following are the scalability targets for a single storage account:
%%
%%    Capacity – Up to 100 TBs
%%    Transactions – Up to 5,000 entities/messages/blobs per second
%%    Bandwidth – Up to 3 gigabits per second
%%
%%The 100TB is a strict limit for a storage account, whereas the transactions and bandwidth are the current targets we’ve built the system to for a single storage account. Note, the actual transaction and bandwidth achieved by your storage account will very much depend upon the size of objects, access patterns, and the type of workload your application exhibits. To go above these targets, a service should be built to use multiple storage accounts, and partition the blob containers, tables and queues and objects across those storage accounts.  By default, a subscription gets 5 storage accounts, and you can contact customer support to get more storage accounts if you need to store more than that (e.g., petabytes) of data.
%%
%%It is expected that a hosted service needs up to as many storage accounts to meet its performance targets given the above, which is typically a handful of storage accounts to up to 10s of storage accounts to store PBs of data.   The point here is that a hosted service should not plan on creating a separate storage account for each of its customers.  Instead, the hosted service should either represent a customer within a storage account (e.g., each customer could have its own Blob Container), or map/hash the customer’s data across the hosted service’s storage accounts.
%%
%%Within a storage account, all of the objects are grouped into partitions as described above. Therefore, it is important to understand the performance targets of a single partition for our storage abstractions, which are:
%%
%%    Single Queue – all of the messages in a queue are accessed via a single queue partition. A single queue is targeted to be able to process:
%%        Up to 500 messages per second
%%    Single Table Partition – a table partition are all of the entities in a table with the same partition key value, and most tables have many partitions. The throughput target for a single partition is:
%%        Up to 500 entities per second
%%        Note, this is for a single partition, and not a single table. Therefore, a table with good partitioning, can process up to a few thousand requests per second (up to the storage account target).
%%    Single Blob – the partition key for blobs is the “container name + blob name”, so we can partition blobs down to a single blob to spread out blob access across our servers. The target throughput of a single blob is:
%%        Up to 60 MBytes/sec
%%
%%The above throughputs are the high end targets for the current system. What can be achieved by your application very much depends upon the size of the objects being accessed, the operations (workload) and the access patterns. We encourage all services to test the performance at the partition level for their workload.
%%
%%When your application reaches the limit to what a partition can handle for your workload, it will start to get back “503 server busy” responses. When this occurs, the application should use exponential backoff for retries. The exponential backoff allows the load on the partition to decrease, and to ease out spikes in traffic to the partition. If this is a regular occurrence, then the application should try to improve its data partitioning and throughput as follows for the different storage abstractions:
%%
%%    Blobs – consider using Windows Azure Content Delivery Network for delivering anonymous access to hot blobs to achieve higher throughput for commonly accessed blobs.
%%    Tables – use a more finely grained PartitionKey for the table in order to allow us to automatically spread the table partitions across more servers.
%%    Queues – batch together multiple work items into a single message to improve queue throughput or use multiple queues.
%%
%%See the next set of upcoming posts on best practices for scaling the performance of Blobs, Tables and Queues.
%%
%%Finally, one question we get is what the expected latency is for accessing small objects:
%%
%%    The latency is typically around 100ms when accessing small objects (less than 10s of KB) from Windows Azure hosted services that are located in the same location (sub-region) as the storage account.
%%    Once in awhile the latency can increase to a few seconds during heavy spikes while automatic load balancing kicks in (see the upcoming post on Windows Azure Storage Architecture and Availability).
%
%\subsection{Azure QueueStorage}
%
%Azure Queues provide a reliable asynchronous message delivery mechanism through distributed
%queues to connect different components of a cloud application. Queues are designed to store a large amount of small messages (with
%maximal individual size of 8 KB\footnote{\cite{AzureStorageResources} (6144 bytes more precisely, see \cite{AzureQueuesCannotBeLarger}))}). Using queues to communicate helps building
%loosely coupled components and mitigates the impact of individual component
%failure.\\
%
%Azure queues are not supposed to respect FIFO logic (First In First Out) as standard queue. On small scale, the Azure queue will behave like a FIFO queue, but if it is more sought, it will adopt a different logic so it can better scale. Therefore, the FIFO assumption cannot be assumed when designing a cloud app.\\
%
%Messages stored in a queue are guaranteed to be returned at least once, but possibly several times:
%this requires one to design \textbf{idempotent} jobs.  When a message is unqueued by a
%worker, this message is not deleted but it becomes invisible for other
%workers. If a worker fails to complete the corresponding job (because it
%throws some exception or because the worker dies), the message becomes
%available after a certain period of time. Through this process, one can make
%sure no job is lost because of e.g., a hardware failure.
%
%\subsection{Azure TableStorage}
%
%Azure Table Storage provides scalable structured storage. Structured storage is provided in the form of Tables, which contain a set of Entities, which contains a set of named Properties. No schema is stored by Windows Azure Table, so all of the properties are stored as $<name, typed ~value>$ pairs. This means that two entities in the same table can have very different properties. Combined size of all data in an entity cannot exceed 1MB. Entities are stored inside the table using a two part key composed of a PartitionKey (primary key) and a RawKey (secondary key). An application can manipulate the entities and query over any of the properties stored in a Table.\\
%
%We didn't use TableStorage in this thesis.
%
%\subsection{Serialization}
%
%\subsection{BlobStorage or TableStorage ?}
%
%BlobStorage and TableStorage are both designed to store data. When should we use one or the other ? Here are some part of the answer :
%\begin{itemize}
%\item One part of the answer is the data locality management and atomicity. Every data object may they be blobs or table entities are stored in terms of their partition key. This partition key determines how Azure must load balance and partition the objects across Azure servers. Partition Key for TableStorage is the tableName + partitionkey (primary key), whereas PartitionKey for blobs are determined by containerName + blobName. Since the partition key is down to the blob name for blobStorage, access to different blobs can be load balanced  across as many servers as needed to scale. But no atomic transactions is authorised for multiple blobs. As explained below in design subsection \ref{Redesign of the algorithms}, jobs will need to be idempotent : no atomic transaction for blobStorage is another difficulty to guarantee the idempotency. Since partition key is up to the primary key in TableStorage, we can run atomic transactions up to 100 entities who share the same table + primarykey (Azure guarantees that all the entities will be changed the specified way, or no one will).
%\item When dealing with really large objects, BlobStorage is easier to manipulate, since each blob can be arbitrarily large (up to 50 GB per blob). We do not need to break an object into a lot of entities, making sure each time each "part entity" respect the 1MB constraint. BUT WE HAVE TO DEAL WITH BLOCKS : SAY LOKAD.CLOUD HANDLES IT
%\item When manipulating objects than can be stored both in TableStorage and BlobStorage, BlobStorage is reported by \cite{Early_Observations_On_Azure} to be more effective in insert operations on blobs larger than 4KBytes, which is below the average size of blobs we are going to use.
%\item Price is another factor that might affect our choice. Using Azure storage, three things are charged : size of our storage, I/O communications, requests to the storage. A single query on TableStorage can load up to 100 elements at a time and costs as much as a single query on blobStorage, see \cite{Azure_Pricing}. If we have to store a lot of small objects on different places, using TableStorage will thus allow to run request on objects by packet of 100, so it would divide up to a factor 100 the bill of storage requests in comparison of running requests one by one on the BlobStorage. If we do not need to store the objects on different places, we could store them in one bigger blob, bounding the request to one. Yet, the more times we have to modify a small part of the storage, the less storing all the data in a single blob is the right thing, since we are going to download and upload the big blob each time a small part of the data need to be updated, making I/O costs predominant comparing to storage requests. So, using TableStorage or BlobStorage on the price criterion will depend on the size of your data, your application, the way the application is designed, your need to update small part of data or entire data each time, ...
%\end{itemize}
%
%To run our experiments, we chose to use the BlobStorage. Atomic updates on multiple object at once will not be a key point in our code, except for counters for which concurrency will be handled by etag mechanism. Price will be estimated but is not a criterion here. Besides, BlobStorage is easier to manipulate and due to its complexity, TableStorage suffered little bugs that affect performances when we ran our experiments (in June 2010).\\
%
%\section{Azure Framework}
%
%\subsection{Global description}
%Windows Azure Platform is Microsoft's Cloud Computing solution, in the form of
%Platform as a Service (PaaS). The underlying cloud operating system (Windows
%Azure) provides services hosting and scalability. It is composed of a storage
%system (that includes Blob Storage and Queue Storage) and of a
%processing system (that includes \emph{web roles} and \emph{worker
%  roles}). We will mainly discuss the storage elements we used in our experiments,
%  namely BlobStorage and QueueStorage.
%
%The architecture of an Azure hosted application is based on two components:
%\textbf{web roles} and \textbf{worker roles}. Web roles are designed for web
%application programming and were only used in our prototypes to monitor tasks executions
%and worker loads. Worker roles
%are designed to run general background processing. Each worker role typically
%gathers several \emph{cloud services} and uses many \emph{workers} (Azure's
%processing units) to execute them. Our prototypes always use only one worker role,
%several services and tens of PU. The computing power is provided by the
%workers.
%
%\subsection{Idempotency}
%
%\subsection{Affinity between workers and storage}
%
%Presently, there are no mechanisms in Azure that provides the same affinity between the workers and the storage than MapReduce does.
%This means that we cannot push some jobs where the data is stored since we can't know on which machine a data piece is stored. Each worker processing data will need to first download the data it needs to process, on the contrary of Google's MapReduce, where the jobs are scheduled and assigned on each worker by the framework in such a way that workers downloading data will be minimized.
%
%This is intended by Microsoft (CITATION NEEDED) to simplify development of cloud apps : one does not need to design its application to take data location into account, Azure providing a satisfactory scale-up and bandwidth. This choice is about simplicity over performance.
%
%\subsection{Workers Communication}
%
%Azure does not offer currently any standard API for distributed computation, neither a low
%level one such as MPI \cite{MPI1996}, nor a more high level one such as Map
%Reduce \cite{MapReduce2004} or Dryad \cite{Dryad2007}. MapReduce could be
%implemented using Azure components (following the strategy of
%\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{GraphLab2010},
%those high level API might be inappropriate for iterative machine-learning
%algorithms such as the k-means or learning vector quantization because of MapReduce overhead on synchronisation and waiting process.
%
%We therefore need to explicitly design our communication process between Azure workers, process that can realized by 2 means :
%
%\begin{enumerate}
%\item Workers can communicate each other through direct IP communications, that is to say re-implement a peer-to-peer network inside Azure cloud. TO DEVELOP
%\item Azure storage system can be used to implement synchronization and
%communication between workers : each worker is pinging (or "listening") a specific queue, and it gets some message from this queue holding
% some blobname in BlobStorage where it can retrieved the data it needs.
%\end{enumerate}
%
%While the first solution comes with higher performance in terms of bandwidth, as reported in AzureScope or ... (CITATION NEEDED), it implies a very tough work which would be like implementing MPI on Azure, which was outside the scope of this thesis. Moreover, the "storage approach" for communication is much more in agreement with Azure's abstractions and design thinking (CITATION NEEDED).
%Thus, we will rely therefore directly on Azure queues and blob storage to build all our communication systems.
%
%\subsection{Sharded Counters}
%
%\subsection{Lokad-Cloud}
%
%\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}}, an
%open-source framework that adds a small abstraction layer to ease Azure
%workers startup and life cycle management, and storage access.
%
%\section{Azure Performance}
%
%\subsection{Performance Tradeoffs in Azure}
%
%\textbf{Read/Write tradeoff} : As reported in \cite{BenchmarkingCloudServices}, distributed storage must choose a tradeoff between read throughput and write throughput. Taking the example of a blob in BlobStorage, a blob cannot be at the same time highly available in read operations and write operations (having a highly available blob for read operations implies the blob to be duplicated, so write operations must change several replicas, and write operations take more time). Azure let the traffic arbitrate this tradeoff : the more request to a blob per time unit, the more the blob is duplicated, the more it becomes available, and the more time it takes to run write operations.\\
%
%\textbf{Responsivity/Efficiency} \cite{BenchmarkingCloudServices} is also reporting some tradeoff between services responsivity and efficiency. Efficiency, as defined as the ratio of work completed per hour per worker is a decreasing function of workers and an increasing function of total workload. Therefore, to achieve the best possible efficiency, we should make sure every worker is overworked. Yet, the more overworked workers are, the least responsive they can be on an additional task. Since the first goal of our algorithm is responsivity (we want to minimize amount of time spent processing the algorithm), we will make sure to provide a pool of workers slightly greater than the number of jobs we can parallelize, so we make sure all jobs that can be run parallelingly, are (having P jobs run by P-1 workers would lead approximately to twice the time P jobs running by P or P+1 workers). \\
%
%\textbf{Synchronous Replication/Replication Latency tradeoff} : TO BE DIGGED. RELATED TO CONSISTENCY
%
%\subsection{Benchmarks}
%The Azurescope website \cite{AzureScope} is reporting performance benchmark run from February/March 2010 to presently (June 2010) on Azure hardware.
%Some paper have also been released very recently on same measures \cite{Early_Observations_On_Azure}.
%Numbers reported in \cite{Early_Observations_On_Azure} seem to be more accurate than \cite{AzureScope}, and the methodology is better explained, so we will use these numbers as reference.
%We will use the numbers observed to understand if bandwidth is an issue on Azure.
%
%\begin{itemize}
%\item BlobStorage Read Bandwidth : 13 MBytes/sec for 1 worker, BlobStorage aggregated read bandwidth bounded to 393 MBytes/sec when using more workers (each worker is using several threads to load data from storage). This quantity describes the bandwidth for several workers to download the same blob from the storage.
%\item BlobStorage Write Bandwidth : 5 MBytes/sec for 1 worker, BlobStorage aggregated write bandwidth bounded to 124 MBytes/sec when using more workers (each worker is using several threads to upload data). This quantity describes the bandwidth for one or several workers to load data from different blobs but within the same container.
%\item Communication throughput between 2 workers : 55 MBytes/sec on large messages on average (which is a very impressive bandwidth).
%\item CPU power depends on type of VM allocated : from 1 core (on small VM) to 8 cores on extra-large VM
%\item Queue throughput is presently 500 messages per second, but might be very increased soon.
%\end{itemize}
%
%Note : Since concurrent write on a blob is very costly, we will not write twice at a same place, unless when using counters.\\
%
%\section{Prices}


%\section{Some Cloud Computing offers}
%\label{cloudOffers}
%
%\subsection{Google App Engine}
%
%\subsection{Amazon}
%
%has MapReduce, but also lower level abstractions.
%
%\subsection{Azure}
%
%Presently, Azure is designed to provide PaaS. As explained in Azure FAQs,
%Microsoft is planning to add Virtual Machines (VM) functionality to enrich the
%different uses of Azure. Therefore, Azure might also become Infrastructure as a
%Service (IaaS). Yet, since no communication or services have been released yet,
%we will assume Azure is only PaaS oriented.

