\section{Introduction}

For the last twenty years, the technology improvements in computing hardware has led to a situation where a gigantic amount of data can be gathered, stored and accessed. The amount of processing power required to explore and use these data largely exceed the capacity of a single retail computer (referred in the following as commodity hardware). Therefore, several computing systems have been devised to tackle this issue. In previous times, more powerful systems known as supercomputers were created by improving custom hardware: this solution is referred as scaling-up. The present trend is on the contrary to run these large computations on a large set of commodity CPU working together, a solution referred as scaling-out. In other words, processing more data nowadays often consists in throwing more commodity hardware at the problem.\\

The parallelization of computation on a large amount of machines gives rise to many problems, like communications between computers, access to shared resources, or workload balance. The past two decades have been constantly providing engineered solutions to theses issues, some of them known as Grid Computing and Peer-to-Peer architectures. These solutions have led to well known software applications or results that had a deep impact on our everyday life, as is the case for Napster, Google Search, social networks like Facebook, etc.\\

While these considerations were challenging only few people a few years ago, the number of software companies involved in large-scale computations is growing quickly. This situation has led to the creation of a new economical market of storage and computation facilities. Some very large software actors decided to provide theses facilities as a commercial service, allowing new players to outsource their computing solution, making computation and storage a facility as electricity already is. These new commercial services are referred to the term Cloud Computing.\\

The fast growing interest in Cloud Computing over the past few years has led to a fuzzy and continuously evolving situation: many have heard of it, but few people actually agree on a specific definition. More importantly, even fewer understand how it can benefit them. From our point of view the best definition of Cloud Computing has been provided by Armbrust et al. in \cite{ARM1}: Cloud Computing \emph{refers to both the applications delivered as services over the Internet and the hardware and systems software in the datacenters that provide those services}.\\

From a consumer point of view, Cloud Computing allows any user to rent a large number of computing instances in several minutes to perform data/compute intensive jobs or build web applications. Since such instances can be dynamically provisioned up or down, it allows users to meet specific scale-up constraints. For example, this scaling elasticity let consumers face weekly data consumption peaks. The storage and compute capacities are provided as a service in a pay-as-you-go manner. Cloud Computing therefore disburden users from the hardware investment.\\

Many Cloud Computing solutions have already been developed. Among these cloud solutions, two categories appears. The first category regroups all the big Cloud Computing providers, as Google, Microsoft or Amazon. They often provide the cheapest prices of Cloud Computing solutions, through well-designed but fixed frameworks. The second category gathers all the small Cloud Computing offers. On the contrary to the former ones, the small actors provide custom cloud computing solutions to their customer. While the storage or CPU costs of these companies often cannot compete to the big providers prices, their economical advantage lies in all the specific services they provide to their customer. More precisely, the small Cloud Computing providers sometimes embody the ability for companies top management to challenge or replace some tasks previously reserved to their Information Technology (IT) department, for example in the case of a conflict between the top management and the IT. As a consequence, the cloud computing adoption may sometimes be the subject of internal political dealings. While the small and custom cloud computing offers is of prime interest as far as economy is concerned, the present chapter will not detail further these offers, as this chapter aims to focus on the most appropriate cloud solutions for intensive computations.\\

A survey of Cloud Computing realized in 2011 by TNS (\cite{CloudSurvey}) provides first feedbacks of early cloud solution adopters. The survey has been done on 3645 IT decision makers in eight different countries. The study states that 33\% of companies adopted cloud primarily to access information from any device rather than to cut costs. Indeed, while a large part of these companies saved money because of cloud adoption (82\%), the savings are on average small (more than half of the companies report less than \$20.000 savings). Besides of the costs reduction, Cloud Computing can contribute to improve services quality. 93 \% of all companies interviewed have noted some improvement in their services since their cloud adoption. Among this improvements, the hardware administration and maintenance disburden is a major element.\\

Cloud Computing is also a subject of interest for academic researchers. The use of Cloud Computing can provide them access to large-scale experiments platform without getting hardware access grant or hardware investments from their administration. All science intensive computing tasks do not fit the Cloud Computing constraints, especially because of the communication throughput and latency constraints. Yet, provided the tasks fit into the scope of Cloud Computing, even scientists with limited experience with parallel systems are provided the resources of a large distributed system.\\

One can foresee that the importance of Cloud Computing in the following years will keep growing, as many companies have expressed their will to resort to these cloud platform solutions. This dynamic is strengthened by the large research and development investments of the major cloud providers. These providers keep investing a lot of work and money to improve their services and tackle economical, ecological and performance challenges. For example, Microsoft planned in 2011 to spend 90 percent of its \$9.6 billion research and development budget on cloud strategy (according to an interview given by the Microsoft International President Jean-Philippe Courtois to Bloomberg (\cite{CloudInvestment})).\\

This chapter is a short overview of the current state of Cloud Computing (in early 2012). It is organized as follows. The second and following section gives a short introduction to the origins of Cloud Computing. The third section is devoted to a more in-depth definition of Cloud Computing. In particular, we compare Cloud Computing and Grid Computing, and give some insights of some of the different abstraction levels it targets. In a forth section, keys are given of cloud storage design, while the fifth section describes the main cloud execution frameworks. The last section of this chapter describes some commercial cloud computing offers.
%
%\subsection{Cloud Objectives}
%
%While Cloud Computing offers differ in their form, as detailed in section \ref{xaas}, they share the same concerns and objectives. This section lists the different challenges cloud address. Some of this points have already been discussed, and others will be discussed in following sections, \\
%
%\begin{itemize}
%\item \textbf{Scaling} is the ability of the system to support huge datasets (multiple terabytes) and very high request rates. Cloud systems achieve scaling by scale-out using large numbers of commodity servers (on the contrary to scale-up which is achieved by using more powerful hardware). An effective scale-out system balance load across servers. The choice between scaling-up and scaling-out is discussed in section \ref{cloudEmergence}.
%\item \textbf{Elasticity} is the ability to dynamically dimension and redimension the number of machines used to adjust to the present computation and storage needs. Elasticity is a side-effect of resources mutualization, as explained in section \ref{gridvsCloud}.
%\item \textbf{Availability} guarantees that every request receives a response. While total availability cannot be guaranteed on large systems, most Cloud Providers have been committing themselves into Service Level Agreement (SLA) to guarantee a minimum rate of effective availability. This availability is made harder as the cloud is made of cheap commodity hardware.
%\item \textbf{Consistency} is a set of properties (strong consistency, eventually consistency, ...) related to the way data are accessed and modified by all the computing instances. It is defined and discussed in section \ref{nosqlParagraph}.
%\item \textbf{Cost-effectiveness} is the ability to execute and complete data/compute intensive tasks on the cloud at a lower price than it would have cost on other hardware. It is a mandatory condition for computing consumers to switch on cloud computing offers. Pricing is briefly discussed in section \ref{cloudOffers}.
%\item \textbf{Hardware Administration free}:
%\item \textbf{Low latency responsiveness} describes the time data/compute requests take to complete and return answer to the consumer. It can concern latency between compute instances and storage, between consumers and storage, or between consumers and compute instances. To lower latency between consumers and the datacenters that host storage and compute instances, cloud providers have been positioning their datacenters all over the world, so that a request could be treated quickly, no matter its geographical origin.
%\item \textbf{Reliable Persistency} guarantees that data stored in the cloud remains available in the long term, independently of hardware failures. Persistency implies that each data chunk is replicated multiple times, which tends to complicate consistency and low latency responsiveness.
%\item \textbf{Security} is a major concern for a lot of potential cloud customers. More specifically, a result of cloud computing outsourcing is that the data privacy and accessibility management are also outsourced. While the cloud provider is in most cases technically more qualified than its customers to address security issues, the cloud provider is also subject to foreign laws. Despite the efforts of cloud providers to clarify the legal situation, it still remains fuzzy.
%\end{itemize}

%\item \textbf{Consistency/availability, throughput and latency tradeoff}:

\section{Origins of Cloud Computing}

\subsection{HPC and commodity hardware computing}

A large part of High Performance Computing (HPC) researches and experiments is implemented on custom HPC hardware, sometimes called supercomputers. While the exact beginning of custom HPC platforms is rather uncertain, it goes back at least to the 1960s and the work of Seymour Cray. These computing platforms have been successfully used in many research fields such as weather forecasting, aerodynamic research, nuclear explosion simulations, molecular dynamics simulation, drug discovery, etc.\\

During the 80's and 90's, the amount of personal computers all around the world has been very significantly increasing. The resulting scale economies has lead to a situation where it is nowadays often more cost-effective to stack cheap retail CPU instead of buying expensive dedicated hardware. In parallel with HPC development on supercomputers, research has therefore been made to distribute computations on commodity hardware.\\

This situation is summarized by Drepper in \cite{Drepper}: \textit{"It is important to understand commodity hardware because specialized hardware is in retreat. Scaling these days is most often achieved horizontally instead of vertically, meaning today it is more cost-effective to use many smaller, connected commodity computers instead of a few really large and exceptionally fast (and expensive) systems"}.\\

To distribute computations on commodity hardware, initial solutions have been provided to network together distinct commodity computers. For example, the Parallel Virtual Machine (PVM) framework originally written in 1989 is a software system that enables a collection of potentially heterogeneous computers to simulate a single large computer. A second major framework is the Message Passing Interface (MPI) framework, whose version 1.0 has been released in 1994. This second framework will be more explicitly described in chapter \ref{chap:KMeans}. The hardware systems composed of multiple commodity computers and networked through software as PVM or MPI are called Beowulf clusters.\\

An halfway solution between supercomputers and Beowulf clusters have been introduced several years ago. This solution mixes the cost efficiency of commodity CPUs with the throughput of dedicated hardware such as custom communication bus and ethernet cards. This solution is therefore composed of commodity CPUs and RAM memory, along with custom and dedicated power supply devices, motherships, and intermachines connections. All the different components are screwed on racks, then gathered in cabinets.\\

Grid Computing is a general term referring to hardware and software configurations for distributing computations on commodity CPUs, whether in Beowulf clusters or in custom racks and cabinets. The following section described more precisely the Grid Computing frameworks.\\

\subsection{Grid Computing}
\label{grid}

Grid Computing (or grid for short) is a distributed computing system composed of many distinct networked computers (sometimes referred as nodes). These computers are usually commercial standard hardware gathered into a grid platform. The different components are by design loosely coupled, which means each computer has little knowledge of the existence and identity of the others. A dedicated low-level software layer called middleware is used to monitor and load-balance tasks between the nodes.\\

On the contrary to supercomputers, the network that connects the different devices of a Grid can be rather slow. In the supercomputers case, all the processors are geographically very close and the communication is endorsed through an efficient local computer bus. In the Grid Computing case, the communication goes through a Local Area Network (LAN) or a Wide Area Network (WAN), which cannot compete with the throughput of a local bus. This communication constraint on Grid Computing induce that Grid is very well-suited on large batch jobs where there are intensive computational requirements and small communication needs between machines.\\

Grid Computing architectures have been implemented in many different ways, some of them being simple gathering of physical devices where administrators and end users are the same persons. In other cases, the system is a more complex entity composed of distinct collaborative organizations, referred as administrative domains. In the latter case, the middleware manages the whole system administration, grants users access to the resources, and monitors all participants to prevent malfunctioning or malicious code to affect the results. One of the first middleware handling the multiple administrative domains is Condor (\cite{Condor}).\\

Grids have been successfully set-up and used in many production fields. For example, it is used by Pratt \& Whitney, by American Express, in banks for Value At Risk estimation, pricing or hedging. It is also used by the National Aeronautics and Space Administration (NASA) in various projects such as the Information Power Grid (IPG) project. Grids have also been provided for academic researches, as it is the case for the Large Hadron Collider (LHC) in the form of the LHC Computing Grid, or for Grid 5000 \cite{grid5000}.\\

One of the well known form of Grid Computing appeared in the 2000's, resulting from two major events of the 90's: the personal computer democratization and the internet explosion. This new form of Grid Computing is known as Peer-to-peer architecture, which is a Grid Computing platform composed of a collaborative set of machines dispatched worldwide and called peers. This architecture removes the traditional client/server model: there is no central administration and peers are equally privileged. They share a portion of their own resources (hard-drive, network bandwidth, CPU, etc.) to achieve the tasks required. Peer-to-peer model has been popularized by very large-scale distributed storage system, resulting in peer-to-peer file-sharing applications like Napster, Kazaa or Freenet \cite{freenet}. Since then, Peer-to-peer architectures also have been successfully applied on very intensive computational tasks. Some of the largest projects are listed by the Berkeley Open Infrastructure for Network Computing (BOINC) \cite{BOINC} like Seti@Home or Folding@Home \cite{folding@home}.\\

\subsection{Emergence of Cloud}
\label{cloudEmergence}

During the 2000s, the internet boom caused some very large companies like Google, Facebook, Microsoft, Amazon, etc. to experience computation and storage challenges on datasets of a scale hardly ever met. In addition to the scale-out of their systems, they faced two supplementary challenges. Firstly, they needed to create abstractions on storage and distributed computations to ease the use of these systems and allow non distributed computing experts to run large-scale computations without getting involved of the actual parallelization process. Secondly, they were concerned with minimizing the operating cost per machine in their datacenters.\\

These constraints implied to partially dropping the previous mainstream datastorage system (this situation is detailed in section \ref{nosqlParagraph}) along with developing new software and hardware frameworks to easily express distributed computations requests. These companies continued the works on commodity hardware and merged different innovative technologies into coherent technology stacks, on which we give a short overview in section \ref{technoStack}. They build large datacenters hosting tens of thousands of commodity machines and used the technology stacks they improved to meet their internal computations and storage needs.\\

Initially, the infrastructures and datacenters as well as the new storage systems and the software frameworks for distributed computations were designed to be tools exclusively dedicated to the companies internal needs. Progressively, some of these tools have been transformed into public commercial products, remotely accessed through Application Programming Interface (API) as paying services. The companies that made these commercial offers became Cloud Computing providers.\\

\section{Cloud design and performance targets}

\subsection{Differences between Cloud Computing and Grid Computing}
\label{gridvsCloud}

A classical distinguishing criterion between Cloud Computing and Grid Computing is the uniqueness or multiplicity of administrative domains. Clouds are assumed to be rather homogeneous servers that reside in datacenters controlled by a single organization, while grids are assumed to be a federation of distinct collaborative organizations sharing (potentially heterogeneous) resources. A consequence of the multiple administrative domains in Grid Computing is that Grids need to handle the additional constraints of a federated environment, e.g., checking credentials and agents identities, but also monitoring the system to detect malicious usages. This additional constraints, but also the potential lower bandwidth between two distinct administrative domains increase the communication burdens and therefore tends to require coarser granularity for the tasks distributed on the Grid than those distributed on the Cloud.\\

A second aspect of cloud computing is the intensive use of Virtual Machines (VM). A VM is a software implementation of a computer that executes programs like it was a physical machine. VM implementations were already available several years ago, but they were uniquely software based and therefore suffered from significant impact on overall performance. Nowadays the VM implementations are both hardware and software based together, reducing from a vast margin the loss due to virtualization. While original motivation for virtual machines was the desire to run in parallel multiple operating systems, cloud computing is intensively using VM to build a greater degree of abstraction of the software environment from its hardware. The use of VM provides to cloud computing desirable features such as application provisioning and maintenance tools, high availability and easier disaster recovery.\\

A third difference between grid and cloud is ownership. In most Grid Computing examples, consumers as a group own the resources they use. In the cloud case, resources are totally outsourced to the cloud provider and don't need access grant anymore: cloud consumers are paying for resources in a pay-as-you-go manner. Cloud Computing consumers don't directly access resources, rather they access them through something like a service they pay for. By adopting cloud, computing consumers do not need to invest in and own hardware and infrastructure (data center, power supply, cooling system, ...) as they would probably need in the Grid Computing case.\\

A side effect of this ownership shift is multi-tenancy and elasticity. Elasticity is the ability for the cloud consumer to provision up or down the number of computing instances as well as the data storage system size. Since different resources are gathered into a specific cloud company actor instead of being hold by each customer, the resources are mutualized. This is this resources mutualization that allows elasticity, in the same way risks mutualization allows insurance mechanisms. This dynamic reconfiguration of the resources allows each consumer to adjust its consumption to its variable load, therefore allowing for an optimum resource utilization. In most cloud computing systems, the computing instances number can be redimensionned in several minutes.\\

A last important difference between cloud and grid is hardware administration and maintenance. Cloud consumers do not care about how things are running at the system end. They merely express their requests for resources and the cloud computing service then maps these requests to real physical resources. It therefore disburdens customers of a part of system administration. As electricity became in the 1930's available to consumers when national electricity grid were developed, cloud computing is an attempt to commoditize IT. The administration part left to the cloud consumer depends on the type of the cloud offer chosen (Infractruture as a Service, Platform as a Service, etc.) as detailed in the following section.\\

From a computing consumer point of view, Cloud Computing is therefore a Grid Computing with a single administrative domain, where the physical resources management has a clear outsourced centralized ownership and hardware is accessed indirectly through abstractions and services. There are much less commitments for the consumer since no initial investments are needed anymore.\\

\subsection{Everything-as-a-Service (XAAS)}
\label{xaas}

A Cloud Computing environment is a three actors world. It is composed of the cloud provider which supplies hardware ownership and administration, the cloud consumer using this hardware to run applications (the cloud consumer often being a software company), and the application consumer, which is using the applications running on the cloud. The three actors are related through the following relations: the cloud consumer is the customer of the cloud provider, and the application consumer is the customer of the cloud consumer.\\

It is up to the cloud consumer to choose which part of the application building he wants to handle by himself and what part will be built using the cloud provider pre-made libraries. This decision mostly depends on the cloud consumer's will to get involved in low-level software considerations. More specifically, there is a tradeoff for the cloud consumer between higher control on the hardware and systems he consumes, and higher level programming abstractions where developing applications is made easier.\\

To address the different choices of the cloud consumers, each cloud system has developed distinct strategies and API, as reported in \cite{BenchmarkingCloudServices}. A mainstream classification of cloud computing offers is therefore based on the programming abstraction level proposed by the cloud providers. Let us provide a short description of these different abstraction levels.

\begin{itemize}
\item \textbf{Infrastructure as a Service} (IaaS) is the lowest cloud abstraction level. Resources (compute, storage, and network) are exposed as a capability. Cloud consumers are relieved of the burden caused by owning, managing or controlling the underlying hardware. They therefore do not access directly the physical machines, but indirectly through VM.
    %The resources available to an application run by a VM are strictly bounded by the resources allocated to this VM.
    IaaS consumers are given almost full control on the VM they rent: they can choose a pre-configured Operating System (OS) image or a custom machine image containing their own applications, libraries, and configuration settings. IaaS consumers can also choose the different internet port through which the VM can be accessed, etc. A VM in IaaS can run any application build by the cloud consumer or anyone else. The main example of IaaS offers is Amazon Elastic Cloud Compute (EC2).

\item \textbf{Platform as a Service} (PaaS) is a higher cloud abstraction level than IaaS, designed to ease applications building and hosting. On the contrary to IaaS on which one can choose the OS, the configuration settings and run any application, PaaS provides much less freedom to the cloud consumer. He is given a fixed and constrained execution environment run on a specialized OS, and he is only in charged of defining the code to be run within this framework. The PaaS environment manages by itself code deployment, hosting and execution. PaaS offers may also provide additional development facilities, such as application versionning and instrumentation, application testing, system monitoring, etc. In the case of PaaS, the storage is mostly provided as a service in the form of a remote abstracted shared memory storage. Main examples of PaaS are Amazon MapReduce with Simple Storage Service (S3), Microsoft Azure, BungeeLabs, etc.

\item \textbf{Software as a Service} (SaaS) is the highest cloud abstraction level where applications are exposed as a service running on a cloud infrastructure. On the contrary to the customer of IaaS or PaaS, the SaaS consumer is not involved in any software development. This customer is only an application consumer communicating through internet with the application which is run on the cloud. Examples include GMail, Salesforce.com, Zoho, etc.\\
\end{itemize}

These distinct abstraction level technologies have different positioning on some tradeoffs:

\begin{itemize}
\item \textbf{abstraction/control tradeoff}: PaaS provides higher abstractions to develop applications than IaaS, but the application build on top of PaaS needs to conform and fit to the framework provided. As outlined in \ref{MapReduceCriticism}, some processes hardly fit into a rigid framework like MapReduce. On the other hand, IaaS will provide to cloud consumers a higher control on the machines but it comes at a price of fewer off-the-shelf abstractions: the same distributed computation challenges will probably need to be repeatedly solved.
\item \textbf{scalability/development cost tradeoff}: Applications run on a PaaS environment prove to be easily scalable. Yet, this scalability comes as a side-product of a strong design of the application such that it fits into the framework provided. On the contrary, applications run on an IaaS environment provides more flexibility on the way the application can be build, but the scalability is customer's responsibility.
\end{itemize}

\subsection{Technology stacks}
\label{technoStack}

Cloud computing frameworks are different hardware and software technologies merged together into technology stacks to handle thousands of machines. This section highlights some of these hardware and software technology stacks developed by first row cloud actors.\\

\subsubsection{Hardware technologies}

In this section we describe how some of the cloud challenges are achieved, namely the minimization of the operating cost per machine and the improvement of bandwidth throughput between machines. The cost issue is managed by lowering energy consumption and improving automation (we refer the reader to \cite{CloudAutomation} for an illuminating introduction to automation challenges in the cloud). The bandwidth throughput are optimized with specialized hardware improvement and network topology refinements.\\

According to Jon Koomey (Consulting Professor at Stanford) on his blog, \textit{"Cloud computing is (with few exceptions) significantly more energy efficient than using in-house data centers"}. Yet, more than $40\% $ of the amortized cost of a datacenter is still energy consumption, according to \cite{CloudCost}. This energy consumption is optimized through special care of the cooling system and minimization of power distribution loss (which respectively account for 33\% and 8\% of the total energy consumption of datacenters, according to \cite{CloudCost}).  While early days datacenters were built without paying much attention to those constraints, the most recently built datacenters are located in places where power or cooling are cheap (for example on a river in Oregon for a Google datacenter, or in the Swedish town of Lulea for some Facebook datacenter).\\

Since automation is a mandatory requirement of scale, a lot of work has been pushed to improve it in datacenters. According to (\cite{CloudCost}), a typical ratio of IT staff members to servers is 1:100 in an efficient enterprise and 1:1000 in an efficient data center. This is partially due to technologies developed by hosting companies (like Rackspace or OVH) or backup companies (like Mozy, funded in 2005). Machines virtualization (for example VM-ware or Microsoft Hyper-V) increase administrators productivity by letting them handle much more machines. Machines are replaced when the physical failing machines in a rack or in a container hits a given ratio. Other techniques of automation can be found in \cite{CloudAutomation} or \cite{lenk2009witcaamotcl}.\\

A key factor in cloud performance bandwidth (whether it is intra or inter datacenters bandwidth) is the spatial disposition of the hardware in the datacenter, as long as the network topology between the different nodes dedicated to data storage and computing. While a lot of research has been made on this subject, major cloud companies have been disclosing few about this major technology challenge and their own implementation. According to \cite{CloudConsistency}, Microsoft plans to upgrade some of its datacenters following the networking architecture studied in \cite{DatacentersDesign}.

\subsubsection{Software technologies}

Several software technology stacks have been developed to ease the access to CPU and storage. They split into three abstraction level : the storage level, the execution level, and the DSL level (domain-specific language).\\

\begin{itemize}
\item The storage level is responsible of preserving the data into the physical storage. It handles all the issues of distributed storage management, of data contention and provides a structured access to the data. It can be SQL or No-SQL storage, as described in section \ref{nosqlParagraph}. This storage can also be used as a communication media between machines.
\item The execution level defines the general execution framework. It specifies how machines are supposed to organize themselves, it monitors workers and restarts failing machines. Depending on the type of Cloud Computing offers (IaaS, PaaS, ...), the execution level can provide on each computing device anything a standard OS could achieve, or it can give a much more constrainted execution framework where the execution flow must be designed in a specific way (the most famous example on such a constrained execution framework is MapReduce, described in section \ref{MapReduceSection}).
\item The DSL is a higher level execution framework than the execution level. It often provides a declarative language instead of the procedural approach provided by the execution level, therefore avoiding users to specify how the work is supposed to be parallelized. It provides automated and under the hood parallelization, scheduling and intermachines communications. The DSL level therefore help users focus only on defining the result they want, and not how they want it to be computed. DSL implementations are often build in a SQL-like manner, allowing people familiar with this syntax to run requests in a similar way on a underlying No-SQL storage. Besides, since it provides a run-time framework to run those requests, it disburdens users from compiling their software each time they modify or add a request. Historically, they have been added years after the introduction of the execution level to provide to non developers specialists a mean to easily use the execution level. PigLatin, Sawzall, DryadLinq are well-known examples of DSL implementations.\\
\end{itemize}

The storage level can be used independently of the two other levels as an outsourced storage service. The execution level can also be used independently of the storage level and the DSL level, provided that no persistency is required. However, most cloud applications are built using both storage and execution level environments. The DSL level is an optionnal layer provided to ease implementation between storage and environment levels. Theoretically, each project at a given level could be substitutable to any other project at the same level so we could for example plug and use any storage framework with any execution framework. In practice, some components are designed to work only with other components (as it is the case for the original MapReduce framework which require some knowledge about where the data are stored for example). Therefore, there are four main frameworks that provide a full technology stack holding the three abstraction level.\\

The first framework has been presented by Google. It is the oldest one. On the storage level, it is composed of Google File System \cite{GoogleFileSystem} and BigTable \cite{bigtable} (the latter one being built on top of Google File System). The execution level is represented by the original MapReduce version (described in section \ref{MapReduceSection}): Google MapReduce \cite{MapReduce2004}. After MapReduce release, Google engineered a dedicated language, on the DSL level, build on top of MapReduce that provides a higher level access to the storage and the computation framework: Sawzall (\cite{sawzall}).\\

The second technology stack is the Apache Hadoop project (\cite{Hadoop}). This is one of the most famous open-source framework for distributed computing. It is composed of several widespread subprojects. The storage level includes HBase, HDFS, Cassandra, etc. The execution level includes Hadoop MapReduce, but can also be hand-tailored with the help of distributed coordination tools as Hadoop Zookeeper. The DSL level includes Hive and Pig. Pig is a compiler that processes Pig Latin code to produce sequences of Map-Reduce programs that can be run for example on Hadoop MapReduce. Pig Latin \cite{piglatin} is a data processing language that is halfway between the high-level declarative style of SQL and the lower procedural style of Map-Reduce. Pig Latin is used for example at Yahoo! \cite{piglatin} to reduce development and execution time of data analysis requests.\\

The third technology stack is composed of Amazon Web Services (AWS) and AWS-compatible open-source components. Its most used storage components are Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), Amazon SimpleDB, Amazon ElastiCache or Amazon Relational Database Service (Amazon RDS). AWS execution level is mostly composed of Amazon Elastic Cloud Compute (Amazon EC2) and Amazon Elastic MapReduce (which uses a hosted Hadoop framework running on top of Amazon EC2 and Amazon S3). AWS is one the first large commercial cloud computing offer (it was initially launched in July 2002, and is estimated as a 1 billion dollar business in 2011 \cite{AmazonBusiness}).\\

The fourth framework has been developed by Microsoft. The storage level is embodyed by distinct independant projects : Azure storage, Cosmos or SQL Server. Cosmos is a Microsoft internal project that has been used for Search teams and BING. Azure storage is a different project, initially built using a part of the Cosmos system (according to \cite{AzureStorage}). Azure storage  is designed to be the storage level standard layer for applications run on Azure. The execution level being hold by Dryad \cite{Dryad2007} or by Azure Compute (or small frameworks like Lokad-Cloud \cite{LokadCloud} (on top of Azure Compute) or Lokad-CQRS \cite{LokadCQRS}). The DSL level is represented through 2 differents projects : DryadLINQ (\cite{DryadLINQ}) and Scope (\cite{scope}).\\

The figure \ref{fig:StorageExecutionAndDsLevel}, taken from a blog post of Mihai Budiu, summarizes the three frameworks.

\begin{figure}[!h]
\begin{center}
\includegraphics[scale = 0.6]{../../Pictures/Cloud/msparallel.jpg}
\end{center}
\caption{Illustration of the three most popular technology stacks for cloud applications building.}
\label{fig:StorageExecutionAndDsLevel}
\end{figure}

\section{Cloud Storage level}
\label{nosqlParagraph}

Persistency is a data characteristic that ensures data outlive the process that created them. A data storage is a system that provides data persistency and is a fundamental element in most application developments. In the case of parallel computing, data storage can also be used for a different purpose as an inter-machines communication medium.\\

While working on a cloud platform, no hardware shared memory is available since the machines are physically distant. In such a situation, inter-machines communications can be achieved through direct IP communications or through a distributed data storage system. While the direct IP communications have been much favored in several systems such as most Peer-to-Peer systems and are still largely used in Amazon EC2, the use of a data storage system as a communication medium is widely used in numerous cloud computing platforms. Since the cloud storage is abstracted and accessed "as a service", it is easier to communicate through a data storage than to physically access to a virtualized machine and the cloud storage has been designed to easily scale and address satisfactory communication throughput.\\

In this section we describe the challenges involved by a large-scale distributed data storage system. We introduce the Relational DataBase Management System (RDBMS), which was several years ago the undisputed data storage system. We explain why some of the guarantees of RDBMS impose constraints hard to meet in a distributed world. We introduce the reader to some new data storage systems, known as No-SQL storage, which relax some of the guarantees of the relational database systems to achieve better scaling performance.

\subsection{Relational storage and ACID properties}
\label{acid}

Relational databases are structured through the relations between data entities. They are inspired by relational algebra and tuple calculus theories, while they do not respect all of their constraints. Relational databases are well understood and offer great guarantees for reliably storing and retrieving data in a robust manner. In addition to these guarantees, the relational model proved to be well-suited to express real-world business data interdependencies. RDBMS are therefore widely used in business applications.\\

A Structured Query Language (SQL) is provided together with the RDBMS to ease the data usage. The SQL tool is a declarative programming language (with some procedural elements) intended for managing data queries. Declarative languages express what computations should be done rather than explicitly defining how the computations should be done. In the case of relational storages, queries are expressed in SQL to define requests to the desired data, while the underlying RDBMS performs the actual operations to answer the requests. The listing \ref{sql_listing} gives an example of a SQL query.\\

\begin{lstlisting}[caption=a SQL query sample, label = sql_listing]
SELECT T1.variable1, T1.variable2
FROM Table1 T1, Table2 T2
WHERE T1.variable1 = T2.variable3
ORDER BY T1.name
\end{lstlisting}

A single "business" operation applied on data stored in a database is called a transaction. A single transaction can affect several pieces of data at once. One can understand the guarantees provided by RDBMS by observing how transactions are applied on the database. The following paragraphs introduce to the main relational storage guarantees by describing how they are traduced in term of transactions management.\\

\begin{itemize}
\item Consistency is a property related to the way data are modified during transactions. There are multiple definitions of consistency, resulting in different consistency level and dependant of whether the storage is distributed or not. Initially, consistency was a property that guaranteed that some user-specified invariants on data were never broken. For example, such an invariant could be that the sum of some variables equals the sum of another variables. With the rise of distributed storages, the definitions of consistency have been multiplied, and consistent distributed storage (also called strongly consistent storages) often refers to the fact that each update is applied to all relevant nodes at the same logical time. A consequence of this definition is that if a piece of data is replicated on different nodes, the data piece will always be the same on each node, for each given instant. A more in depth definition of consistency can be found in \cite{CAP_Theorem}.\\

\item Atomicity guarantees each transaction is applied as if it was instantaneous. More specifically, a transaction composed of $n$ database elementary operations is atomic if all the operations occur or none occur (in the case of one or several operations fail). A storage has the atomicity property if any transaction can be guaranteed to be atomic. This property guarantees that any transaction applied on a coherent database does not corrupt it.\\

\item Isolation guarantees that each transaction should happen independently of other transactions that occur at the same time.\\

\item Durability enforces that data are persisted permanently and that transactions remain permanent, even in the presence of a system failure.\\
\end{itemize}

These four important guarantees are gathered under the acronym ACID, which stands for Availability, Consistency, Isolation and Durability. These properties are guaranteed by every relational storage. In the case of a distributed relational storage, some other very mandatory properties can be desired, such as Partition tolerance and availability.\\

\begin{itemize}
\item A network is partitioned when all messages sent from nodes in one component of the partition to nodes in another component are lost. Distributed storage are partition tolerant when they continue to run in the presence of a partition failure.\\

\item A distributed storage guarantees availability if every request received by a non-failing node in the system returns a response.\\
\end{itemize}
\subsection{CAP Theorem and the No-SQL positioning}
\label{Cap_theorem_section}

The CAP theorem was first expressed by Eric Brewer at the 2000 Symposium on Principles of Distributed Computing (PODC). The CAP theorem has then been proved in 2002 by Seth Gilbert and Nancy Lynch in \cite{CAP_Theorem}. This theorem is a theoretical impossibility result, which states that a distributed storage cannot achieve both Availability, Consistency, and Partition Tolerance.\\

More specifically, this theorem is a theoretic traduction of the well known practical challenges of applying ACID transactions on data split on multiple machines. The ACID constraints require a lot of inter-machines communication to complete a single transaction (see for example \cite{TwoPhaseCommitProtocol}) and lead to serious lock contention that deeply decrease transactional throughput.\\

Moreover, guaranteing that data stored in cheap and unreliable hardware are highly available implies multiple replications of each single piece of data. Since data servers are expected to fail from time to time, data are duplicated several times (Azure storage is reported to hold for each data 6 replicas : 3 replicas in a first datacenter and 3 others in a backup datacenter (\cite{AzureStorage})), so a data server can die without any data loss. To guarantee strong consistency while keeping multiple versions of any data pieces in different locations causes large overheads, as reported in \cite{CLAP_Presentation}.\\

Scaling traditional ACID-compliant relational databases to distributed shared-nothing architectures with cheap machines therefore turned out to be very difficult. Data storage developed in the 2000's known as NoSQL storage (for Not Only SQL) such as Cassandra, HBase, BigTable, SimpleDB, etc. therefore dropped some of the ACID guarantees to achieve better scaling and lower latency. As stated by Daniel Abadi and Alexander Thomson in their blog, "NoSQL really means NoACID" (see \cite{NoSQLmeansNoAcid}).\\

Some of them (like BigTable or Azure BlobStorage) decided to provide weakened atomicity and isolation guarantees for transactions. Indeed, transactions that modify more than one data are not guaranteed to be either all done or none. Therefore, no logical transaction impliying several pieces of data can be guaranteed to be atomic at the storage level. It is up to the storage user to ensure logical transactionalities, and that user-specified invariants are not modified.\\

Other cloud storage systems like Cassandra have been dropping strong consistency between replicas to only provide a weakened consistency guarantee called eventual consistency. We refer the reader to the following paper for an introduction to eventual consistency (\cite{Eventually_Consistent}), while an introduction to BASE (Basically Available, Soft state, Eventually consistent) can be found here (\cite{BASE}). While the eventual consistency is satisfactory enough for a given number of well known web applications such as Facebook, the loss of strong consistency is a very difficult issue for many other real world applications, which make application development much harder. This is a key aspect of No-SQL storage: the responsabilities are partly shifted and the scalability comes at a cost of a stronger developer burden. The persistency of different cloud storage systems have been investigated in \cite{CloudConsistency}.\\

%Recently, the relational database model features, known as ACID, have been challenged. Brewer's CAP theorem (Consistency, Availability and Partition tolerance) \cite{CAP_Theorem} states that distributed storage won't be able to propose ACID properties in an acceptable way. Abadi recently explained some defaults of CAP theorem \cite{CLAP_Presentation}. During this time, web services delivered by very large companies like Google, Amazon, Yahoo, Microsoft... have been partially abandoning relational database and ACID properties, starting the NoSQL movement (NoSQL standing for Not Only SQL). Even some SQL-related people recently expressed their interest for NoSQL database, like Ayende (NHibernate contributor) did  \cite{Slaying_Relational_Dragons}.


\subsection{Cloud Storage Taxonomy}

Along with the calling into question of ACID properties, the relational data model which is related to them has also been challenged. No-SQL storages therefore provide different data models, often noted to provide a more fuzzy division between the database model and the application. In this paragraph we provide some description of the most used database models. A more exhaustive list of these models can be found at \cite{NoSQL}.\\

Document-oriented databases assume data are organized around the concept of documents. Each document holds data and is persisted through a standard data format such as XML, YAML, JSON, etc. or even through binary or .pdf files. On the contrary to RDBMS where the data scheme is fixed, document-oriented databases do not impose that each document owns the same fields than the others. Thus, new information can be added on a document without needing to fill the same information for all the other documents available. Each document can be accessed through a unique key, which is often a string. CouchDB, MongoDB, SimpleDB are well known document-oriented databases.\\

Graph databases are structured on the standard graph theory concepts of nodes and edges. The information are stored on nodes, each node representing a different entity. Each entity is entitled a set of properties describing the entity. Edges connect nodes to nodes or nodes to properties and describes the relationship between the nodes and the properties. Because of this graph structure, data and more specifically nodes are given a adjacency notion which is index-free and only driven by the data nature. Horton, FlockDB, AllegroGraph are graph databases.\\

Key-Value pairs databases are schema-free databases. Data is persisted on a material device in the form of a serialized programming language datatype, typically an object. Any type can be persisted and data are accessed through a key, which can be a string or a hash of the data content. Apache Cassandra, Dynamo, Velocity, BigTable, Azure BlobStorage are key-value pairs storage.\\

%We have been noticing that system un-availability is not only theoretical. When accessing to the storage, we can get notified an Azure internal Exception. Therefore, we have to implement retry policies so we can try again to run a read or write operation. Since storage is designed to be accessed by internal workers but also by outside (we can think for example of a picture or a video stored and devoted to be downloaded a lot of time), it makes no sense to guarantee 100\% availability since internet connection does not give such guarantee.\\
%
%\begin{itemize}
%\item \textbf{Block commits} : Data bigger than 4MBytes must be split into blocks before being uploaded. Once every block has been uploaded into Azure, the "transaction" is commited, and data is finally wrote on Azure side. This 2-phase upload (partial uploads, then integral commit) is TO BE FINISHED
%\item \textbf{Etag} : As reported in the BlobStorage and TableStorage technical papers \cite{Azure_Storage_Resources}, Azure provides a kind of timestamp, called etag. When updating some data, we first download the data, update it, and upload it with the etag provided. When replacing the data, Azure checks that etag of data being replaced matches the etag provided by the upload. If it does not, the overwrite is rejected, an exception is thrown, so the user can retry. This mechanism helps to deal with multiple workers trying to modify a data at the same time, but is very probably also used inside Azure to ensure consistency between the different replica of a same data.
%\item \textbf{Very few overwrites} Most applications do not need to overwrite data. Some typical usage could be for example a big video upload once, and hundreds or thousands of read request on this data (Facebook usecase). Clusterings we run on Azure required very few overwrite (mostly for counters), as it will be shown.
%\end{itemize}



%The phrase ``software as a service'' (SaaS), as explained in Vecchiola et al. \cite{VEC1} refers to \emph{solutions that are at the top end of the Cloud computing stack and they provide end users with an integrated service comprising hardware, development platforms, and applications}. For example the popular webmail GMAIL is based on a SaaS model. The application is accessible for the user through the internet as opposed to a traditional desktop application (e.g. Microsoft Excel 2010). At a much lower level we have the ``infrastructure as a service'' (IaaS) which provides low level hardware and the users may control a large part of the software environment. However, the creation of automatic provisioning of resources is not easy due to few infrastructure abstractions. Amazon EC2 is one of the most popular IaaS on the market. The third area of ``as a service'' paradigm is platform as a service (PaaS) that can be though as an intermediate abstraction level between IaaS and SaaS. As defined in Vecchiola et al in \cite{VEC1} a PaaS solution provide \emph{an application or development platform in which users can create their own application that will run on the cloud}. A PaaS solution comes with a framework and a set of API to build the application for the cloud. It also comes with a portal that will help the developer to run the application on the top of a fully abstracted infrastructure. In both PaaS and IaaS approaches the elasticity of resources is guarantee by a fine use of visualizations tools which help to perform quickly replications of virtual machines. This aspect is therefore much more abstracted in a PaaS offer. The main actors proposing PaaS solution are Google AppEngine and Microsoft Azure.\\
%
%Cloud Computing solutions have a deep impact on economic models. The pay-as-you-go concept brings the possibility to a company to pay only for what it uses. As an example it can allow $100$ servers for few hours to respond a peak traffic on their web application and one small server after that. Indeed, the cost of $100$ hours of CPU with one machine is the same that using $100$ machines for one hour. Cloud computing short-term (in minutes) elasticity ability of downsizing and upsizing resources available will make the risk of underprovisioning disappear and will reduce the costs due to overprovisioning. Amazon CEO Jeff Bezos in \cite{BEZ1} tells the story of Animoto, a company that used intensively the elasticity of Amazon EC2. Their resource needs pass in three days from $50$ Amazon EC2 instances to $3500$. The software company Lokad propose various SaaS application (Salescast, Callcast, etc...) which use data from their customer (mostly enterprizes) and deliver figures for business intelligence (mostly forecasts and related criterion). The Lokad forecasting engine, deployed in Microsoft Azure PaaS, make complex computations and then requires a lot of CPU. Lokad is bound by the fact to deliver its forecasts within an hour therefore some peak of consumption naturally appear making the automatic short term scaling from $2$ to $500$ VM instances in few minutes.\\
%
%The potential field of application of Cloud Computing is not restricted to IT business. The scientific computing needs to perform large scale experiments requiring large number of computing resources. Usually, scientists used super computers or clusters to run those research works. However, they were inevitably linked to massive costs of hardware and maintenance. Such approach are also difficult to setup and operate to a non expert user so as a physician or mathematician. A new model is brought through Cloud Computing to the scientist community. The storage and computing resources are dynamically provisioned as needed and the experimenting costs would be cut off for non profitable organization such as academic research institutions.\\

\section{Cloud Execution Level}

SOMETHING HERE

\subsection{MapReduce}
\label{MapReduceSection}

\subsubsection{General context}

MapReduce has two different meanings. It first describes a general framework concept explained in this section. It also designates the original implementation of this abstract framework, developed by Google (\cite{MapReduce2004}). Other implementations of this framework concept have then been realized, like Hadoop (\cite{Hadoop}), or CloudMapReduce (\cite{CloudMapReduce2010}). Unless explicitly stated otherwise, the term MapReduce will refer in this section to the framework abstraction concept.\\

MapReduce is a framework that eases distributed computation runs and deployments. It is born from the realization that \textit{"The issues of how to parallelize the computation, distribute the data, and handle failures conspire to obscure the original simple computation with large amounts of complex code to deal with these issues"}, as stated by the original MapReduce paper (\cite{MapReduce2004}). MapReduce is a scale-agnostic programming abstraction (see \cite{Helland}) originally designed to avoid programmers to solve these issues repeatedly and to provide access to a huge amount of computation resources to programmers without any experience with parallel and distributed systems.\\

MapReduce is a gathering of ideas already been discussed in the computer science literature for a long time (event for decades for some ideas) and it is not the first framework to implement some of these ideas. MapReduce is still a well-known abstraction concept, vastly due to the large success of its first implementation: Google MapReduce. This framework implementation proved to be very easy to use for distributed computing non-experts, and achieved very good overall scalability as well as a strong resilience to failures.\\

In the following paragraphs, we provide a very short overview of this framework. We refer the reader to the Hadoop definitive guide \cite{HadoopGuide} for in-depth details of the most famous open-source implementation and to \cite{MapReduceForTextMining} for multiple examples of MapReduce algorithms design.\\

\subsubsection{Programming Model}

MapReduce provides an execution environment to process heavy computations or computations on large data sets. The leading idea of MapReduce is that it is not an experimental framework but a production tool. As a direct consequence, MapReduce provides a simple and robust abstraction that can be easily handled by a wide range of distributed computing non-experts.\\

Indeed, MapReduce 

As stated, this framework provides a strong resilience to failures. Reliable commodity hardware usually have a Mean-Time Between Failures (MTBF) of 3 years. In a typical 10,000 server cluster, this implies that ten of servers are failing every day. Thus, failures in large-scale datacenters are a frequent event and a framework like MapReduce is designed to be resilient to single point of failures\footnote{With the noticeable exception of the master, but ...}. Such a resilience is hard to achieve since the computing units are dying silently, i.e. without notification of their failure. MapReduce 

MapReduce partitions the input data, maps the job into a set of sub-tasks, schedules the tasks across a large set of machines taking into account CPU/data affinity, handles the machines failures and manage the inter-machine communications.\\

The MapReduce framework takes a set of input key/value pairs and returns a set of output key/value pairs. The framework handles everything by itself except 2 functions that must be user-defined : Map and Reduce, as inspired by the \textit{map} and \textit{reduce} functions in functional languages like Lisp.\\

Map is a function which is applied independently and in parallel to each data chunk. It takes an input key/value pair and produces a set of intermediate key/value pairs.\\

Reduce is an aggregation function which combines map results. It takes as input a key of intermediate key/value pairs and a set of values for that key. It merges (or reduces) them into to form a smaller set of values associated to that key (typically, it reduces the set of values into 1 or 0). The input intermediate values are supplied through an iterator so the reducer could handle data that do not fit into its own memory. Typically, Reduce jobs are much smaller than Map jobs in term of time of computation, but not necessarily in term of I/O.\\

One or several masters monitor the load on mappers and reducers, and ensure that all mappers and reducers are still alive, relaunching the tasks for the machines that do not respond anymore.\\

Provided that the map and reduce operators are deterministic functions of their input values, MapReduce has Leslie Lamport's sequential consistency property, which means the MapReduce result is the same as would have been produced by a sequential execution of the program. This property is very useful for programmers since a MapReduce run can therefore be thought as a sequential run speeded-up. WHAT IF REDUCE NON COMMUTATIVE ?

\includegraphics[height=1.0\linewidth]{../../Pictures/mapreduce.pdf}

\subsubsection{Examples}

\cite{MapReduceForMachineLearning} has been presenting less trivious examples of machine-learning problems that can be adressed through a MapReduce framework.

\subsubsection{General issues raised}

Failures handling
Bisection bandwidth
Workers/data affinity
Straggler issues and redundant executions

\subsubsection{Recent criticisms of the MapReduce framework for Machine Learning}
\label{MapReduceCriticism}

Since its release in 2004, MapReduce has proved to be very suited to multiple large computations problems on machine clusters. In particular, MapReduce is performing very well on embarrassingly parallel algorithms on large computationally intensive batches. These algorithms require few or zero inter-machine communications and therefore allow to achieve speed-up close to the optimal.\\

In addition to the quality of the implementations of Google's and Hapache's MapReduce, a key factor in MapReduce success and adoption by the community is the ease to use it. Because MapReduce only supports tasks that can expressed in a Map/Reduce logic, it is rather straightforward to parallelize a task using this framework: either the task is intrinsically in a Map/Reduce form and the expression of the task within the framework is obvious, either the task cannot be expressed within MapReduce.\\

The ease to use and to figure how to adapt a given sequential algorithm into the framework has driven the interest of statistical and machine-learning researchers and practitioners that had no knowledge in distributed computations. As a consequence of this wide adoption, numerous research works have been made to modify and adapt algorithms originally not suited to the MapReduce framework and to suggest MapReduce implementations of the modified algorithms. CITATION NEEDED. Yet, since the MapReduce framework is rigid, numerous cases have proved to be difficult or inefficient to adapt to the framework:\\

\begin{itemize}
\item The MapReduce framework is by nature related to functional programming. The Map procedure is applied to each data chunk independently. Therefore, the MapReduce framework is not suited to algorithms where the application of the Map procedure to some data chunks pre-require the results of the same procedure to other data chunks. In other words, the MapReduce framework is not suited when the computations between the difference pieces of data are not independent and impose a specific chronology.
\item MapReduce is designed to provide a single execution of the map and of the reduce steps and does not directly provide iterative calls. It is therefore not directly suited for the numerous Machine Learning problems implying iterative processing (Expectation-Maximisation (EM), Belief Propagation, etc.). The implementation of these algorithms in a MapReduce framework require the user to engineer a solution that organize results retrieval and scheduling of the multiple iterations so each map iteration is launched after the reduce of the previous iteration is completed and so each map iteration is fed with results provided by the previous reduce iteration.
\item Both the thinking of the abstract framework and of google's practical implementation have been originally designed to address production needs and specifically robustness. As a result, the primary concern of the framework is to handle failures and guarantee the computation results. The MapReduce efficiency is therefore partly lowered by theses reliability constraints. As an example, the serialization on hard-disks of computation results turns to be rather costly in some cases.
\item MapReduce is not suited to asynchronous algorithms.
\end{itemize}

MapReduce is the reference framework for distributed computations. Its parallelization scheme is the natural choice because of its simplicity. But because of its limitations,
some voices have started to raise to question pertinency of MapReduce in the case of fine-granularity tasks in Machine Learning. As outlined in the alternative GraphLab framework introduction paper (\cite{Graphlab2010}) : \textit{"The overhead associated with the fault-tolerant, disk-centric approach is unnecessarily costly when applied to the typical cluster and multi-core settings encountered in ML research"}. This position has also been shared for example by John Langford at the NIPS workshop LCCC in december 2010.\\

The questioning of the MapReduce framework has led to richer distributed frameworks where more control and freedom are left to the discretion of the framework user, at the price of more complexity for the user. Among theses frameworks, Graphlab and Dryad are introduced in the following paragraphs.\\

\subsection{GraphLab}

Graphlab is a new framework directly designed to address parallelization challenges of machine learning algorithms. Graphlab has been initially developed based on the observation that MapReduce was insufficiently expressive for some machine learning algorithms, while low-level APIs usage lead to too much development overhead.\\

The Graphlab representation model is much more complex and rich than the MapReduce one. It consists of a directed graph representing the data and computational dependencies and of a set of update fonctions specifying the local computations to be run. By adopting a graph modeling, Graphlab supports structured data dependencies and is therefore suited to computation schemes where updates scopes overlap and some computations are dependent of other computation results. More specifically, the original Graphlab introduction paper states that it is targeted to "compactly express asynchronous iterative algorithms with sparse computational dependencies".\\

Like MapReduce, Graphlab provides a higher abstraction level than MPI and prevents users from managing by themselves synchronization, data races and deadlocks challenges. It also provides ad-hoc tasks schedulers.\\

Graphlab is therefore a machine learning parallelization framework which provides more expressiveness than MapReduce, at the cost of a more complex usage. To our knowledge, the Graphlab framework is only available at the time of this thesis writing on shared-memory multiprocessors setting (only tested on single machines up to 16 cores), making Graphlab unavailable or unefficient for distributed cloud computations.\\

\subsection{Dryad and DryadLINQ}
\label{cloud_dryad}

Dryad (\cite{Dryad2007}) is a general-purpose distributed computation framework designed to scale on any single administrative domain, be it multi-core single computers or datacenters with thousands of computing instances. Dryad is not targeted to multiple administrative domains environment and do not handle all the authentication and access grants required for most Grid Computing environments.\\

Like Graphlab, Dryad exposes a much more expressive computing framework than MapReduce in the form of a Direct Acyclic Graph (DAG). In this graph, each vertex is a computation task and edges are data and intermediate results channels. Dryad infrastructure maps the logical DAG of computation tasks onto actual physical resources without requiring the developer to get involved in this map process. In addition, Dryad handles scheduling across resources, failure recoveries, communication concurrencies, etc.\\

Whereas in MapReduce the developer was left little implementation choices than how to fit its computation task into the rigid framework of MapReduce, Dryad provides much more control to the developer through the expresiveness of the DAG. This control gain can be explained by a different approach of the frameworks: while MapReduce was focused on accessibility to provide computation power to the widest class of developers, Dryad is more concerned about expressiveness and performance.\\

In addition to a richer expressiveness of the execution flow, Dryad provides a higher control than MapReduce on communication flow and tasks scheduling. This optional higher control let programmers with better understandings of the underlying hardware performance and topology or of the computation tasks to improve the overall performance. As an example, Dryad provides different communication media between computing nodes: direct communication through TCP pipes, indirect communication through files written to hard-drive or shared-memory buffers. Whereas the default Dryad settings is to use files, the developer can choose to resort to a different medium, e.g. in the case of a single multi-core computer.\\

As a side result of this higher control and expressiveness, Dryad is reported to be more demanding on the developer's knowledge and requires several weeks to be mastered. To make Dryad more user friendly, Microsoft has also been developing an additional framework build on top of Dryad and mimicking the LINQ library interface (REF NEEDED).

\subsection{Event centric frameworks}

\subsection{HPC on cloud}

We refer to \cite{BridgingtheGap}, \cite{BenchmarkingAmazonEC2}, \cite{HighPerformanceComputingWithClouds}, \cite{TOP500},  \cite{BenchmarkingCloudServices},...

%\section{Azure Storage}
%
%\subsection{ACID framework}
%
%\subsection{Relational Database}
%
%\subsection{Emergence of the No-SQL movement}
%\label{nosqlParagraph}
%
%Recently, the relational database model features, known as ACID, have been challenged. Brewer's CAP theorem (Consistency, Availability and Partition tolerance) \cite{CAP_Theorem} states that distributed storage won't be able to propose ACID properties in an acceptable way. Abadi recently explained some defaults of CAP theorem \cite{CLAP_Presentation}. During this time, web services delivered by very large companies like Google, Amazon, Yahoo, Microsoft... have been partially abandoning relational database and ACID properties, starting the NoSQL movement (NoSQL standing for Not Only SQL). Even some SQL-related people recently expressed their interest for NoSQL database, like Ayende (NHibernate contributor) did  \cite{Slaying_Relational_Dragons}. Most NoSQL data storages are censored here \cite{NoSQL}. The properties of these new data storage systems have been described as BASE (Basically available, Soft State, Eventually consistant) \cite{BASE}.\\
%
%One of the key aspect of distributed data storage is data replication (Azure holds for each data 6 replicas : 3 replicas in a first datacenter and 3 others in a backup datacenter REFERENCE NEEDED). Since data servers are expected to fail from time to time, data are duplicated several times, so a data server can die without any data loss. This replication process (even more if replication is not stored on the same physical support) has a deep impact on consistency and performance as reported in \cite{CLAP_Presentation}. Some storage like Cassandra have been abandoning consistency guarantees for eventually consistency \cite{Eventually_Consistent}.\\
%
%Azure does not provide (at least yet) consistency or availability guarantees. Since the Azure code is not disclosed, we cannot certify any of these properties. We have been noticing that system un-availability is not only theoretical. When accessing to the storage, we can get notified an Azure internal Exception. Therefore, we have to implement retry policies so we can try again to run a read or write operation. Since storage is designed to be accessed by internal workers but also by outside (we can think for example of a picture or a video stored and devoted to be downloaded a lot of time), it makes no sense to guarantee 100\% availability since internet connection does not give such guarantee.\\
%
%To our knowledge, Azure has not been guaranteeing consistency too, even if they say they do. Yet, we have not been able to experience any consistency issue. Three points here :
%\begin{itemize}
%\item \textbf{Block commits} : Data bigger than 4MBytes must be split into blocks before being uploaded. Once every block has been uploaded into Azure, the "transaction" is commited, and data is finally wrote on Azure side. This 2-phase upload (partial uploads, then integral commit) is TO BE FINISHED
%\item \textbf{Etag} : As reported in the BlobStorage and TableStorage technical papers \cite{Azure_Storage_Resources}, Azure provides a kind of timestamp, called etag. When updating some data, we first download the data, update it, and upload it with the etag provided. When replacing the data, Azure checks that etag of data being replaced matches the etag provided by the upload. If it does not, the overwrite is rejected, an exception is thrown, so the user can retry. This mechanism helps to deal with multiple workers trying to modify a data at the same time, but is very probably also used inside Azure to ensure consistency between the different replica of a same data.
%\item \textbf{Very few overwrites} Most applications do not need to overwrite data. Some typical usage could be for example a big video upload once, and hundreds or thousands of read request on this data (Facebook usecase). Clusterings we run on Azure required very few overwrite (mostly for counters), as it will be shown.
%\end{itemize}
%
%\subsection{Azure BlobStorage}
%
%Azure team has been communicating few on their underlying system architecture that handles the storage and how they achieve scalability.
%Most of information presented here relies on the Windows Azure Storage Team blog (http://blogs.msdn.com/b/windowsazurestorage/) and are not guaranteed.\\
%
%Azure Blob Storage enables applications to store large objects, up to
%50 GB each.  It supports a massively scalable blob system, where hot blobs will
%be served from many servers to scale read access. Blobs are composed of a
%string (that is used as a key to store the value), a value composed of a
%binary object, and a timestamp (etag) (that indicates the last write on this
%blob). In addition of Get and Put methods, blobs whose key shares a given
%prefix can be listed. Optimistic non locking atomic read-modify-write
%operations can be implemented using a timestamp matching condition: a write
%succeed if and only if the timestamp of the storage matches the one provided
%by the write operation.
%
%\subsubsection{Blob Storage Data Model}
%
%All the objects stored in the Blobstorage are stored through a key/value pair system. Each object is stored and accessed through the key, which is a string composed of 2 parts : container name and blob name. The container name is scoped by the account, which means you can freely name a container no matter how other customers build their container names.
%To build a container name string, you need to comply to several rules edicted by Azure :
%
%\begin{itemize}
%\item Container names must start with a letter or number, and can contain only letters, numbers, and the dash (-) character.
%\item Every dash (-) character must be immediately preceded and followed by a letter or number. There should be no consecutive dash characters
%\item All letters in a container name must be lowercase (This rule is often broken by mistake while using Camel Case)
%\item Container names must be from 3 through 63 characters long.
%\end{itemize}
%
%BlobName is a second string that is unique for any blob stored inside the container. ContainerName and BlobName are separated by the character"/".\\
%Here is some example of a valid key : \\
%
%some-container/blobname1\\
%
%Caution : using an incorrect blob name will cause a very unhelpful error message : IndexOutOfRange Exception.
%
%\subsubsection{Type of blobs}
%
%//To be rewritten :
%%
%%There are two types of blobs supported:
%%
%%    Block Blobs – targeted at streaming workloads.
%%        Each blob consists of a sequence/list of blocks.
%%        Max block blob size is 200GB
%%        Commit-based Update Semantics – Modifying a block blob is a two-phase update process.   It first consists of uploading blocks as uncommitted blocks for a blob.  Then after they are all uploaded, the blocks to add/change/remove are committed via a PutBlockList to create the updated blob.   Therefore, updating a block blob is a two-phase update process where you upload all changes, and then commit them atomically.
%%        Range reads can be from any byte offset in the blob.
%%    Page Blobs – targeted at random write workloads.
%%        Each blob consists of an array/index of pages.
%%        Max page blob size is 1TB
%%        Immediate Update Semantics – As soon as a write request for a sequential set of pages succeeds in the blob service, the write has committed, and success is returned back to the client.  The update is immediate, so there is no commit step as there is for block blobs.
%%        Range reads can be done from any byte offset in the blob.
%
%\subsubsection{Partition}
%
%To load balance the objects access, Azure is using a partitioning system. While a same partition key can be shared among multiple entities in TableStorage, the partition key is unique for each blob in BlobStorage and defined by ContainerName + BlobName. Azure is automatically load balancing the blobs across their servers based upon these partitions.\\
%
%For the TableStorage, all objects with the same partition key value are grouped into the same partition and are accessed from the same partition server. Thus, atomic operations can be run across objects in the same partition (because they share the same physical machine). Yet, since all objects stored in a partition are managed by the same partition server, scalability in this partition will be much harder to achieve.\\
%
%For the BlobStorage, since each object is stored in a different partition key, access to different blobs (even with the same ContainerName) can be load balanced across as many servers as needed in order to scale access.
%
%\subsubsection{Architecture and scalability}
%
%The storage access architecture is composed of the 3 following layers:
%
%\begin{enumerate}
%\item \textbf{The Front-End layer} is composed of multiple servers processing the incoming requests and routing them to the layer below : the partition layer. Each front-end server holds in cache the mapping between partitions and partition servers so it can routes request to the correct partition server.
%\item \textbf{The Partition layer} is composed of multiple servers managing the different partitions. One partition server can manage one or multiple partitions, but a given partition is managed by only one partition server. For the BlobStorage, this is obvious since the partition key is down to the blob name. A partition master monitors each partition server's load and determines load balancing.
%\item \textbf{The Distributed File System (DFS) layer} is responsible of persisting data on disk and replicating the data across multiple servers to avoid data loss. All partition servers can access any of the DFS servers.
%\end{enumerate}
%
%
%%Now that we have given a high level description of storage accounts, storage abstractions and how they are grouped into partitions, we want to talk about the scalability targets for storage accounts, objects and their partitions.
%%
%%The following are the scalability targets for a single storage account:
%%
%%    Capacity – Up to 100 TBs
%%    Transactions – Up to 5,000 entities/messages/blobs per second
%%    Bandwidth – Up to 3 gigabits per second
%%
%%The 100TB is a strict limit for a storage account, whereas the transactions and bandwidth are the current targets we’ve built the system to for a single storage account. Note, the actual transaction and bandwidth achieved by your storage account will very much depend upon the size of objects, access patterns, and the type of workload your application exhibits. To go above these targets, a service should be built to use multiple storage accounts, and partition the blob containers, tables and queues and objects across those storage accounts.  By default, a subscription gets 5 storage accounts, and you can contact customer support to get more storage accounts if you need to store more than that (e.g., petabytes) of data.
%%
%%It is expected that a hosted service needs up to as many storage accounts to meet its performance targets given the above, which is typically a handful of storage accounts to up to 10s of storage accounts to store PBs of data.   The point here is that a hosted service should not plan on creating a separate storage account for each of its customers.  Instead, the hosted service should either represent a customer within a storage account (e.g., each customer could have its own Blob Container), or map/hash the customer’s data across the hosted service’s storage accounts.
%%
%%Within a storage account, all of the objects are grouped into partitions as described above. Therefore, it is important to understand the performance targets of a single partition for our storage abstractions, which are:
%%
%%    Single Queue – all of the messages in a queue are accessed via a single queue partition. A single queue is targeted to be able to process:
%%        Up to 500 messages per second
%%    Single Table Partition – a table partition are all of the entities in a table with the same partition key value, and most tables have many partitions. The throughput target for a single partition is:
%%        Up to 500 entities per second
%%        Note, this is for a single partition, and not a single table. Therefore, a table with good partitioning, can process up to a few thousand requests per second (up to the storage account target).
%%    Single Blob – the partition key for blobs is the “container name + blob name”, so we can partition blobs down to a single blob to spread out blob access across our servers. The target throughput of a single blob is:
%%        Up to 60 MBytes/sec
%%
%%The above throughputs are the high end targets for the current system. What can be achieved by your application very much depends upon the size of the objects being accessed, the operations (workload) and the access patterns. We encourage all services to test the performance at the partition level for their workload.
%%
%%When your application reaches the limit to what a partition can handle for your workload, it will start to get back “503 server busy” responses. When this occurs, the application should use exponential backoff for retries. The exponential backoff allows the load on the partition to decrease, and to ease out spikes in traffic to the partition. If this is a regular occurrence, then the application should try to improve its data partitioning and throughput as follows for the different storage abstractions:
%%
%%    Blobs – consider using Windows Azure Content Delivery Network for delivering anonymous access to hot blobs to achieve higher throughput for commonly accessed blobs.
%%    Tables – use a more finely grained PartitionKey for the table in order to allow us to automatically spread the table partitions across more servers.
%%    Queues – batch together multiple work items into a single message to improve queue throughput or use multiple queues.
%%
%%See the next set of upcoming posts on best practices for scaling the performance of Blobs, Tables and Queues.
%%
%%Finally, one question we get is what the expected latency is for accessing small objects:
%%
%%    The latency is typically around 100ms when accessing small objects (less than 10s of KB) from Windows Azure hosted services that are located in the same location (sub-region) as the storage account.
%%    Once in awhile the latency can increase to a few seconds during heavy spikes while automatic load balancing kicks in (see the upcoming post on Windows Azure Storage Architecture and Availability).
%
%\subsection{Azure QueueStorage}
%
%Azure Queues provide a reliable asynchronous message delivery mechanism through distributed
%queues to connect different components of a cloud application. Queues are designed to store a large amount of small messages (with
%maximal individual size of 8 KB\footnote{\cite{AzureStorageResources} (6144 bytes more precisely, see \cite{AzureQueuesCannotBeLarger}))}). Using queues to communicate helps building
%loosely coupled components and mitigates the impact of individual component
%failure.\\
%
%Azure queues are not supposed to respect FIFO logic (First In First Out) as standard queue. On small scale, the Azure queue will behave like a FIFO queue, but if it is more sought, it will adopt a different logic so it can better scale. Therefore, the FIFO assumption cannot be assumed when designing a cloud app.\\
%
%Messages stored in a queue are guaranteed to be returned at least once, but possibly several times:
%this requires one to design \textbf{idempotent} jobs.  When a message is unqueued by a
%worker, this message is not deleted but it becomes invisible for other
%workers. If a worker fails to complete the corresponding job (because it
%throws some exception or because the worker dies), the message becomes
%available after a certain period of time. Through this process, one can make
%sure no job is lost because of e.g., a hardware failure.
%
%\subsection{Azure TableStorage}
%
%Azure Table Storage provides scalable structured storage. Structured storage is provided in the form of Tables, which contain a set of Entities, which contains a set of named Properties. No schema is stored by Windows Azure Table, so all of the properties are stored as $<name, typed ~value>$ pairs. This means that two entities in the same table can have very different properties. Combined size of all data in an entity cannot exceed 1MB. Entities are stored inside the table using a two part key composed of a PartitionKey (primary key) and a RawKey (secondary key). An application can manipulate the entities and query over any of the properties stored in a Table.\\
%
%We didn't use TableStorage in this thesis.
%
%\subsection{Serialization}
%
%\subsection{BlobStorage or TableStorage ?}
%
%BlobStorage and TableStorage are both designed to store data. When should we use one or the other ? Here are some part of the answer :
%\begin{itemize}
%\item One part of the answer is the data locality management and atomicity. Every data object may they be blobs or table entities are stored in terms of their partition key. This partition key determines how Azure must load balance and partition the objects across Azure servers. Partition Key for TableStorage is the tableName + partitionkey (primary key), whereas PartitionKey for blobs are determined by containerName + blobName. Since the partition key is down to the blob name for blobStorage, access to different blobs can be load balanced  across as many servers as needed to scale. But no atomic transactions is authorised for multiple blobs. As explained below in design subsection \ref{Redesign of the algorithms}, jobs will need to be idempotent : no atomic transaction for blobStorage is another difficulty to guarantee the idempotency. Since partition key is up to the primary key in TableStorage, we can run atomic transactions up to 100 entities who share the same table + primarykey (Azure guarantees that all the entities will be changed the specified way, or no one will).
%\item When dealing with really large objects, BlobStorage is easier to manipulate, since each blob can be arbitrarily large (up to 50 GB per blob). We do not need to break an object into a lot of entities, making sure each time each "part entity" respect the 1MB constraint. BUT WE HAVE TO DEAL WITH BLOCKS : SAY LOKAD.CLOUD HANDLES IT
%\item When manipulating objects than can be stored both in TableStorage and BlobStorage, BlobStorage is reported by \cite{Early_Observations_On_Azure} to be more effective in insert operations on blobs larger than 4KBytes, which is below the average size of blobs we are going to use.
%\item Price is another factor that might affect our choice. Using Azure storage, three things are charged : size of our storage, I/O communications, requests to the storage. A single query on TableStorage can load up to 100 elements at a time and costs as much as a single query on blobStorage, see \cite{Azure_Pricing}. If we have to store a lot of small objects on different places, using TableStorage will thus allow to run request on objects by packet of 100, so it would divide up to a factor 100 the bill of storage requests in comparison of running requests one by one on the BlobStorage. If we do not need to store the objects on different places, we could store them in one bigger blob, bounding the request to one. Yet, the more times we have to modify a small part of the storage, the less storing all the data in a single blob is the right thing, since we are going to download and upload the big blob each time a small part of the data need to be updated, making I/O costs predominant comparing to storage requests. So, using TableStorage or BlobStorage on the price criterion will depend on the size of your data, your application, the way the application is designed, your need to update small part of data or entire data each time, ...
%\end{itemize}
%
%To run our experiments, we chose to use the BlobStorage. Atomic updates on multiple object at once will not be a key point in our code, except for counters for which concurrency will be handled by etag mechanism. Price will be estimated but is not a criterion here. Besides, BlobStorage is easier to manipulate and due to its complexity, TableStorage suffered little bugs that affect performances when we ran our experiments (in June 2010).\\
%
%\section{Azure Framework}
%
%\subsection{Global description}
%Windows Azure Platform is Microsoft's cloud computing solution, in the form of
%Platform as a Service (PaaS). The underlying cloud operating system (Windows
%Azure) provides services hosting and scalability. It is composed of a storage
%system (that includes Blob Storage and Queue Storage) and of a
%processing system (that includes \emph{web roles} and \emph{worker
%  roles}). We will mainly discuss the storage elements we used in our experiments,
%  namely BlobStorage and QueueStorage.
%
%The architecture of an Azure hosted application is based on two components:
%\textbf{web roles} and \textbf{worker roles}. Web roles are designed for web
%application programming and were only used in our prototypes to monitor tasks executions
%and worker loads. Worker roles
%are designed to run general background processing. Each worker role typically
%gathers several \emph{cloud services} and uses many \emph{workers} (Azure's
%processing units) to execute them. Our prototypes always use only one worker role,
%several services and tens of PU. The computing power is provided by the
%workers.
%
%\subsection{Idempotency}
%
%\subsection{Affinity between workers and storage}
%
%Presently, there are no mechanisms in Azure that provides the same affinity between the workers and the storage than MapReduce does.
%This means that we cannot push some jobs where the data is stored since we can't know on which machine a data piece is stored. Each worker processing data will need to first download the data it needs to process, on the contrary of Google's Map Reduce, where the jobs are scheduled and assigned on each worker by the framework in such a way that workers downloading data will be minimized.
%
%This is intended by Microsoft (CITATION NEEDED) to simplify development of cloud apps : one does not need to design its application to take data location into account, Azure providing a satisfactory scale-up and bandwidth. This choice is about simplicity over performance.
%
%\subsection{Workers Communication}
%
%Azure does not offer currently any standard API for distributed computation, neither a low
%level one such as MPI \cite{MPI1996}, nor a more high level one such as Map
%Reduce \cite{MapReduce2004} or Dryad \cite{Dryad2007}. Map reduce could be
%implemented using Azure components (following the strategy of
%\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{Graphlab2010},
%those high level API might be inappropriate for iterative machine learning
%algorithms such as the k-means or learning vector quantization because of MapReduce overhead on synchronisation and waiting process.
%
%We therefore need to explicitly design our communication process between Azure workers, process that can realized by 2 means :
%
%\begin{enumerate}
%\item Workers can communicate each other through direct IP communications, that is to say re-implement a peer-to-peer network inside Azure cloud. TO DEVELOP
%\item Azure storage system can be used to implement synchronization and
%communication between workers : each worker is pinging (or "listening") a specific queue, and it gets some message from this queue holding
% some blobname in BlobStorage where it can retrieved the data it needs.
%\end{enumerate}
%
%While the first solution comes with higher performance in terms of bandwidth, as reported in AzureScope or ... (CITATION NEEDED), it implies a very tough work which would be like implementing MPI on Azure, which was outside the scope of this thesis. Moreover, the "storage approach" for communication is much more in agreement with Azure's abstractions and design thinking (CITATION NEEDED).
%Thus, we will rely therefore directly on Azure queues and blob storage to build all our communication systems.
%
%\subsection{Sharded Counters}
%
%\subsection{Lokad-Cloud}
%
%\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}}, an
%open-source framework that adds a small abstraction layer to ease Azure
%workers startup and life cycle management, and storage access.
%
%\section{Azure Performance}
%
%\subsection{Performance Tradeoffs in Azure}
%
%\textbf{Read/Write tradeoff} : As reported in \cite{BenchmarkingCloudServices}, distributed storage must choose a tradeoff between read throughput and write throughput. Taking the example of a blob in BlobStorage, a blob cannot be at the same time highly available in read operations and write operations (having a highly available blob for read operations implies the blob to be duplicated, so write operations must change several replicas, and write operations take more time). Azure let the traffic arbitrate this tradeoff : the more request to a blob per time unit, the more the blob is duplicated, the more it becomes available, and the more time it takes to run write operations.\\
%
%\textbf{Responsivity/Efficiency} \cite{BenchmarkingCloudServices} is also reporting some tradeoff between services responsivity and efficiency. Efficiency, as defined as the ratio of work completed per hour per worker is a decreasing function of workers and an increasing function of total workload. Therefore, to achieve the best possible efficiency, we should make sure every worker is overworked. Yet, the more overworked workers are, the least responsive they can be on an additional task. Since the first goal of our algorithm is responsivity (we want to minimize amount of time spent processing the algorithm), we will make sure to provide a pool of workers slightly greater than the number of jobs we can parallelize, so we make sure all jobs that can be run parallelingly, are (having P jobs run by P-1 workers would lead approximately to twice the time P jobs running by P or P+1 workers). \\
%
%\textbf{Synchronous Replication/Replication Latency tradeoff} : TO BE DIGGED. RELATED TO CONSISTENCY
%
%\subsection{Benchmarks}
%The Azurescope website \cite{AzureScope} is reporting performance benchmark run from February/March 2010 to presently (June 2010) on Azure hardware.
%Some paper have also been released very recently on same measures \cite{Early_Observations_On_Azure}.
%Numbers reported in \cite{Early_Observations_On_Azure} seem to be more accurate than \cite{AzureScope}, and the methodology is better explained, so we will use these numbers as reference.
%We will use the numbers observed to understand if bandwidth is an issue on Azure.
%
%\begin{itemize}
%\item BlobStorage Read Bandwidth : 13 MBytes/sec for 1 worker, BlobStorage aggregated read bandwidth bounded to 393 MBytes/sec when using more workers (each worker is using several threads to load data from storage). This quantity describes the bandwidth for several workers to download the same blob from the storage.
%\item BlobStorage Write Bandwidth : 5 MBytes/sec for 1 worker, BlobStorage aggregated write bandwidth bounded to 124 MBytes/sec when using more workers (each worker is using several threads to upload data). This quantity describes the bandwidth for one or several workers to load data from different blobs but within the same container.
%\item Communication throughput between 2 workers : 55 MBytes/sec on large messages on average (which is a very impressive bandwidth).
%\item CPU power depends on type of VM allocated : from 1 core (on small VM) to 8 cores on extra-large VM
%\item Queue throughput is presently 500 messages per second, but might be very increased soon.
%\end{itemize}
%
%Note : Since concurrent write on a blob is very costly, we will not write twice at a same place, unless when using counters.\\
%
%\section{Prices}


\section{Some Cloud Computing offers}
\label{cloudOffers}

\subsection{Google App Engine}

\subsection{Amazon}

has MapReduce, but also lower level abstractions.

\subsection{Azure}

Presently, Azure is designed to provide PaaS. As explained in Azure FAQs,
Microsoft is planning to add Virtual Machines (VM) functionality to enrich the
different uses of Azure. Therefore, Azure might also become Infrastructure as a
Service (IaaS). Yet, since no communication or services have been released yet,
we will assume Azure is only PaaS oriented.

