\section{Introduction}

The coming shift to cloud computing is also impacting major software companies previously deeply oriented on desktop software or OS development. Among these firms,   

The Microsoft cloud computing offer is a rich cloud computing solutions. Be

Microsoft is offering multiple cloud computing solutions to its customers. The Office suite and Microsoft Windows have been the An increasing part of the software applications presently sold by Microsoft has been migrated to the cloud and is offered alternatively as SaaS. Microsoft Office $365$ for example, is a commercial software services offering a set of products from Microsoft. It is publicly available since 28 June 2011 and includes the Microsoft Office suite, Exchange Server, SharePoint Server, etc.\\

In addition to its SaaS offers, Microsoft has designed a cloud computing solution targeted for independent software vendors. This solution, referred as Windows Azure Platform, is a web application hosting solution in the form of Platform as a Service (PaaS) (see \ref{xaas}). Presently, it is marketed for software developers to store data and to deploy and run robust and scalable applications. Built upon geographically distributed datacenters all over the world, Windows Azure Platform provides low-latency responsiveness for web applications and a solid framework for CPU-intensive processing tasks.\\

This chapter is organized as follows. The first following section is an overview of Windows Azure. We give there a definition of the main components of Windows Azure and explain how it is positioned compared to other cloud computing offers. The section \ref{azure_storage} defines the different parts of Azure persistent storage system and the architecture proposed to build a scalable storage system. The section \ref{azure_framework} presents several aspect of cloud applications, especially the design constraints related to the communications between workers and failures handling. The section \ref{azure_performance} is dedicated to raw performances of the storage and the computing instances. We report for example bandwidth and Flops we manage to get. These performances have a crucial impact on algorithms since they determine their design and potential performances, as detailed in chapter \ref{chap:KMeans}. Last section \ref{azure_prices} introduce some key numbers to estimate the total price of running algorithms on Azure.

\section{Overview of Azure}

Windows Azure is composed of three main components: Windows Azure platform, Windows Azure  AppFabric, and SQL Azure.\\

\subsection{Windows Azure Platform}

Windows Azure platform is composed of a persistent storage (see section \ref{azure_storage}) and of a cloud operating system called Azure Compute that provides computing environment for applications. Azure Compute is based on three components: \textbf{web roles}, \textbf{worker roles} and \textbf{Virtual Machines (VM) roles}. Developers are not expected to handle the individual machines by themselves: they implement the logic for each role they need, and an Azure-side engine called Azure Fabric will handle the deployment of as many virtual machines (VM) as requested by the developer to run each specific role.\\

Web roles are designed for web application programming. Web Roles allow public computers to connect to your application over standard HTTP and HTTPS ports. Virtual machines running a given web role are pre-configured with IIS7 (Internet Information Services) and specifically designed to run Microsoft web-programming technologies as ASP.NET or Windows Communication Foundation (WCF), but they also support native code such as PHP or Java to build web applications.\\

Worker roles are designed to run general background processing. The general background processing can be dependant of a web role (handling the processing required by the web role) or independent. One of the difference between web and worker roles is that worker roles don't come with a pre-installed IIS. Worker roles can be defined using .NET framework.\\

For web and worker roles, developers will only need to upload their website code (web role) or .NET code (worker role) to Azure through a package from the Azure administration portal, or directly from Visual Studio, and Azure automation will deploy the package to the Azure VM instances, completely abstracting the actual VM from the user.\\

VM roles are designed to provide developers a much wider scope of possibilities and especially to control the operating system image. VM roles should not be used unless Worker and Web roles do not fit the developer purpose, as it is the case for example when one have long and complicated installations in the operating system or a setup procedure that cannot be automated. In VM roles, the developer will upload his own virtual hard drive (VHD) that holds a custom operating system (more specifically a Windows Server 2008 R2 image) that will be run on the different VM running the VM role.
%The VM Role shares the same programming model and the same limitations of the Web and Worker roles : programs running on VM roles must be stateless since single instance failures should not %affect the role behaviour.

\subsection{Windows AppFabric}
\label{azure_fabric}

Windows Azure AppFabric (not to be confused with Azure Fabric described above) is a set of optional technologies that helps building and managing cloud applications. One of the main component of Windows Azure AppFabric is the Service Bus module, that provides a messaging system in the form of senders/listeners. includes : Service Bus, Caching, Inte   ....\\

\subsection{Windows SQL Azure}

SQL Azure ...\\

\section{Windows Azure Storage}
\label{azure_storage}

Windows Azure Storage (WAS) is the storage component of the Windows Azure Cloud Platform. It is a public cloud service available since November 2008 and presently (in January 2012) holds 70PB of data. It is used as an independent storage service but also as the persistency storage for the applications run on Windows Azure Cloud Platform. According to \cite{AzureStorage}, the WAS is used internally by Microsoft for applications such as social networking search, serving video music and game content but also outside Microsoft by thousands of customers.\\

The WAS has been designed to be highly scalable, so a single piece of data can be simultaneously accessed by multiple computing instances and so a single application can persist terabytes of data. For example, the ingestion engine of Bing used to gather and index all the Facebook and Twitter content is reported to store around 350 TeraBytes of data in Azure.\\

The WAS is composed of different storage components, designed to target different purposes. The following paragraphs describe this components.

\subsection{WAS components}

WAS is composed of four different elements: Windows Azure BlobStorage (or BlobStorage for short), Windows Azure TableStorage (or TableStorage for short), Windows Azure QueueStorage (or QueueStorage for short) and Windows Azure SQL.\\
\begin{itemize}
\item The BlobStorage is a simple scalable Key-Value pairs storage system. It is designed to store serialized data, may they be files or Object-Oriented Languages object instances.
\item The TableStorage is also an alternative Key-Value pairs storage system. It provides atomicity guarantees in a constrained scope, as detailed below.
\item The QueueStorage is a scalable queues system designed to handle very small objects. It is used as a delivery message mechanism across computing instances.
\item Azure SQL is a relational cloud database built on SQL Server technologies. It is designed to be an on demand RDBMS and requires no setup, installation and management.\\
\end{itemize}

A common usage pattern for the different storage elements is as follows: I/O data transfert is hold through the BlobStorage, overall workflow instructions and job messages are hold by the QueueStorage and describes how the blobs need to be processed, and intermediate results or service state are kept in the TableStorage or the BlobStorage.\\

\subsubsection{Azure BlobStorage}

Azure BlobStorage is a large-scale key-value pair storage system. It is designed to be the main storage component of most azure applications, but is also an efficient device to provide Input/Output with the application customers.\\

The objects stored in the BlobStorage are files or class instances serialized. They are stored in Binary Large OBjects (referred as Blobs). The size of a blob is bounded to 50GBytes. The blobs are stored on multiple nodes, and a complex load-balance system (partially described in section \ref{internal_architecture}) ensures the system is scalable and strongly consistent. The strong consistency is an important aspect of the BlobStorage since it has an impact on overall latency and scalability of the BlobStorage, as explained in \ref{Cap_theorem_section}. This property is a key characteristic of the WAS, as many other No-SQL storage do not provide this guarantee (NAME 2 OF THEM). The strong consistency implies that each blob is immediately accessible once it has been added or modified, and that any subsequent read from any machine will immediately see the changes made by the previous write operations.\\

Each object stored in the BlobStorage is accessed through its key. The key is a string composed of 2 parts: a container name and a blob name. The container name is scoped by the account, so two different accounts can have same container names. The container names must be compliant with the following Azure naming rules:

\begin{itemize}
\item Container names must be from 3 to 63 characters long.
\item Container names must start with a letter or number, and can contain only letters, numbers, and the dash (-) character.
\item Every dash (-) character must be immediately preceded and followed by a letter or number. There should be no consecutive dash characters
\item All letters in a container name must be lowercase (This rule is often broken by mistake while using Camel Case)
\end{itemize}

Blob updates mechanism can be achieved through an optimistic nonblocking atomic read-modify-write mechanism. A nonblocking update algorithm consists in updates executed speculatively, assuming no concurrent machine is updating the data at the same time. The nonblocking update algorithms do not imply synchronization or locking mechanisms when an update is executed. However, a check is performed at the end of the update to make sure that no conflicts have occured. In the case of a conflict, the update is aborted and need to be redone. Nonblocking update algorithms are often called optimistic because they bet on the fact conflicts are statistically unlikely to happen. If concurrent writes happen, then the update mechanism is likely to take more time than a simple locking update process. Yet on average, the optimistic update algorithms are much more efficient than other locking algorithms.\\

This optimistic update mechanism is allowed in Azure through timestamp referred as etag, following Azure terminology. This timestamp indicates the exact date of the last successful write operation applied to this blob inside the BlobStorage. The Azure blob update is performed as follows: the blob to be updated is downloaded by the computer in charge of the blob update in addition to the corresponding etag of the blob. The actual value of the blob is locally updated by the computer, then pushed back to Azure BlobStorage with the previous etag. If the returned etag matches the present etag of the blob version stored in the WAS, then the blob is actually updated. If the two etags do not match, then a distinct machine has been concurrently updating the same blob, and the updating process is relaunched.\\

The Azure BlobStorage API provides methods to retrieve and push blobs, but also to list all the blobs whose key share a given prefix.\\

Another key feature of data storage is the way multiple data can be accessed and modified simultaneously. As detailed in section \ref{acide}, a single transaction may indeed involve multiple pieces of data at once. In the context of RDBMS, transactions are guaranteed to be atomic, so each piece of data concerned by a transaction is updated accordingly or none is (if an update fails). The Azure BlobStorage does not give any primitive to atomically modify multiple blobs. Because of the internal design of the WAS, the different blobs are stored on multiple machines distributed on separate Storage Stamps (we refer the reader to more in depth details in section \ref{internal_architecture}). This physical distance between the storage nodes of different blobs involved in a transaction preclude low latency atomic multi-blob transactions.\\

Blobs can be stored in two different blobs formats : block blobs and page blobs.\\

\begin{itemize}
\item Block blobs are optimized for streaming. Each block blob consists of a sequence of blocks, each block being identified by a block ID. The size of a block is bounded to 4MB.
Writing to a block blob is a two-step process relying on a commit-based update semantics. Firstly, a block is uploaded to Windows Azure using the \textit{PutBlock} method of BlobStorage API, but it does not become part of the blob until it is commited. The second step consists in a commit-phase : when all the blocks have been uploaded, the user calls the \textit{PutBlockList} that specifies the list of block IDs that make up the blob. The list of blocks that form the new blob version is commited atomically.\\

\item Page blobs are targeted for random writes. Each blob consists of an array of pages, each page being a range of data that is identified by its offset from the start of the blob. Pages blobs update is an immediate update semantics: as soon as a write request for a page or a sequential set of pages succeeds on the storage side, the write is committed, and client is notified back of write success.\\
\end{itemize}

Remarque sur la sérialisation

\subsubsection{Azure TableStorage}
\label{azure_TableStorage}

Azure TableStorage is another key-value pair storage component of the WAS. It is targeted to provide a more adapted storage to structured data used in Azure applications. The structured storage is provided in the form of Tables. A single application may create many tables. Each table contains a set of entities. To each entity is associated two keys in the form of strings: the first key is called the PartitionKey and can be shared with other entities of the same table, while the second key is specific to each entity and is called the RowKey. A given entity is therefore uniquely referenced as the combination of its corresponding PartitionKey and RawKey. Each entity holds a set of $<name, typed ~value>$ pairs named Properties.\\

In a given table, each entity is analogous to a "row" of a RDBMS and is a traditional representation of a basic data item. This item is composed of several values, filled in the entity's properties. Each table has a flexible schema, so two different entities of a given table can have very different properties. Yet, a common design pattern consists in adopting a fixed schema within a given table so all the entities have the exact same set of properties.\\

Because of the flexible schema, even in the case of the previous design pattern where the schema is \textit{de facto} fixed, the TableStorage does not provide any way to represent relationships between the entities' properties on the contrary to other structured storage as RDBMS. Another consequence of this flexible schema is the lack of "secondary" indices in a given table: the only way to enumerate entities in a table rely on the entity key, namely the RowKey. Therefore, selecting entities according to a criterion based on a given property will lead to execution time linear in the number of entities in the table (on the contrary to RDBMS, where secondary indices may lead to requests time proportional to the logarithm of the number of entities).\\

An additional feature of the TableStorage is to allow reads, creations or updates of multiple entities in a single command. These commands, referred to entity group transactions, support executing up to 100 entities in a single batch, provided the fact the entities are in the same table and have the same partition key. In addition to atomicity guarantees described in section \ref{partition_name}, the entity group transactions are of prime interest since they provide a way to significantly lower (up to a factor 100) the price of storage I/O in the case of small objects being stored.

\subsubsection{Azure QueueStorage}

Azure Queues provide a reliable asynchronous message delivery mechanism through distributed
queues to connect different components of a cloud application. Queues are designed to store a large amount of small messages (with
maximal individual size of 8 KB\footnote{\cite{AzureStorageResources} (6144 bytes more precisely, see \cite{AzureQueuesCannotBeLarger}))}). Using queues to communicate helps building
loosely coupled components and mitigates the impact of individual component
failure.\\

Azure queues are not supposed to respect FIFO logic (First In First Out) as standard queue. On small scale, the Azure queue will behave like a FIFO queue, but if it is more sought, it will adopt a different logic so it can better scale. Therefore, the FIFO assumption cannot be assumed when designing a cloud app.\\

Messages stored in a queue are guaranteed to be returned at least once, but possibly several times:
this requires one to design \textbf{idempotent} jobs.  When a message is unqueued by a
worker, this message is not deleted but it becomes invisible for other
workers. If a worker fails to complete the corresponding job (because it
throws some exception or because the worker dies), the message becomes
available after a certain period of time. Through this process, one can make
sure no job is lost because of e.g., a hardware failure.

\subsubsection{Partition, Load Balancing and Consistency}
\label{partition_name}

To load balance the objects access, the WAS is using a partitioning system. This partitioning system is build upon a partition key in the form of a string. For the BlobStorage, the partition key of a blob is the concatenation of its ContainerName and of its BlobName strings. In the case of the BlobStorage, each partition is therefore only holding a single blob, and it is impossible to gather several blobs into a single partition. For the TableStorage, the partition key is the PartitionKey string already introduced in section \ref{azure_TableStorage}. In this case, multiple entities can be gathered in a single partition provided the fact they are in the same table and share the same partition key.\\

The load balancing system is separated in two load balancing levels. The first load balancing level is achieved by Azure itself and provides an automatic inter-partitions load balancing. Nodes dedicated to address data requests are called partition servers. The inter-partitions load balancing system maps partition servers and partitions. A single partition server can handle multiple partitions. But for a given partition, a single partition server is in charged of addressing all the requests to this partition, while the effective data of the partition can be stored on multiple nodes (we refer the reader to the section \ref{internal_architecture} for more in depth explanations). The WAS monitors the usage pattern of the request on the partitions and can dynamically and automatically reconfigure the map between partition servers and partitions.\\

Since all the requests related to a single partition are addressed by a single partition server, the partition is therefore the smallest unit of data controlled by Azure for load balancing. The second load balancing system is handled by the user. This second system refers to the user's ability to adjust the partition granularity so a given partition handles more or less data. More specifically, the WAS consumer needs to tune the partition granularity in respect to a consistency/scalability tradeoff. The bigger a partition is, the more transactions on multiple data pieces can be done atomically. Yet, since all the requests to this partition are addressed by a single partition server, the bigger a partition is the more busy the partition server will be.\\

For the BlobStorage, since each object is stored in a different partition key, access to different blobs (even with the same ContainerName) can be load balanced across as many servers as needed in order to scale access.

\subsection{Elements of internal architecture}
\label{internal_architecture}

The internal implementation of WAS is rather complex, and leverages a multi-layer design. This design let the storage be at the same time strongly consistent, highly available and partition tolerant, which is claimed by the CAP theorem to be difficult to achieve at the same time. This section provides a short insight of the underlying infrastructure and shows how the previous guarantees are achieved despite of the CAP theorem. A more in depth presentation of WAS has been disclosed by the Azure team itself and can be found in \cite{AzureStorage}.

\subsubsection{Storage stamps}

Following the terminology of \cite{AzureStorage}, the computing instances in a datacenter are gathered among storage stamps. Each Storage Stamp is a cluster of about 10-20 racks, each rack holding about 18 disk-heavy storage nodes. While the first generation of storage stamps hold about 2PB of raw data in each stamp, the next generation hold up to 30PB of raw storage in each stamp. To lower the cost of cloud storage, Azure team keeps the storage stamps highly utilized. To keep sufficient throughput and high availability even in the presence of a rack failure, each stamp is not used above 70\% of its capacity. When a storage stamp is filled up to this bound, the location service migrates some accounts to different stamps, to keep the storage stamp on a capacity usage ratio of 70\%.\\

Each rack within a storage stamp is supported on isolated hardware. Each rack is supplied in bandwidth and energy with independent and redundant networking and power, to be a separate fault domain.\\

The datacenters hosting WAS services are spread in several regions of North America, Europe and Asia. In each location, a datacenter holds multiple storage stamps. When creating a new cloud application Azure, customer are asked to choose a location affinity among this three continents. While the cloud consumer cannot choose the exact datacenter location being primarily used by its application, this location affinity choice let the user lower its application latency and improve its responsiveness.\\

\subsubsection{Front End Layer}

The Front-End layer is composed of multiple servers processing the incoming requests and routing them to the partition layer. Each front-end server holds in cache the mapping between partition names (as defined in section \ref{partition_name}) and partition servers of the partition layer. When receiving a customer request, the Front-End layer server authenticates and authorizes the request, then routes it to the partition server that primarily manages the partition name related to the request. The most frequently accessed data are cached inside the Front-End servers and are directly returned to corresponding requests to partially unload the partition layer servers.

\subsubsection{Partition Layer}

The Partition layer is composed of multiple servers managing the different partitions. One partition server can manage one or multiple partitions, but a given partition is managed by only one partition server. For the BlobStorage, this is obvious since the partition key is down to the blob name. A partition master monitors each partition server's load and determines load balancing.\\

\subsubsection{Stream Layer}

The Stream layer is responsible of persisting data on disk and replicating the data across multiple servers within a storage stamp. The stream layer is a kind of distributed file system within a stamp.\\

Following \cite{AzureStorage}, the minimum unit of data for writing and reading is called a Block. A typical Block is bounded to 4 MegaBytes. An extent is a set of consecutive and concatenated blocks. Each file stored by the stream layer is referred as a stream. Each stream is a list of pointers referring to extent locations.\\

The target extent size used by the partition layer is 1 GigaBytes. The very large objects are split by the partition layer into as many extents as needed. In the case of small objects, the partition layer appends many of them to the same extent and even in the same block. Because of the checksum validation is performed at the block level, the minimum read size for a read operation is a block. It does not mean that when a customer request a 1 kilobyte object the entire block is send back to the customer, but that the stream layer reads the entire block holding the object before returning the actual small object.\\

Because a stream is only an ordered collection of pointers to extents, a new stream can be very quickly created as a new collection of pointers of existing extents.\\

A key feature of the stream layer is that all writes are append-only. When a write occurs, the last extent is appended with one or multiple blocks. This append operation is atomic: either all the entire blocks are appended, or none are. The atomicity is guaranteed by a two-step protocol. Blocks are firstly written into the disk and appended after the last block of the last extent of the stream. When these write operations are completed, the stream block and extent pointers list is updated accordingly.\\

An extent has a target size, defined by the partition layer. When an append enlarge an extent to the point it exceed this target size, the extent is sealed and a new extent is added to the end of the stream. Therefore, all the extents but the last one are immutable in a stream and only the last one can be appended.\\

Partition servers and stream servers are co-located on the same storage nodes.

\subsubsection{Extent replication and strong consistency}

A key feature of data storage is data persistency, durability and availability. Since data servers are expected to fail from time to time, data are duplicated several times, so a data server can die without any data loss. Yet, in the case of a networking partitioning, availability and consistency are two properties that hard to met at the same time, as stated by the CAP theorem (see \ref{Cap_theorem_section}).\\

The default azure policy is to keep three replicas for each extent within the same storage stamp (according to \cite{AzureStorage}). This policy is referred as the intra-stamp replication policy and is intended to guarantee no data loss, even in the event of several disk, nodes or rack failures. A second policy designed to prevent data loss in the rare event of a geographical disaster is called inter-stamp replication policy. This replication process ensures that each data extent is stored in two different datacenters, geographically distant. This section details how the inter and intra-stamp replication policies fit into the CAP theorem constraints.\\

The stream layer is responsible of the intra-stamp replication process. When an extent is created, the Stream Manager promotes one of the three extent nodes (EN) into a primary EN. The partition layer is told of the primary EN, and all the write requests of the partition layer are send to this primary EN. The primary EN is in charge of applying the write request to its extent but also to make sure the two other replicas are updated the same way. It is only once all the three extents have been updated that the partition layer is notified by the primary EN of the success of the write request. Therefore, this update process is kept as quick as possible since it impacts customer's request latency as no write success is returned before all the extents are updated. Because of all the three extents are located in the same storage stamp, this update process can be kept quick. This update mechanism ensures strong consistency among the three extent replicas, and each read request can be addressed by any of the EN since all the three replicas are always guaranteed to be the same.\\

During a writing event, if an extent node is unreachable or the write operation fails on this node, the partition layer is notified the write operation has failed. The Stream Manager then sealed the extent in the state it was before the write failure, adds a new unsealed extent in the end of the stream and promotes one of the three extent nodes as a primary node for this extent. This operation is reported in \cite{AzureStorage} to be completed on average within 20ms. The partition layer can therefore resume the write operation in a short time without waiting for the failing nodes to become available or operational again.\\

The partition layer is responsible of the inter-stamp replication process. It is a low priority \textit{asynchronous} process scheduled so that it does not impact customer's requests performance. On the contrary to the intra-stamp replication process run by the stream layer, the inter-stamp replication process is not designed to keep data durable in the event of hardware failures. Rather, it is designed to store each data chunk in two distinct datacenters in the rare event of a geographical disaster happening in one place. It is also used to migrate accounts between stamps to preserver the 70\% usage ratio of each storage stamp.\\

\subsection{Serialization}

\subsection{BlobStorage or TableStorage ?}

BlobStorage and TableStorage are both designed to store data. When should we use one or the other ? Here are some part of the answer :
\begin{itemize}
\item One part of the answer is the data locality management and atomicity. Every data object may they be blobs or table entities are stored in terms of their partition key. This partition key determines how Azure must load balance and partition the objects across Azure servers. Partition Key for TableStorage is the tableName + partitionkey (primary key), whereas PartitionKey for blobs are determined by containerName + blobName. Since the partition key is down to the blob name for blobStorage, access to different blobs can be load balanced  across as many servers as needed to scale. But no atomic transactions is authorised for multiple blobs. As explained below in design subsection \ref{Redesign of the algorithms}, jobs will need to be idempotent : no atomic transaction for blobStorage is another difficulty to guarantee the idempotency. Since partition key is up to the primary key in TableStorage, we can run atomic transactions up to 100 entities who share the same table + primarykey (Azure guarantees that all the entities will be changed the specified way, or no one will).
\item When dealing with really large objects, BlobStorage is easier to manipulate, since each blob can be arbitrarily large (up to 50 GB per blob). We do not need to break an object into a lot of entities, making sure each time each "part entity" respect the 1MB constraint. BUT WE HAVE TO DEAL WITH BLOCKS : SAY LOKAD.CLOUD HANDLES IT
\item When manipulating objects than can be stored both in TableStorage and BlobStorage, BlobStorage is reported by \cite{Early_Observations_On_Azure} to be more effective in insert operations on blobs larger than 4KBytes, which is below the average size of blobs we are going to use.
\item Price is another factor that might affect our choice. Using Azure storage, three things are charged : size of our storage, I/O communications, requests to the storage. A single query on TableStorage can load up to 100 elements at a time and costs as much as a single query on blobStorage, see \cite{Azure_Pricing}. If we have to store a lot of small objects on different places, using TableStorage will thus allow to run request on objects by packet of 100, so it would divide up to a factor 100 the bill of storage requests in comparison of running requests one by one on the BlobStorage. If we do not need to store the objects on different places, we could store them in one bigger blob, bounding the request to one. Yet, the more times we have to modify a small part of the storage, the less storing all the data in a single blob is the right thing, since we are going to download and upload the big blob each time a small part of the data need to be updated, making I/O costs predominant comparing to storage requests. So, using TableStorage or BlobStorage on the price criterion will depend on the size of your data, your application, the way the application is designed, your need to update small part of data or entire data each time, ...
\end{itemize}

To run our experiments, we chose to use the BlobStorage. Atomic updates on multiple object at once will not be a key point in our code, except for counters for which concurrency will be handled by etag mechanism. Price will be estimated but is not a criterion here. Besides, BlobStorage is easier to manipulate and due to its complexity, TableStorage suffered little bugs that affect performances when we ran our experiments (in June 2010).\\

\section{Windows Azure Compute}
\label{azure_framework}

\subsection{Idempotency}

Let's consider the general problem of message processing. Some worker will consume a message and then update some data stored in a durable way (for example in BlobStorage).
The message is consumed by the worker, communication is made between the worker and storage, data is updated in the storage, but the worker dies before it acknowledges the queue that the message has been processed and should be deleted. Queues have been designed to react to machine failures as a "at-least-once" messaging device : in a failure, message is requeued after some timespan and another worker will process the same message, resulting in a second update operation being performed on the same data chunk.\\

Technically, this issue derives from the fact there are no direct mechanism to couple the message consumption and deletion with the update of the persistent data chunk in such a way that one of these two events cannot happen without the other. The absence of this coupling leads to failure windows between the end of data update and message deletion in which the message is delivered and processed more than once.

\begin{figure}
\begin{center}
\includegraphics[height=18em]{../../Schemes/idempotencyIssues.pdf}
\end{center}
\caption{Some scenario of data updates.}
\label{fig:processService}
\end{figure}

\subsection{Affinity between workers and storage}

Presently, there are no mechanisms in Azure that provides the same affinity between the workers and the storage than MapReduce does.
This means that we cannot push some jobs where the data is stored since we can't know on which machine a data piece is stored. Each worker processing data will need to first download the data it needs to process, on the contrary of Google's Map Reduce, where the jobs are scheduled and assigned on each worker by the framework in such a way that workers downloading data will be minimized.

This is intended by Microsoft (CITATION NEEDED) to simplify development of cloud apps : one does not need to design its application to take data location into account, Azure providing a satisfactory scale-up and bandwidth. This choice is about simplicity over performance.

\subsection{Workers Communication}

Azure does not offer currently any standard API for distributed computation, neither a low
level one such as MPI \cite{MPI1996}, nor a more high level one such as Map
Reduce \cite{MapReduce2004} or Dryad \cite{Dryad2007}. Map reduce could be
implemented using Azure components (following the strategy of
\cite{CloudMapReduce2010}), yet, as pointed out in e.g. \cite{Graphlab2010},
those high level API might be inappropriate for iterative machine learning
algorithms such as the k-means or learning vector quantization because of MapReduce overhead on synchronisation and waiting process.

We therefore need to explicitly design our communication process between Azure workers, process that can realized by 2 means :

\begin{enumerate}
\item Workers can communicate each other through direct IP communications, that is to say re-implement a peer-to-peer network inside Azure cloud. TO DEVELOP
\item Azure storage system can be used to implement synchronization and
communication between workers : each worker is pinging (or "listening") a specific queue, and it gets some message from this queue holding
 some blobname in BlobStorage where it can retrieved the data it needs.
\end{enumerate}

While the first solution comes with higher performance in terms of bandwidth, as reported in AzureScope or ... (CITATION NEEDED), it implies a very tough work which would be like implementing MPI on Azure, which was outside the scope of this thesis. Moreover, the "storage approach" for communication is much more in agreement with Azure's abstractions and design thinking (CITATION NEEDED).
Thus, we will rely therefore directly on Azure queues and blob storage to build all our communication systems.

\subsection{Sharded Counters}

\subsection{Lokad-Cloud}

\textit{Lokad-Cloud}\footnote{\url{http://code.google.com/p/lokad-cloud/}}, an
open-source framework that adds a small abstraction layer to ease Azure
workers startup and life cycle management, and storage access.

\section{Azure Performance}
\label{azure_performance}

\subsection{Performance Tradeoffs in Azure}

\textbf{Read/Write tradeoff} : As reported in \cite{BenchmarkingCloudServices}, distributed storage must choose a tradeoff between read throughput and write throughput. Taking the example of a blob in BlobStorage, a blob cannot be at the same time highly available in read operations and write operations (having a highly available blob for read operations implies the blob to be duplicated, so write operations must change several replicas, and write operations take more time). Azure let the traffic arbitrate this tradeoff : the more request to a blob per time unit, the more the blob is duplicated, the more it becomes available, and the more time it takes to run write operations.\\

\textbf{Responsivity/Efficiency} \cite{BenchmarkingCloudServices} is also reporting some tradeoff between services responsivity and efficiency. Efficiency, as defined as the ratio of work completed per hour per worker is a decreasing function of workers and an increasing function of total workload. Therefore, to achieve the best possible efficiency, we should make sure every worker is overworked. Yet, the more overworked workers are, the least responsive they can be on an additional task. Since the first goal of our algorithm is responsivity (we want to minimize amount of time spent processing the algorithm), we will make sure to provide a pool of workers slightly greater than the number of jobs we can parallelize, so we make sure all jobs that can be run parallelingly, are (having P jobs run by P-1 workers would lead approximately to twice the time P jobs running by P or P+1 workers). \\

\textbf{Synchronous Replication/Replication Latency tradeoff} : TO BE DIGGED. RELATED TO CONSISTENCY

\subsection{Benchmarks}
The Azurescope website \cite{AzureScope} is reporting performance benchmark run from February/March 2010 to presently (June 2010) on Azure hardware.
Some paper have also been released very recently on same measures \cite{Early_Observations_On_Azure}.
Numbers reported in \cite{Early_Observations_On_Azure} seem to be more accurate than \cite{AzureScope}, and the methodology is better explained, so we will use these numbers as reference.
We will use the numbers observed to understand if bandwidth is an issue on Azure.

\begin{itemize}
\item BlobStorage Read Bandwidth : 13 MBytes/sec for 1 worker, BlobStorage aggregated read bandwidth bounded to 393 MBytes/sec when using more workers (each worker is using several threads to load data from storage). This quantity describes the bandwidth for several workers to download the same blob from the storage.
\item BlobStorage Write Bandwidth : 5 MBytes/sec for 1 worker, BlobStorage aggregated write bandwidth bounded to 124 MBytes/sec when using more workers (each worker is using several threads to upload data). This quantity describes the bandwidth for one or several workers to load data from different blobs but within the same container.
\item Communication throughput between 2 workers : 55 MBytes/sec on large messages on average (which is a very impressive bandwidth).
\item CPU power depends on type of VM allocated : from 1 core (on small VM) to 8 cores on extra-large VM
\item Queue throughput is presently 500 messages per second, but might be very increased soon.
\end{itemize}

Note : Since concurrent write on a blob is very costly, we will not write twice at a same place, unless when using counters.\\

\section{Prices}
\label{azure_prices}
