In this chapter, we will investigate various Azure Storage options for our game services. In the previous chapter, our service just read data from resource files. That isn’t very useful for most services, but it allowed us to focus on understanding Azure Functions without getting into storage complexities. However, Azure provides many storage capabilities. All the way from raw unstructured data to relational databases, and everything in between. We will investigate these options and create repositories for applicable options.
The Azure Storage platform includes the following data services:
- Azure Blobs: A massively scalable object store for text and binary data.
- Azure Files: Managed file shares for cloud or on-premises deployments.
- Azure Queues: A messaging store for reliable messaging between application components.
- Azure Tables: A NoSQL store for schema-less storage of structured data.
- Azure Disks: Block-level storage volumes for Azure VMs, which is not applicable for our use.
- Azure Cosmos DB: A NoSQL document store for non-structured and structured data.
- Azure SQL Database: A managed instance of SQL database for structured, relational data.
Along with SQL Server, Azure supports several different SQL implementations: PostgreSQL, MySQL, MariaDB, etc. We won’t investigate each type of SQL database, but the general principles will be similar regardless of the database technology chosen (just the setup of each database will be different).
Regardless of the storage service selected, Azure Storage services are:
- Durable and highly available. Redundancy ensures that our data is safe in the event of transient hardware failures. We can also opt to replicate data across datacenters or geographical regions for additional protection from local catastrophe or natural disaster.
- Secure. All data written to an Azure storage account is encrypted by the service. Azure Storage provides us with fine-grained control over who has access to the data.
- Scalable. Azure Storage is designed to be scalable to meet the data storage and performance needs of many applications.
- Managed. Azure handles hardware maintenance, updates, and critical issues for us.
- Accessible. Data in Azure Storage is accessible from anywhere in the world over HTTP or HTTPS. There are client libraries for Azure Storage in a variety of languages, including .NET, Java, Node.js, Python, PHP, Ruby, Go, and others.
As we move up the stack from blob storage up through full-fledged databases, the capabilities of the stores increase greatly. But the costs of Azure Storage also increases with those capabilities. We will review the cost links at the end of this lesson.
Azure Blob is an object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data, such as text or binary data.
Blob storage is ideal for:
- Serving images or documents directly to a browser.
- Storing files for distributed access.
- Streaming video and audio.
- Storing data for backup and restore, disaster recovery, and archiving.
- Storing data for analysis by an on-premises or Azure-hosted service.
Objects in Blob storage can be accessed from anywhere in the world via HTTP or HTTPS. Users or client applications can access blobs via URLs, the Azure Storage REST API. We will build a repository that reads our game data from Blob storage.
Azure Files enables us to set up highly available network file shares. That means that multiple VMs can share the same files with both read and write access. We can also read the files using the REST interface or the storage client libraries. File shares can be used for many common scenarios:
- Many on-premises applications use file shares. This feature makes it easier to migrate those applications that share data to Azure.
- Configuration files can be stored on a file share and accessed from multiple VMs.
- Resource logs, metrics, and crash dumps are just three examples of data that can be written to a file share and processed or analyzed later.
Although Azure Files are useful for various scenarios, they aren’t optimal for our Azure Functions services, so we won’t use them in future lessons.
The Azure Queue service is used to store and retrieve messages. Queues are generally used to store lists of messages to be processed asynchronously.
For example, if we want our customers to be able to upload pictures, and we want to create thumbnails for each picture. We could have our customer wait for our service to create the thumbnails while uploading the pictures. An alternative would be to use a queue. When the customer finishes their upload, write a message to the queue. Then have an Azure Function retrieve the message from the queue and create the thumbnails.
While queues are very useful in many scenarios, like processing purchases on an e-commerce website. It is not applicable to our current game data scenario, so we won’t be exploring it in further detail in this chapter.
Azure Table Storage
Azure Table storage is a service that stores non-relational structured data in the cloud, providing a key/attribute store with a schema-less design. Because Table storage is schema-less, it’s easy to adapt the data as the needs of our application evolve. Access to Table storage data is fast and cost-effective for many types of applications, and is typically lower in cost than traditional SQL for similar volumes of data. This should work pretty well for our game data.
Table storage usage includes:
- Storing TBs of structured data capable of serving web scale applications
- Storing datasets that don’t require complex joins, foreign keys, or stored procedures and can be de-normalized for fast access
- Quickly querying data using a clustered index
- Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries
Azure Cosmos DB
Azure Cosmos DB is a fully managed NoSQL database for modern app development. Quick response times, and automatic and instant scalability, guarantee speed at any scale. App development is faster and more productive thanks to data distribution anywhere in the world, open source APIs, and SDKs for popular languages. As a fully managed service, Azure Cosmos DB takes database administration out of our hands with automatic management, updates, and patching. It also handles capacity management with cost-effective server-less and automatic scaling options that respond to demand.
Key benefits include:
- Guaranteed speed at scale: real-time access with fast read and write latencies globally, and throughput and consistency
- Simplified application development: integrated with Azure Functions, supports multiple database APIs, and supports various popular platforms: .NET Core, Java, NodeJS, and Python.
- End-to-end database management, with serverless and automatic scaling matching your application needs
We will implement a repository that uses Cosmos DB for our game data.
Azure SQL Database
Azure SQL Database is a fully managed platform as a service (PaaS) database engine that handles most of the database management functions such as upgrading, patching, backups, and monitoring without our involvement. Azure SQL Database is always running on the latest stable version of the SQL Server database engine and patched OS with 99.99% availability.
Azure SQL Database is based on the latest stable version of the Microsoft SQL Server database engine. We can use advanced query processing features, such as high-performance in-memory technologies and intelligent query processing. We get the newest SQL Server capabilities with no overhead for patching or upgrading.
We will update our data to be a little more relational and show using a database backend for our game data.
With the Azure pricing tiers, there are many cost effective storage options. The cheapest option is Azure Blob storage, but it is also the least feature-rich. Azure SQL is typically the most expensive, but we are using a full SQL database engine, so that is understandable. And it is still less expensive than running your own SQL database on-premise or even in an Azure VM.
There are also some options, like Table storage and Cosmos DB, that can run in a serverless fashion, where you pay for the amount of data and number of operations (measured in DTUs) used rather than paying for fixed capacity. These provide higher value features, and can be cost effective as you start your new services.
Here are some links with pricing details for different Azure Storage options:
- Azure Storage Pricing: for the base storage types
- Azure Tables Storage Pricing
- Pricing model of Azure Cosmos DB
- Price guidance & managing costs – SQL Server on Azure
Pricing is always a concern when using resources in our cloud provider. Azure gives us lot of options with different pricing tiers, including per-use (DTU) pricing for some of the high-cost storage options. This gives us the benefit of cheaper costs up-front while building and growing our service. And, it is possible to move from the per-use model to fixed price costing when our needs increase.
As we can see, there are a lot of different options for storage in Azure. Some of these options are applicable to our service requirements, so we will investigate those further. In the coming lessons, we will update our services to consume different repository implementations. These repositories will target different storage options, so we can see them each in action. We will discuss some of the pros and cons of each as we use them.
While it isn’t common to support multiple storage backends in an application, doing so in our sample allows us to learn each technology with a sample that we are already familiar with. And with microservices, different independent services can use the storage technology that best supports the service. So, it is plausible to have services that use Cosmos DB, which communicate with other services that use Blob Storage and even SQL databases. Knowing them all is beneficial to any web service developer.