sqs

Storage Gateway

AWS Storage Gateway is a service that connects an on-premises software appliance with cloud-based storage to provide seamless and secure integration between an organization’s on-premises IT environment and AWS’s storage infrastructure. The service enables you to securely store data to the AWS cloud for scalable and cost-effective storage.

AWS Storage Gateway software appliance is available for download as a virtual machine (VM) image that you install on a host in your data-center. Once you’ve installed your gateway and associated it with your AWS account through our activation process, you can use the AWS Management Console to create either gateway-cached or gateway-stored volumes that can be mounted as iSCSI devices by your on-premises applications.

Storage Gateway in two different models Gateway-cached and Gateway-stored

Gateway-cached :-

Gateway-cached volumes allow you to utilize Amazon S3 for your primary data, while retaining some portion of it locally in a cache for frequently accessed data. These volumes minimize the need to scale your on-premises storage infrastructure, while still providing your applications with low-latency access to their frequently accessed data. You can create storage volumes up to 32 TBs in size and mount them as iSCSI devices from your on-premises application servers. Data written to these volumes is stored in Amazon S3, with only a cache of recently written and recently read data stored locally on your on-premises storage hardware.

Gateway-stored :-

Gateway-stored volumes store your primary data locally, while asynchronously backing up that data to AWS. These volumes provide your on-premises applications with low-latency access to their entire data sets, while providing durable, off-site backups. You can create storage volumes up to 1TB in size and mount them as iSCSI devices from your on-premises applications servers. Data written to your gateway-stored volumes is stored on your on-premises storage hardware, and asynchronously backed up to Amazon S3 in the form of Amazon EBS snapshots.

Storage Gateway Pricing

With AWS Storage Gateway, you pay only for what you use. AWS Storage Gateway has four pricing components: gateway usage (per GB per month) so the number of gateway using per month, snapshot storage usage (per GB per month), volume storage usage (per GB per month), and data transfer out (per GB per month).

DynamoDB

Amazon DynamoDB is a fast and flexible No-SQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. It’s flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.

DynamoDB Configuration

1. It’s always going to be stored on SSD storage. So there is no magnetic storage you going to get very good and very high IOPS from it.
2. It is Spread Across 3 different geographically distinct data-centers. So if you write a database you write a record to a particular a AZ that is going to be replicated across to your other two AZ and in terms of that replication you can choose between two options;

Eventual Consistent Reads Strongly Consistent Reads

Difference Between Eventual Consistent Reads and Strongly Consistent Reads

Eventual Consistent Reads :-

Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the updated data. (Best Read Performance). So this means that when you run a read query against your database you might be querying at two unavailability zone that has not yet had that data that is been from the initial write which should be another AZ has not been replicated across. So let say you are writing your data to AZ-(A) and then when you going to read the database it might not be the in AZ-(B) depending on the length of the time between that write and that read that would be Eventual Consistent.

Strongly Consistent Reads :-

A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read. So with Strongly Consistent Reads you would not get the same read performance you get with the Eventual Consistent Reads but you do more less guarantee reflect that after somebody has written record to AZ-(A) nobody would be able to read that record in AZ-(B) into has been replicated across. So just keep that in mind when you designing your application whether or not (We are talking about millisecond you know we are not talking within a seconds) so it’s really up to you and your application team is to which one the you would choose most people choose the default which is you know we can afford for my data to be out of date or not replicated within a second so that its Eventual Consistent Reads.

Pricing

1. Provisioned Throughput Capacity

Write Throughput $0.0065 per hour for every 10 units. Read Throughput $0.0065 per hour for every 50 units.

1. Storage costs of $0.25Gb per month.

Pricing Example

Let’s assume that your application needs to perform 1 million writes and 1 million reads per day, while storing 3GB of data.

First, you need to calculate how many writes and reads per second you need. 1 million evenly spread writes per day is equivalent to 1,000,000 (writes) /24 (hours) /60 (minutes) /60 (seconds) = 11.6 writes per second.

A DynamoDB Write Capacity Unit can handle 1 write per second, so you need 12 Writes Capacity Units. Similarly, to handle 1 million strongly consistent reads per day, you need 12 Read Capacity Units.

Using on-demand pricing in the US East (N. Virginia) Region. You got 12 Write Capacity Units would cost $0.1872 per day and 12 Read Capacity Units would cost $0.0374 per day. So your total cost of provisioned throughput capacity is $0.1872 + $ 0.03474 = $0.2246 per day. Storage costs $0.25 per GB per month.

Assuming a 30-day month, your 3GB would cost you 3 * $0.25/30 = $0.025 per day. Combining these numbers, the total cost of your DynamoDB table would be $0.2246 (for provisioned throughput capacity) + $0.025 (for storage) = $0.2496 per day or $7.50 per month.

Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

So we have an example of a web application may be you upload an image file to this web application and then what that application will do is load the SQS that a user has uploaded it an image file and that job needs to be executed on its stored that message on the SQS system.

Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.

Using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages pro-grammatically using the Amazon SQS API.

The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

Amazon SQS ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue. A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.

Amazon SQS is engineered to always be available and deliver messages. One of the resulting trade offs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.

To illustrate, suppose you have a number of image files to encode. In an Amazon SQS message for each file specifying the command (jpeg-encode) and the location of the file in Amazon S3. A pool of Amazon EC2 instances running the needs image processing software does the following:

1. Asynchronously pulls the task messages from the queue so that go in and pull the messages from queue. SQS always pulls the messages never the message queue never pushed out you have got EC2 instances constantly polling the queue trying to polling the data down.
2. Polling the messages from the queue then Retrieves the filename of the image.
3. Processes the conversion so might apply watermark.
4. Writes the image back to Amazon S3
5. Writes a “task complete” message to another queue
6. Deletes the original task message
7. Checks for more messages in the worker queue
8. 30 Seconds visibility Time Out by Default. Maximum of 12 hours.

SQS Pricing

1. First 1 million Amazon SQS Requests per month are free.
2. $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request).
3. A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
4. Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.