AWS S3

Amazon Simple Storage Service #

AWS S3 is a highly scalable and durable object storage service provided by Amazon Web Services (AWS). It is designed to store and retrieve data from anywhere on the web.

AWS S3 is widely used by organizations for data storage, backup, content distribution, data archiving, and as a foundation for building cloud-native applications. Its simplicity, scalability, and durability make it a fundamental component of many AWS cloud solutions.

Object Storage #

S3 allows object storage and retrieval, which can include virtually any type of data, such as documents, images, videos, backups, and application data. S3 is object storage, not a file system or a block device: it is not appropriate for data that is modified in place all the time, such as the files of a database (use EBS or EFS for that).

Scalability #

S3 is designed to scale automatically as the storage needs grow (unlimited amounts of data with no need to do capacity planning).

Security #

S3 offers multiple security features: IAM and bucket policies, Block Public Access, encryption at rest (all new objects are encrypted by default, with S3-managed keys or with KMS keys) and encryption in transit with HTTPS. Access control lists (ACLs) are legacy and disabled by default in new buckets, use policies instead.

Availability #

S3 is designed for high availability, providing a reliable platform for hosting data that needs to be accessible 24/7.

Durability #

Data stored in S3 is highly durable and is redundantly stored across multiple devices and, in most storage classes, across multiple Availability Zones within an AWS Region, ensuring data resilience.

• 11 Nines Durability: S3 is designed for 99.999999999% durability of objects.
• Multi-AZ Redundancy: By default, S3 data is stored across multiple Availability Zones (AZs) within a region. This means even if one AZ suffers a catastrophic event, data remains safe and accessible in other zones.
• Redundant devices: objects are stored on multiple devices, so the failure of a device does not cause data loss.
• Replication: S3 Replication can also copy objects automatically to a bucket in another Region or account.
• Data integrity checks: S3 employs checksums and other techniques to constantly verify data integrity. Any corrupted data is automatically detected and repaired, ensuring reliable access.

Data Lifecycle Management #

S3 Lifecycle policies transition objects to other storage classes (such as S3 Glacier) based on specific criteria, helping to optimize costs over time.

Versioning #

S3 allows versioning for objects, which helps recover previous versions of data in case of accidental deletions or modifications.

Data Transfer Acceleration #

AWS provides a feature called Amazon S3 Transfer Acceleration, which speeds up the uploading and downloading of objects using Amazon CloudFront’s globally distributed edge locations.

Integration #

S3 integrates seamlessly with other AWS services, making it a key component for building scalable and resilient cloud-based applications.

Data Analytics #

You can use S3 as a data lake to store and analyze large datasets, integrating with Amazon Athena and Amazon Redshift.

Storage Classes #

S3 offers several types of storage classes based on data access frequency and durability requirements:

S3 Standard #

This is the default storage class offering high durability, availability, and performance. It’s suitable for frequently accessed data.

S3 Intelligent-Tiering #

It automatically moves objects between access tiers as the access patterns change (frequent, infrequent and archive instant access, plus optional archive tiers). It optimizes costs by charging lower fees for data that is not accessed, with no retrieval fees.

S3 Standard-IA (Infrequent Access) #

Designed for data that is accessed less frequently but requires rapid access when needed. It offers lower storage fees but slightly higher retrieval fees compared to S3 Standard.

S3 One Zone-IA #

Similar to S3 Standard-IA but stores data in a single Availability Zone, reducing costs. However, the data is lost if that zone is destroyed, so it is only suitable for data that can be re-created.

S3 Glacier #

The S3 Glacier storage classes are low-cost classes designed for long-term archiving and backup, where retrieval speed is less critical. They are used for compliance, data archival and backups of large volumes of data over extended periods.

• S3 Glacier Instant Retrieval: for rarely accessed data that still needs millisecond access.
• S3 Glacier Flexible Retrieval: retrieval in minutes to hours, with a lower storage cost.

Objects are moved to these classes with a lifecycle policy or by uploading them directly with the storage class, and they are managed like any other S3 object. The original standalone Amazon Glacier service, with vaults and archives, is a previous generation of the service.

S3 Glacier Deep Archive #

This is the most economical option for archiving data. It offers the lowest storage costs but has the longest data retrieval times (up to hours).

S3 Express One Zone #

A high-performance class for latency-sensitive applications, with single-digit millisecond access, stored in one Availability Zone in a directory bucket.

Using S3 #

S3 API #

For Cloud native applications, the S3 API allows developers to programmatically manage and access objects (files) stored in Amazon S3 buckets. Developers can interact with the S3 API using various SDKs (Software Development Kits) provided by AWS in different programming languages such as Python, Java, JavaScript, and more. AWS Command Line Interface (CLI) also uses the S3 API to perform operations on S3 buckets and objects.

Mountpoint for Amazon S3 #

An open-source file client that makes it easy for your file-aware Linux applications to connect directly to Amazon Simple Storage Service (Amazon S3) buckets.

AWS Output to S3 #

Many AWS services can be configured to output files (objects) to S3. Examples are Amazon RDS, Amazon Redshift, AWS Backup, CloudTrail Logs, and Amazon EMR (Elastic MapReduce).

S3 as a Static Website Hosting #

An S3 bucket can be configured to host static files for a website, accessed with a fully qualified domain name using DNS. To serve it over HTTPS and with low latency, put CloudFront in front of the bucket.

With Terraform #

The resources are aws_s3_bucket and the resources for each feature, such as aws_s3_bucket_versioning, aws_s3_bucket_server_side_encryption_configuration, aws_s3_bucket_public_access_block and aws_s3_bucket_lifecycle_configuration. S3 is also the most common backend to store the Terraform state.

See tutorials:

• AWS with Terraform: The Essential Guide (19/21) – Terraform Backends