Scale - the dataset exceeds memory, the model exceeds GPU capacity; no larger machine exists

Reliability - hardware fails, power drops, updates restart the OS; one machine is one failure domain

Geography - Tokyo to a Los Angeles server is ~100 ms round-trip; distance sets the floor

Elasticity - demand moves 10× in hours; new hardware takes weeks

Whatever capacity is bought is wrong twice: idle at baseline, saturated at peak.

Operating infrastructure requires

• Physical hardware: servers, storage, networking
• Facilities: power, cooling, physical security
• Staff: installation, maintenance, monitoring
• Capacity planning with lead times of weeks to months

These costs are largely fixed: a datacenter serving 100 users costs nearly as much as one serving 10,000.

The rental model

• Providers run infrastructure at scale and amortize the fixed costs across many customers
• Capacity is metered, appears on demand, and bills by use

Providers

• Amazon Web Services, Microsoft Azure, Google Cloud Platform
• One underlying model, different APIs

Provisioning for peak strands capacity

• Retail sizes for Black Friday: 10× normal load
• Tax software sizes for April: 50×
• Measured enterprise utilization averages 15-25%
• A 3-5 year hardware lifecycle locks the surplus in

Staffing has a floor

• Network, storage, security, and database specialists at $150K-$300K salaries, plus on-call coverage
• A 50-person company cannot staff a 10-person infrastructure team

Pooled peaks average out

• Peak times differ by customer; the pooled load is smoother than any single customer’s
• Pooled utilization reaches 60-70% against the enterprise 15-25%

Scale lowers unit costs

• Bulk purchasing: hundreds of thousands of servers a year
• Custom hardware and industrial power rates
• Automation and specialist staff amortized across millions of customers

CapEx: locked in upfront

• Servers cost $5K-$50K each; a modest deployment runs $500K-$5M before the first customer
• Depreciates over 3-5 years regardless of use
• Lead time of weeks to months
• A wrong forecast leaves stranded capital or lost customers

OpEx: scales with usage

• Starts at $0; billed per hour, second, or request
• Stopped resources stop billing
• Capacity adjusts in minutes
• Experiments are cheap to try and discard

Market position

• Roughly a third of the market, the largest share; Azure and GCP follow
• First mover: S3 launched March 2006, EC2 August 2006
• 20 years of continuous production operation

API stability

• 200+ services across compute, storage, networking, ML
• APIs versioned by date: S3 requests still accept version 2006-03-01
• Code written against old versions keeps running

Creating an EC2 instance:

POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
Authorization: AWS4-HMAC-SHA256 ...

Action=RunInstances
&ImageId=ami-0abcdef1234567890
&InstanceType=t3.micro
&MinCount=1&MaxCount=1

The response contains an instance ID - a handle to a running VM in AWS infrastructure.

Three interfaces construct the same request:

Console - web UI

CLI - command-line tool

SDK - library (Python, Go, etc.)

The Console suits exploration; the CLI and SDK enable automation. Anything the Console does, code can do.

Scale (2026):

• 39 regions, 124 availability zones
• Every region: minimum 3 AZs
• 100+ Tbps network backbone capacity

Each region has its own:

• Compute and storage infrastructure
• Control plane (API endpoints)
• Network connectivity

Resources created in us-east-1 do not exist in eu-west-1. An outage in one region does not directly affect others.

Region selection determines:

• Physical location of data and compute
• Latency to end users
• Applicable legal jurisdiction (e.g., GDPR)
• Available services (varies by region)
• Pricing (varies ~10-20%)

us-east-1 contains six AZs:

us-east-1a through us-east-1f

Physical characteristics:

• Separate buildings, miles apart
• Independent power (different substations)
• Independent cooling
• Independent network paths

Interconnected:

• Dedicated high-bandwidth fiber
• Single-digit millisecond latency between AZs

Failure isolation

• Datacenter failures happen: power outages, cooling failures, network cuts, fires
• AZs are engineered so a failure in one facility does not spread to the others; if us-east-1a loses power, us-east-1b through us-east-1f continue operating
• The isolation is strong but not absolute

AZ names are per-account:

• An account’s us-east-1a may map to a different physical facility than another account’s
• AWS randomizes the mapping to spread load across facilities
• Cross-account coordination uses AZ IDs: use1-az1, use1-az2, etc.

Global - exist once, no region

• IAM users and roles, Route 53 zones, CloudFront distributions
• IAM policies apply across all regions

Regional - one region, span its AZs

• S3 buckets, VPCs, Lambda functions, DynamoDB tables
• Data replicated across AZs for durability

Per-AZ - hardware in one facility

• EC2 instances, EBS volumes, subnets

The attachment constraint

• Per-AZ resources attach only within their AZ
• An EBS volume in us-east-1a cannot attach to an instance in us-east-1b

Specified in the request

• A virtual machine with 2 CPUs and 8 GB RAM
• A storage bucket for objects
• A PostgreSQL database with 100 GB

Decided by AWS

• Which physical server hosts the VM
• Which storage arrays hold the data
• When workloads migrate for maintenance

RunInstances     → VM running on some server
CreateBucket     → storage on some drives
CreateDBInstance → database on some hardware

Latency

• Data travels at finite speed
• Virginia to Tokyo ≈ 150 ms round-trip; fiber sets a floor near 110 ms

Jurisdiction

• Data stored in eu-west-1 physically resides in Ireland, subject to EU law

Failure

• Hardware fails; AWS absorbs many failure modes, and some propagate to applications

The abstraction hides operational complexity, not physical reality.

Orders of magnitude

Local operations measure in nanoseconds to microseconds; network operations in milliseconds - a factor of 10⁴ to 10⁶.

A web request making 10 database queries

• Local SQLite: 10 × 0.1 ms = 1 ms
• Same-AZ RDS: 10 × 3 ms = 30 ms (round trip plus query execution)
• Cross-region: 10 × 100 ms = 1 s

The algorithm is unchanged; the environment multiplies its cost.

Design responses:

• Batch operations (1 query returning 100 rows, not 100 queries)
• Cache aggressively (avoid repeated network calls)
• Co-locate data and compute (same AZ when possible)

Local failure model

• A crash stops everything; out of memory kills the process; a full disk fails every write
• Failure is total and obvious: fix and restart

Distributed failure model

• A database responds, but in 2 s instead of 20 ms
• S3 returns errors for some requests, not all
• One of five servers is unreachable
• The network drops 0.1% of packets

Failure is partial and subtle: the system keeps running, incorrectly.

Example: uploading a file

# Local: works or throws
with open('output.csv', 'w') as f:
    f.write(data)

# S3: the network can drop mid-upload
s3.put_object(Bucket='b', Key='output.csv', Body=data)
# Succeeded? Partially? Retry?

Required patterns

• Timeouts - bound every wait
• Retries with backoff - retry transient failures without hammering the service
• Idempotency - make retries safe to repeat
• Health checks - detect degraded components

At scale these are continuous operation, not edge cases.

What the instance receives:

• Virtualized CPU cores
• Allocated memory
• Virtual network interface
• Block device for storage

From inside the VM, this looks like a physical machine. The OS sees CPUs, RAM, disks, network interfaces.

What stays with AWS:

• Physical server selection and placement
• Hypervisor operation
• Hardware failure handling
• Physical network infrastructure
• Datacenter operations

The request specifies resources; AWS selects the physical server that provides them.

General purpose (t3, m5):

Balanced CPU-to-memory ratio. Suitable for most workloads without extreme requirements in either dimension.

• m5: consistent performance
• t3: burstable (accumulates CPU credits when idle)

Compute optimized (c5):

High CPU-to-memory ratio. For CPU-bound workloads: batch processing, scientific modeling, video encoding.

c5.4xlarge: 16 vCPUs, 32 GB memory (2:1 ratio)

Memory optimized (r5, x1):

High memory-to-CPU ratio. For workloads that keep large datasets in memory: in-memory databases, caching, analytics.

r5.4xlarge: 16 vCPUs, 128 GB memory (1:8 ratio)

GPU instances (p3, g4):

Include NVIDIA GPUs. For ML training, inference, graphics rendering.

p3.2xlarge: 8 vCPUs, 61 GB, 1× V100 GPU

Storage optimized (i3, d2):

High sequential I/O. For data warehousing, distributed filesystems.

How it works:

• Each t3 size has a baseline CPU utilization
• Below baseline, the instance accumulates credits
• Above baseline, the instance spends credits
• Exhausted credits throttle the instance to baseline

t3.micro can burst to 100% CPU, but sustained usage above 10% depletes credits.

Implications:

Good for:

• Variable workloads with idle periods
• Development and testing
• Small services with occasional spikes

Not good for:

• Sustained high CPU usage
• Predictable heavy computation
• Latency-sensitive services under load

For sustained workloads, m5 or c5 provide consistent performance without the credit system.

AWS-generated IDs

Example: i-0abcd1234efgh5678

IDs are immutable - an instance keeps its ID through stop/start cycles - and region-scoped.

User-defined names

Some resources take user-chosen names:

• S3 buckets: globally unique (my-company-data-2025)
• IAM users/roles: account-scoped (EC2-S3-Reader)
• Tags: key-value metadata on any resource

{
  "Name": "training-server",
  "Environment": "dev",
  "Project": "ml-pipeline"
}

Tags never affect behavior - they exist for organization, billing attribution, and automation (e.g., “terminate all instances tagged Environment=dev”).

Inbound rules specify what traffic can reach the instance:

Type        Port    Source
─────────────────────────────
SSH         22      0.0.0.0/0
HTTP        80      0.0.0.0/0
HTTPS       443     0.0.0.0/0
PostgreSQL  5432    10.0.0.0/16
Custom      8080    sg-0abc1234

Each rule: protocol, port range, source (CIDR or security group).

Default inbound: deny all

Outbound rules specify what traffic the instance can send:

Type        Port    Destination
─────────────────────────────────
All         All     0.0.0.0/0

Default outbound: allow all

Stateful behavior:

• An allowed inbound request (HTTP on port 80) gets its response out automatically - no outbound rule needed
• An allowed outbound request gets its response back in the same way

Stop preserves the instance: EBS volumes remain, the private IP is kept, and a later start may land on a different physical host.

Terminate destroys the instance permanently. There is no recovery.

Compute bills only while running; storage bills as long as volumes exist.

On-Demand

• Pay by the second, no commitment
• Full price, maximum flexibility
• Default for most workloads

Reserved / Savings Plans

• Commit to 1 or 3 years of usage
• Up to 72% discount (3-year, all upfront)
• Predictable workloads: web servers, databases
• Savings Plans: more flexible than Reserved Instances

Spot Instances

• Spare EC2 capacity at up to 90% discount
• AWS can reclaim with 2-minute warning
• Historically <5% of instances interrupted per month (varies by type)

Spot for ML training

• p3.2xlarge (1× V100): $3.06/hr on-demand, ~$1/hr spot
• Training is interruptible - checkpoint to S3, resume on a new instance

Characteristics:

• Network-attached, not a local disk
• Persists independently of instance lifecycle
• Detaches and reattaches to different instances
• Replicated within its AZ for durability

Volume types:

The AZ constraint:

Volumes attach only within their AZ - the data lives on drives in that facility. A volume in us-east-1a attaches only to instances in us-east-1a.

Snapshots:

Point-in-time backup stored in S3 (regionally). New volumes can be created from a snapshot in any AZ within the region.

The request arrives at S3:

PUT /my-bucket/results.json HTTP/1.1
Host: s3.us-east-1.amazonaws.com
Authorization: ???
Content-Type: application/json

{"accuracy": 0.94, ...}

S3 must establish:

1. Who sent this request?
2. May that identity write to this bucket?

Without proof of both, S3 rejects the request.

IAM provides both.

IAM User - a human operator

• Long-term credentials: password for the console, access keys for the API
• Created per person, account-scoped

IAM Role - an assumable identity

• No permanent credentials
• Trusted entities assume it and receive temporary credentials

AWS Service - services act as principals too

• ec2.amazonaws.com can appear as a principal, e.g. when EC2 assumes a role on an instance’s behalf

Root account - full access to everything

• Used for account setup and billing
• Day-to-day work happens as IAM identities, never as root

Long-term credentials:

• Access Key ID + Secret Access Key
• Created for IAM users
• Valid until explicitly revoked
• Stored in ~/.aws/credentials or environment variables

# ~/.aws/credentials
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFE...

Risk: a leaked key grants access until someone notices and revokes it.

Short-term credentials:

• Access Key ID + Secret Access Key + Session Token
• Obtained from STS (Security Token Service)
• Expire automatically (15 minutes to 12 hours)
• Not individually revocable - expiration is the mechanism

{
  "AccessKeyId": "ASIAXXX...",
  "SecretAccessKey": "xxx...",
  "SessionToken": "FwoGZX...",
  "Expiration": "2026-09-08T14:30:00Z"
}

Benefit: a leaked credential expires on its own.

Actions and resources take patterns:

s3:GetObject        # One operation
s3:*                # All S3 actions
ec2:Describe*       # All Describe actions
*                   # Everything (dangerous)

arn:aws:s3:::my-bucket/data/*   # One prefix
arn:aws:s3:::my-bucket/*        # All objects
arn:aws:s3:::*                  # All buckets
*                               # Everything

Broad patterns are rarely appropriate; the widest ones are dangerous.

Different actions need different resources:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"
    },
    {
      "Sid": "ReadWriteObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

s3:ListBucket authorizes against the bucket ARN; object operations authorize against /* - a distinction developed with S3.

Sid (statement ID) is optional - useful for documentation and debugging.

{
  "Statement": [
    {"Effect": "Allow", "Action": "s3:*",
     "Resource": "*"},
    {"Effect": "Deny", "Action": "s3:DeleteBucket",
     "Resource": "*"}
  ]
}

This allows all S3 actions except DeleteBucket. The deny wins.

Least privilege follows directly: start with nothing, add only what is needed.

Trust policy - who can assume this role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Service": "ec2.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }]
}

This says: the EC2 service can assume this role.

Permissions policy - what the role can do:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:PutObject"
    ],
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

This says: read and write objects in my-bucket.

Instance profile = container that holds an IAM role

When an instance launches with an instance profile:

1. The EC2 service assumes the role on the instance’s behalf
2. Temporary credentials are made available inside the instance
3. Credentials are served at the metadata endpoint (IMDSv2 token required)

http://169.254.169.254/latest/meta-data/iam/security-credentials/MyRole

The response is the expiring-credential JSON: AccessKeyId, SecretAccessKey, Token, Expiration.

SDKs handle this automatically.

Application code:

import boto3

s3 = boto3.client('s3')
s3.get_object(
    Bucket='my-bucket',
    Key='data.json'
)

boto3 queries the metadata endpoint, retrieves credentials, signs the request. No credentials in code.

Credentials refresh automatically before expiration.

Block storage (EBS):

• Raw blocks, OS manages the filesystem
• Attached to one instance
• POSIX semantics: open, read, write, seek, close
• Directories, permissions, links
• Mutable: change byte 1000 without touching the rest

The model local code assumes.

Object storage (S3):

• Key-value store over HTTP
• Accessible from anywhere
• HTTP semantics: PUT, GET, DELETE
• No directories, no permission bits
• Immutable: replace the entire object or nothing

A different model optimized for different access patterns.

No rename

Renaming old-name.csv to new-name.csv requires:

1. Copy the object to the new key
2. Delete the object at the old key

For a 5 GB file, that copies 5 GB (within S3, but still a copy).

Filesystems rename by changing a pointer. S3 has no pointers.

No move

Same as rename - copy, then delete.

No append

Adding 100 bytes to a 5 GB file requires:

1. Download 5 GB
2. Append 100 bytes locally
3. Upload 5.000000095 GB

Or: store separate objects and concatenate at read time.

Filesystems append by extending an allocation. S3 objects are immutable blobs.

No partial update

Changing byte 1000 requires replacing the entire object.

Durability: probability data survives

S3 Standard: 99.999999999% (11 nines)

S3 stores copies across multiple facilities in the region, designed to sustain simultaneous loss of two facilities.

10 million objects: expect to lose 1 every 10,000 years.

A successful PUT means the data is safe.

Availability: probability of successful access

S3 Standard: designed for 99.99%

About 53 minutes/year of potential unavailability.

Availability failures are transient - a retry succeeds. Data is not lost, only temporarily unreachable.

A GET may fail occasionally; the data is still there.

VPC properties:

• Isolated network within AWS
• The account defines the IP address range
• Spans all AZs in a region

Other accounts cannot see into a VPC. Traffic between VPCs is isolated by default.

Default VPC:

Every region has a default VPC created automatically. An instance launched without networking configuration goes there.

The default VPC suffices for simple deployments. Production environments typically use custom VPCs with deliberate network design.

These are private addresses - not routable on the public internet. Within the VPC, instances communicate with them directly.

Reaching the internet requires either:

• A public IP (mapped to the private IP)
• NAT (translates private to public)

Public subnet:

Route table includes:

Destination     Target
10.0.0.0/16     local
0.0.0.0/0       igw-xxx  ← Internet Gateway

Traffic to addresses outside the VPC routes to the Internet Gateway. Instances with public IPs can receive inbound traffic from the internet.

Private subnet:

Route table includes:

Destination     Target
10.0.0.0/16     local

No route to the internet. Traffic to external addresses has nowhere to go.

Instances here cannot be reached from the internet - no inbound path exists.

Auto-assigned public IP:

• Assigned at launch from AWS’s pool
• Released when the instance stops
• New IP on restart
• $0.005/hour (~$3.65/month)

aws ec2 run-instances \
    --associate-public-ip-address \
    ...

Elastic IP:

• Static public IP allocated to the account
• Persists until released
• Can move between instances
• Same $0.005/hour, attached or not

Needed when the address must stay fixed: DNS records, firewall whitelists.

Gateway Endpoints (S3, DynamoDB):

• A route table entry directs the traffic
• No NAT, no IGW for this traffic
• Free
• Traffic stays on the AWS network

Destination          Target
pl-xxx (S3 prefix)   vpce-xxx

Interface Endpoints (most other services):

• An ENI in the subnet with a private IP
• The service becomes reachable at that IP
• Hourly charge plus data processing
• Used by private subnets that call AWS APIs

Public subnet - internet-facing components

• Load balancer (receives user traffic)
• NAT gateway (enables private outbound)
• Bastion host if needed (SSH jump box)

Private subnet - internal components

• Application servers (behind the load balancer)
• Databases (no internet exposure)
• Outbound via NAT (updates, external APIs)

Computation beyond VMs:

• Lambda - event-driven functions without servers
• SQS/SNS - asynchronous communication between components

Persistence beyond objects:

• RDS - managed relational databases (PostgreSQL, MySQL)
• DynamoDB - managed key-value/document store

The model:

1. A function (Python, Node.js, etc.) is uploaded to Lambda
2. An event triggers it: an HTTP request, an S3 upload, a schedule
3. Lambda runs the function and bills for the execution time
4. Between invocations, nothing runs and nothing bills

No instances to manage. No servers to patch. Code runs, then nothing exists until the next trigger.

# Lambda function
def handler(event, context):
    # event contains trigger data
    # (HTTP request body, S3 event, etc.)

    name = event.get('name', 'World')

    return {
        'statusCode': 200,
        'body': f'Hello, {name}!'
    }

This function runs only when invoked. Between invocations, no charge accrues.

A queue holds messages between components.

• Producers put messages in
• Consumers take messages out and process them
• Producer and consumer never communicate directly

# Producer: send message
sqs.send_message(
    QueueUrl='https://sqs.../my-queue',
    MessageBody='{"task": "process", "id": 123}'
)

# Consumer: receive and process
messages = sqs.receive_message(QueueUrl='...')
for msg in messages.get('Messages', []):
    process(msg['Body'])
    sqs.delete_message(...)  # Acknowledge

Decoupling in time

• The producer sends and continues - it does not wait
• The consumer processes when ready
• A slow consumer means messages accumulate
• A crashed consumer means work waits instead of vanishing

Decoupling in scale

• One producer, many consumers
• Add consumers to process faster
• Remove consumers when load drops

Decoupling in deployment

• Producer and consumer deploy independently
• Different codebases, different release schedules
• The only agreement is the message format

Failure isolation

• A consumer crash does not affect the producer
• A producer crash does not lose queued work
• Failed messages retry without a new request

The concept:

A topic is a channel. Publishers send messages to topics; the topic delivers each message to its subscribers.

Subscriber types:

• Lambda functions
• SQS queues
• HTTP endpoints
• Email addresses
• SMS

What RDS is:

• An EC2 instance running database software
• Instance type from db.t3.micro to db.r5.24xlarge
• Storage from 20 GB to 64 TB
• Standard SQL - same queries, same libraries

import psycopg2

conn = psycopg2.connect(
    host='mydb.abc123.us-east-1.rds.amazonaws.com',
    database='myapp',
    user='admin',
    password='...'
)
# Standard PostgreSQL from here

Constraints to know:

• Single-writer: one primary instance handles writes
• Vertical scaling: more capacity means a bigger instance (requires restart)
• AZ-bound: the instance lives in one AZ (Multi-AZ adds a standby)
• Connection limits: instance size determines max connections

Access patterns:

• Get by key: O(1), single-digit milliseconds
• Query by partition: items sharing a partition key
• Scan: reads the entire table (expensive, avoid)

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Users')

# Write: ~5ms
table.put_item(Item={
    'user_id': 'u123',
    'name': 'Alice',
    'email': 'alice@example.com'
})

# Read by key: ~5ms
response = table.get_item(Key={'user_id': 'u123'})

Constraints to know:

• No joins: denormalize or make multiple requests
• No ad-hoc queries: the table design follows the access patterns
• Item size limit: 400 KB per item
• Query requires a partition key: querying arbitrary attributes is inefficient

Pricing (on-demand):

No idle cost: zero requests, zero charges.

RDS for:

• Complex queries with joins and aggregations
• ACID transactions across multiple tables
• Relational integrity (foreign keys, constraints)
• Ad-hoc queries and reporting
• Schema enforcement

The relational model fits when the queries keep changing.

DynamoDB for:

• Predictable single-digit-millisecond latency
• Very high request rates
• Key-based access patterns only
• Zero cost when idle
• Horizontal scaling without capacity planning

The key-value model fits when the access patterns are fixed in advance.

EBS: Capacity-based

Charges for provisioned size, not used space.

A 100 GB gp3 volume: $8/month

The charge is the same whether the volume holds 1 GB or 100 GB - the fee buys reserved capacity.

S3: Usage-based

Charges for actual storage plus operations.

100 GB stored: $2.30/month

S3 charges grow with the data actually stored. An empty bucket costs nothing.

Local training

data = pd.read_csv('dataset.csv')      # Local disk
model = train(data)                     # Local CPU/GPU
torch.save(model, 'model.pth')          # Local disk

Everything shares memory, disk, and failure fate. If the process crashes, everything stops together. If the disk fails, data and model are both gone.

One machine, one failure domain.

Distributed training

data = load_from_s3('bucket', 'data.csv')  # Network call
model = train(data)                         # EC2 instance
save_to_s3(model, 'bucket', 'model.pth')   # Network call

Components are networked. S3 can fail while EC2 runs. EC2 can terminate while S3 persists. The network can drop between them.

Data outlives compute. Compute is ephemeral. The network connects - and separates - them.

Local operations

Total per batch: ~20 ms. Predictable. Consistent.

With S3 in the loop

Total per batch: 65-165 ms. Variable. Depends on the network.

Checkpointing every batch adds 3-8× overhead. Checkpointing every 100 batches does not.

Independent failure domains

• S3: available (11 nines durability), but individual requests fail transiently
• EC2: the instance can terminate - spot reclamation, hardware failure, a bug
• Network: packets drop, connections time out, DNS fails
• Application code: exceptions, OOM, infinite loops

S3 keeps serving while an EC2 instance is down; EC2 keeps computing while an S3 request fails. The boundaries are the application’s to handle.

Failure scenarios

The pattern: S3 is durable, EC2 is ephemeral. Save state to S3 frequently enough that losing EC2 is recoverable.

Data flow pattern

• EC2 reads training data from S3
• EC2 processes (trains the model)
• EC2 writes checkpoints and the final model to S3

Why this works

• S3 is durable - data survives EC2 termination
• EC2 is ephemeral - no state lives only on the instance
• If EC2 terminates: launch a new instance, load the last checkpoint, continue

Recovery

No state is lost because state lives in S3, not EC2.

Without checkpoints

• Training runs for 10 hours
• The EC2 spot instance is reclaimed at hour 8
• Result: 8 hours of compute wasted, start over

With checkpoints every epoch

• Training runs for 10 hours (100 epochs)
• A checkpoint saves to S3 after each epoch
• EC2 reclaimed at hour 8 (epoch 80)
• A new instance loads the epoch-80 checkpoint
• Result: 20 minutes lost, not 8 hours

Checkpoint frequency trade-off

Choose based on:

• Training cost per hour (expensive means checkpoint more)
• S3 write latency tolerance
• Spot interruption frequency (2-minute warning)

For spot instances: handle the interruption signal - the 2-minute warning is enough to save a checkpoint.

The problem

# Upload model to S3
s3.put_object(Bucket='b', Key='model.pt', Body=data)
# Network timeout. Did it succeed?
# Is retrying safe if it did?

For S3 put_object: yes, safe to retry. The same key overwrites with the same content. No harm.

# Append to a log file?
# S3 has no append.
# Each put_object replaces the object -
# which is what makes retries safe.

Design for idempotency

Safe to retry:

• Writing a file to a deterministic path
• Overwriting a checkpoint with a newer version
• Reading data (no side effects)

Not safe to retry (without care):

• Incrementing a counter
• Appending to a log
• Sending a notification

Pattern: use deterministic keys

# Good: deterministic path
key = f"checkpoints/epoch_{epoch:04d}.pt"

# Bad: timestamp creates duplicates on retry
key = f"checkpoints/{datetime.now()}.pt"

Compute (EC2)

• A running instance bills whether or not it is working; a stopped one still bills for EBS

Design response: terminate when done. Use spot instances for fault-tolerant work. Leave nothing running overnight.

Storage (S3)

• Billed per GB-month; data persists until deleted

Design response: delete intermediate files. Use lifecycle policies for old checkpoints.

Data transfer

• Free into AWS and within an AZ; metered across AZs, across regions, and out to the internet

Design response: keep data and compute in the same region. Avoid repeatedly downloading large datasets from S3 to a local machine.

Example costs for a training job

Compute dominates. Optimize instance usage first.

Real student mistakes

Safe practices

• Use t3.micro for development and testing
• Use GPU instances only while actually training
• Terminate instances whose purpose is unclear

# Check what's running
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].[InstanceId,State.Name,InstanceType]' \
  --output table

Startup

1. The EC2 instance launches with an IAM role
2. The SDK discovers credentials via instance metadata
3. Download training data from S3
4. Download the latest checkpoint (if resuming)

Training loop

5. Train for N epochs
6. After each epoch: save a checkpoint to S3
7. On a spot interruption warning: save immediately, exit gracefully

Completion

8. Save the final model to S3
9. The instance terminates
10. The model persists in S3 for serving

What can fail and what happens

Every recovery in the table works only because the design provides:

• State in S3, not EC2
• Checkpoints frequent enough to resume from
• Idempotent operations

Download to local file

import boto3

s3 = boto3.client('s3')

# Download to local filesystem
s3.download_file(
    Bucket='my-bucket',
    Key='data/training.csv',
    Filename='/tmp/training.csv'
)

# Then read locally
import pandas as pd
df = pd.read_csv('/tmp/training.csv')

Use when:

• File is large (streaming would hold in memory)
• The file is read multiple times
• Library expects a file path

Stream directly into memory

import boto3
import pandas as pd
from io import BytesIO

s3 = boto3.client('s3')

# Get object returns a streaming body
response = s3.get_object(
    Bucket='my-bucket',
    Key='data/training.csv'
)

# Read directly into pandas
df = pd.read_csv(response['Body'])

Use when:

• File fits comfortably in memory
• Only one read is needed
• Disk I/O is unnecessary

Upload from file

import boto3

s3 = boto3.client('s3')

# Upload a local file
s3.upload_file(
    Filename='model.pt',
    Bucket='my-bucket',
    Key='models/experiment_001/final.pt'
)

upload_file switches to multipart upload above 8 MB (the TransferConfig default).

Upload with metadata

s3.upload_file(
    Filename='model.pt',
    Bucket='my-bucket',
    Key='models/final.pt',
    ExtraArgs={
        'Metadata': {
            'accuracy': '0.95',
            'epochs': '100'
        }
    }
)

Upload from memory

import boto3
from io import BytesIO
import torch

s3 = boto3.client('s3')

# Save model to memory buffer
buffer = BytesIO()
torch.save(model.state_dict(), buffer)
buffer.seek(0)  # Rewind to beginning

# Upload buffer contents
s3.put_object(
    Bucket='my-bucket',
    Key='models/final.pt',
    Body=buffer.getvalue()
)

Use when:

• Object is already in memory
• Writing to disk is unnecessary
• Object is small enough to hold in memory

List objects with a prefix

import boto3

s3 = boto3.client('s3')

response = s3.list_objects_v2(
    Bucket='my-bucket',
    Prefix='checkpoints/experiment_001/'
)

for obj in response.get('Contents', []):
    print(f"{obj['Key']}: {obj['Size']} bytes")

Output:

checkpoints/experiment_001/epoch_0010.pt: 45678 bytes
checkpoints/experiment_001/epoch_0020.pt: 45702 bytes
checkpoints/experiment_001/epoch_0030.pt: 45689 bytes

Pagination for many objects

list_objects_v2 returns max 1000 objects. For more:

paginator = s3.get_paginator('list_objects_v2')

for page in paginator.paginate(
    Bucket='my-bucket',
    Prefix='data/'
):
    for obj in page.get('Contents', []):
        print(obj['Key'])

Find latest checkpoint

response = s3.list_objects_v2(
    Bucket='my-bucket',
    Prefix='checkpoints/experiment_001/'
)

if response.get('Contents'):
    # Sort by key (works if zero-padded)
    latest = sorted(
        response['Contents'],
        key=lambda x: x['Key']
    )[-1]
    print(f"Latest: {latest['Key']}")

Common errors

from botocore.exceptions import ClientError

try:
    s3.download_file('bucket', 'key', 'local')
except ClientError as e:
    error_code = e.response['Error']['Code']

    if error_code == 'NoSuchKey':
        # Object doesn't exist
        print("File not found in S3")
    elif error_code == 'AccessDenied':
        # Permission issue
        print("Check IAM policy")
    elif error_code == '403':
        # Often bucket vs object ARN issue
        print("Check resource ARN in policy")
    else:
        raise

Retry with backoff

import time
from botocore.exceptions import ClientError

def download_with_retry(bucket, key, local, max_retries=3):
    for attempt in range(max_retries):
        try:
            s3.download_file(bucket, key, local)
            return  # Success
        except ClientError as e:
            error_code = e.response['Error']['Code']

            # Don't retry permanent errors
            if error_code in ['NoSuchKey', 'AccessDenied']:
                raise

            # Retry transient errors
            if attempt < max_retries - 1:
                wait = 2 ** attempt  # 1, 2, 4 seconds
                time.sleep(wait)
            else:
                raise

boto3 has built-in retry logic for some errors; explicit handling sets the policy.

Get token, then query

import requests

METADATA = 'http://169.254.169.254/latest/meta-data'

# Step 1: Get session token (required for IMDSv2)
token = requests.put(
    'http://169.254.169.254/latest/api/token',
    headers={'X-aws-ec2-metadata-token-ttl-seconds': '300'},
    timeout=1
).text

headers = {'X-aws-ec2-metadata-token': token}

# Step 2: Query with token
instance_id = requests.get(
    f'{METADATA}/instance-id', headers=headers, timeout=1
).text

region = requests.get(
    f'{METADATA}/placement/region', headers=headers, timeout=1
).text

instance_type = requests.get(
    f'{METADATA}/instance-type', headers=headers, timeout=1
).text

Why IMDSv2?

IMDSv1 allowed simple GET requests - vulnerable to SSRF attacks, where malicious input tricks the application into fetching credentials.

IMDSv2 requires:

• PUT request to get token (can’t be triggered via SSRF)
• Token in header for all subsequent requests
• Token expires (TTL set per request, 6 hours maximum)

New instances default to IMDSv2-only. Old code using direct GET requests will fail with 401 Unauthorized.

boto3 handles IMDSv2 automatically when fetching credentials; the token exchange matters only for direct metadata queries.

Check for interruption notice

import requests

def check_spot_interruption():
    """Returns termination time if spot will be interrupted."""
    try:
        token = requests.put(
            'http://169.254.169.254/latest/api/token',
            headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'},
            timeout=1
        ).text
        response = requests.get(
            'http://169.254.169.254/latest/meta-data/'
            'spot/instance-action',
            headers={'X-aws-ec2-metadata-token': token},
            timeout=1
        )
        if response.status_code == 200:
            data = response.json()
            return data.get('time')  # Termination time
    except requests.exceptions.RequestException:
        pass
    return None

Returns None normally. Returns timestamp when termination is imminent.

Graceful training loop

def train_with_interruption_handling(model, data):
    for epoch in range(num_epochs):
        # Check before each epoch
        if check_spot_interruption():
            print("Spot interruption! Saving checkpoint...")
            save_checkpoint(model, epoch)
            return "interrupted"

        # Train one epoch
        train_epoch(model, data)

        # Regular checkpoint
        if epoch % checkpoint_frequency == 0:
            save_checkpoint(model, epoch)

    return "completed"

The 2-minute warning is enough time to save a checkpoint.

Search order

1. Explicit in code (never do this)

   boto3.client('s3',
       aws_access_key_id='AKIA...',
       aws_secret_access_key='...')
   

2. Environment variables

   AWS_ACCESS_KEY_ID=AKIA...
   AWS_SECRET_ACCESS_KEY=...
   

3. Credentials file (~/.aws/credentials)

   [default]
   aws_access_key_id = AKIA...
   aws_secret_access_key = ...
   

4. Config file (~/.aws/config)

5. Instance metadata (EC2 role)

6. Container credentials (ECS/EKS)

Best practice by environment

Never in code. Keys in code get committed to git, leaked, compromised.

Verify the active identity

import boto3

sts = boto3.client('sts')
identity = sts.get_caller_identity()

print(f"Account: {identity['Account']}")
print(f"ARN: {identity['Arn']}")
# Shows whether a user or role is active

“Access Denied” on S3

botocore.exceptions.ClientError:
An error occurred (AccessDenied) when calling
the GetObject operation: Access Denied

Checklist:

1. Does the IAM policy grant the action? (s3:GetObject)
2. Does the resource ARN match? (bucket vs object)
3. Is there a bucket policy denying access?
4. Is the object in the right bucket/path?

The bucket vs object ARN trap

{
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::my-bucket"
}

Wrong. GetObject needs object ARN:

{
  "Resource": "arn:aws:s3:::my-bucket/*"
}

“Access Denied” on ListBucket

An error occurred (AccessDenied) when calling
the ListObjectsV2 operation: Access Denied

ListBucket needs bucket ARN:

{
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::my-bucket"
}

Not object ARN:

{
  "Resource": "arn:aws:s3:::my-bucket/*"
}

Complete policy for read/write

{
  "Statement": [
    {
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::bucket/*"
    },
    {
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::bucket"
    }
  ]
}

Check the active identity

import boto3

sts = boto3.client('sts')
print(sts.get_caller_identity())

Locally: the IAM user. On EC2: the instance role.

Different identities have different permissions.

Check what region

session = boto3.session.Session()
print(f"Region: {session.region_name}")

S3 buckets are regional. EC2 and bucket must agree; a mismatch adds transfer costs and latency.

Common causes

Verify instance role

# From the EC2 instance (IMDSv2: token first)
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Should return role name, e.g.:
# EC2-S3-Access-Role

An empty response means no role is attached.

S3 operations hang

Possible causes:

1. Security group blocks outbound traffic

   • EC2 needs outbound HTTPS (443) to reach S3
   • Check security group outbound rules

2. No internet gateway (private subnet)

   • Private subnets need NAT gateway or VPC endpoint for S3
   • Public subnets need internet gateway

3. DNS resolution failing

   • Test: nslookup s3.amazonaws.com

Quick network test

# Can we reach S3?
curl -I https://s3.us-east-1.amazonaws.com

# Can we resolve DNS?
nslookup s3.amazonaws.com

Set timeouts explicitly

from botocore.config import Config

config = Config(
    connect_timeout=5,
    read_timeout=30,
    retries={'max_attempts': 3}
)

s3 = boto3.client('s3', config=config)

Without explicit timeouts, operations can hang indefinitely.

VPC endpoint for S3

If in a private subnet without NAT:

VPC Endpoint (Gateway type) for S3
├── No NAT gateway needed
├── No internet gateway needed
├── Traffic stays in AWS network
└── Often faster and cheaper

Monitor memory usage

import psutil

def log_memory():
    mem = psutil.virtual_memory()
    print(f"Memory: {mem.used / 1e9:.1f}GB / "
          f"{mem.total / 1e9:.1f}GB "
          f"({mem.percent}%)")

# Call periodically during training
for epoch in range(num_epochs):
    log_memory()
    train_epoch(model, data)

From command line

# Current memory
free -h

# Watch continuously
watch -n 1 free -h

# Or use htop for interactive view
htop

Common causes

Reduce memory usage

# Store scalars, not tensors
losses = []
for batch in data:
    loss = train_step(batch)
    losses.append(loss.item())  # .item() not loss

# Clear GPU memory
torch.cuda.empty_cache()

# Use gradient checkpointing for large models
model.gradient_checkpointing_enable()

If still OOM: use a larger instance type, or redesign to process in smaller chunks.