ply.wtf
IT & InfrastructureStep-by-step guide

Expanding an EC2 disk without downtime: EBS, partition and filesystem on Ubuntu and Amazon Linux

13 min read
  • aws
  • ec2
  • ebs
  • ubuntu
  • amazon-linux
  • linux

Growing a disk on EC2 is three steps, and people reliably do only the first. You enlarge the EBS volume, AWS says “completed”, and df -h still shows the old size — because the volume grew underneath a partition and a filesystem that neither of them knows about.

All three steps run online. No reboot, no detach, no downtime.

The three layers

EBS volume        100 GiB   ← step 1: AWS console or CLI
└─ partition       50 GiB   ← step 2: growpart
   └─ filesystem   50 GiB   ← step 3: resize2fs (ext4) or xfs_growfs (XFS)

Each layer has to be told separately. Skip one and the space is there but unusable.

Where each step runs

This is the part that confuses people reading half a guide, so to be explicit:

Step Runs where
1 — grow the EBS volume AWS side. Web console in a browser, or aws CLI from your own machine
2 — grow the partition On the instance, over SSH
3 — grow the filesystem On the instance, over SSH

Step 1 changes what AWS provides. Steps 2 and 3 change what Linux does with it, and Linux is only reachable by logging into the machine. There is no way to do steps 2 and 3 from the console — nothing in the AWS interface can resize your filesystem for you.

Finding the volume ID

Every command in step 1 needs a volume ID, in the form vol-0123456789abcdef0. Four ways to get it, depending on where you are standing.

From the web console

EC2 → Instances → click your instance → Storage tab. The Block devices table lists every attached volume with its device name, size and Volume ID as a clickable link.

The device name column tells you which is the root volume: /dev/sda1 or /dev/xvda is root, anything else is a data volume.

From the CLI, by instance

If you know the instance but not the volume:

aws ec2 describe-volumes \
  --filters Name=attachment.instance-id,Values=i-0123456789abcdef0 \
  --query 'Volumes[].[VolumeId,Size,VolumeType,Attachments[0].Device]' \
  --output table

From the CLI, by name tag

If you tag your instances — and you should:

aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=my-app-server" \
  --query 'Reservations[].Instances[].BlockDeviceMappings[].[DeviceName,Ebs.VolumeId]' \
  --output table

From inside the instance

Useful when you are already on the box and do not know which instance you are on. First, the instance id from the metadata service — IMDSv2 needs a token, and on current AMIs IMDSv1 is often disabled:

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")

curl -sH "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id

Better still, on Nitro instances the NVMe serial number is the volume ID:

lsblk -o NAME,SIZE,SERIAL,MOUNTPOINTS
NAME          SIZE SERIAL                MOUNTPOINTS
nvme0n1       100G vol0123456789abcdef0
├─nvme0n1p1  49.9G                       /

Note the missing dash — AWS strips it. vol0123456789abcdef0 on the disk is vol-0123456789abcdef0 in the console. Amazon Linux ships a helper that formats it properly:

sudo /sbin/ebsnvme-id /dev/nvme0n1

This is the quickest way to answer “which volume is mounted on /data” without leaving the shell.

Before you start

Take a snapshot. Resizing is safe and well-trodden, but it touches the partition table of a live root volume, and a snapshot costs pennies:

aws ec2 create-snapshot \
  --volume-id vol-0123456789abcdef0 \
  --description "before resize $(date +%F)"

From the console, the same thing: EC2 → Volumes → select → Actions → Create snapshot.

Record what you have now, on the instance:

lsblk
df -hT

lsblk shows the device and partition sizes; df -hT shows the filesystem sizes and the filesystem type, which decides which command you need in step 3.

Step 1 — Grow the EBS volume

From the web console

  1. Open the EC2 console and make sure the region selector, top right, is on the region the instance lives in. A volume in another region simply will not appear in the list, and this is the single most common “my volume is missing” moment.
  2. Elastic Block Store → Volumes in the left sidebar.
  3. Find your volume. If the list is long, paste the volume ID into the search box, or filter by attachment.instance-id. The Attached resources column shows which instance each one belongs to.
  4. Select it, then Actions → Modify volume.
  5. In the dialog, set the new Size (GiB). This is also where you can change Volume type (gp2gp3) and, for gp3, set IOPS and Throughput independently of size.
  6. Click Modify, then confirm.

The volume’s State column now reads in-use - modifying, then in-use - optimizing, then back to plain in-use. To watch the detail: select the volume, open the Status check tab, or the Modifications tab which shows progress as a percentage.

You do not have to wait for it to finish. Once the state reaches optimizing, the new size is already visible to the instance and you can move on to step 2. Optimizing is AWS rebalancing the volume in the background, and it can take hours on a large disk.

From the CLI

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --size 100

Watch it:

aws ec2 describe-volumes-modifications \
  --volume-id vol-0123456789abcdef0 \
  --query 'VolumesModifications[0].[ModificationState,Progress,TargetSize]' \
  --output table

The state moves modifyingoptimizingcompleted. You can continue at optimizing — the new size is already visible to the instance, and optimizing is background work on AWS’s side.

While you are here, you may as well move an old gp2 volume to gp3, which is cheaper and lets you set IOPS and throughput independently of size:

aws ec2 modify-volume \
  --volume-id vol-0123456789abcdef0 \
  --size 100 --volume-type gp3 --iops 3000 --throughput 125

Two limits worth knowing before you plan around them:

  • You can only ever increase. EBS volumes cannot be shrunk. Making one smaller means creating a new volume, copying the data, and swapping it.
  • One modification per volume per 6 hours. Get the size right rather than creeping up in increments.

Connect to the instance

Everything from here runs on the instance, not on your machine and not in the AWS console. If you are still in a browser tab, this is where you leave it.

SSH

ssh -i ~/.ssh/my-key.pem ubuntu@203.0.113.42

The username depends on the AMI, and getting it wrong is the usual cause of Permission denied (publickey) on a perfectly good key:

AMI Username
Ubuntu (all releases) ubuntu
Amazon Linux 2 / 2023 ec2-user
Debian admin
RHEL ec2-user

The key file must not be readable by anyone else, or SSH refuses it outright:

chmod 400 ~/.ssh/my-key.pem

If you already keep a ~/.ssh/config, put the instance in it and forget the flags:

Host my-app
    HostName 203.0.113.42
    User ubuntu
    IdentityFile ~/.ssh/my-key.pem
    IdentitiesOnly yes

Then ssh my-app is the whole command. There is a fuller treatment of SSH keys and agents in the Git and GitHub guide — the mechanics are identical.

Connection refused or timing out? Almost always the security group: the instance needs an inbound rule allowing TCP 22 from your address. EC2 → Instances → select → Security tab click the security group → Edit inbound rules. Use My IP rather than 0.0.0.0/0.

Without SSH

Two alternatives, both worth knowing if the key has gone missing or port 22 is closed:

  • EC2 Instance Connect — in the console, select the instance and click Connect. It pushes a temporary key and opens a browser terminal. Still needs port 22 reachable, and works on Amazon Linux and recent Ubuntu AMIs.
  • Session ManagerConnect → Session Manager. No open port at all, no key, and it works on instances with no public IP. It needs the SSM agent running (preinstalled on Amazon Linux and Ubuntu AMIs) and an instance role with AmazonSSMManagedInstanceCore. On a production box this is the better answer: nothing to leak and every session logged in CloudTrail.

Once you have a shell — by whichever route — confirm you are on the right machine and that the new space has arrived:

lsblk

The disk should show the new size while the partition still shows the old one. If the disk is still the old size, step 1 has not landed yet: check the modification state and re-read lsblk.

Step 2 — Grow the partition

From here on, every command runs on the instance over SSH.

Find the device name

This is where the first surprise lands. On Nitro-based instances — anything current: M5, C5, T3, M6i, C7g and later — EBS volumes appear as NVMe devices:

/dev/nvme0n1        the disk
/dev/nvme0n1p1      the first partition

On older Xen instances it is /dev/xvda and /dev/xvda1. Check rather than assume:

lsblk

Typical output on an Ubuntu Nitro instance after the EBS volume was grown to 100 GiB:

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme0n1       259:0    0  100G  0 disk
├─nvme0n1p1   259:1    0 49.9G  0 part /
├─nvme0n1p14  259:2    0    4M  0 part
└─nvme0n1p15  259:3    0  106M  0 part /boot/efi

The disk is 100G, the root partition is still 49.9G. That gap is what step 2 closes.

About partitions 14 and 15: Ubuntu AMIs place a small BIOS boot partition and an EFI partition at the start of the disk, numbered 14 and 15. They look like they are in the way. They are not — partition 1 is still the last one physically, so it can grow into the free space. Ignore them.

growpart

# Ubuntu — usually already present
sudo apt update && sudo apt install -y cloud-guest-utils

# Amazon Linux 2 / 2023 — preinstalled, but just in case
sudo dnf install -y cloud-utils-growpart

Then:

sudo growpart /dev/nvme0n1 1

Note the space. The device and the partition number are two separate arguments. growpart /dev/nvme0n1p1 is wrong and gives a confusing error.

Confirm:

lsblk

The partition should now match the disk. The filesystem still will not.

If the volume has no partition table

Additional data volumes are frequently formatted directly, with no partition table at all. Then lsblk shows the disk with a mount point and no children:

nvme1n1       259:4    0  200G  0 disk /data

In that case skip growpart entirely and go straight to step 3, using the device itself. Running growpart on an unpartitioned device will only produce an error.

Step 3 — Grow the filesystem

Which command depends on the filesystem, not the distribution — but the defaults line up neatly:

AMI Default root filesystem Command
Ubuntu 20.04 / 22.04 / 24.04 ext4 resize2fs
Amazon Linux 2 xfs xfs_growfs
Amazon Linux 2023 xfs xfs_growfs

Check rather than trust:

df -hT /

ext4 — Ubuntu

Takes the device:

sudo resize2fs /dev/nvme0n1p1

XFS — Amazon Linux

Takes the mount point, not the device. This trips people up constantly:

sudo xfs_growfs -d /

For a data volume, pass its mount point:

sudo xfs_growfs -d /data

XFS can only ever grow. There is no shrink, at all, by design.

Verify

df -hT
lsblk

df should now show the new size. Nothing needs restarting — the filesystem grew under the running processes.

LVM

If the root or data volume is managed by LVM — common on custom AMIs and RHEL-derived images — there are two extra layers:

# 1. Grow the partition as above
sudo growpart /dev/nvme0n1 1

# 2. Tell the physical volume it got bigger
sudo pvresize /dev/nvme0n1p1

# 3. Extend the logical volume into the free space, and the filesystem with it
sudo lvextend -r -l +100%FREE /dev/mapper/vg0-root

The -r flag resizes the filesystem in the same command, choosing resize2fs or xfs_growfs for you. Check what you are working with first:

sudo pvs
sudo vgs
sudo lvs

Things that go wrong

growpart fails with unexpected output or failed [sfdisk].

Almost always a full or unwritable /tmp — growpart writes its working files there, and this failure mode is not signposted at all. Check and work around it:

df -h /tmp
sudo TMPDIR=/var/tmp growpart /dev/nvme0n1 1

NOCHANGE: partition 1 is size X. it cannot be grown.

The kernel has not picked up the new disk size yet. Either the EBS modification is still in modifying, or the rescan has not happened:

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0
sudo partprobe /dev/nvme0n1        # or: echo 1 | sudo tee /sys/class/block/nvme0n1/device/rescan
lsblk

Nothing works past 2 TiB.

An MBR partition table cannot address beyond 2 TiB, full stop. Check which you have:

sudo parted /dev/nvme0n1 print | grep "Partition Table"

msdos is MBR, gpt is GPT. Converting a live root volume from MBR to GPT is genuinely risky — if that is where you are, the sane path is to create a new GPT volume of the size you want, copy the data across, and swap. Current Ubuntu and Amazon Linux AMIs already use GPT.

resize2fs: Bad magic number in super-block.

You pointed it at the disk rather than the partition, or at an XFS filesystem. Re-read lsblk and df -hT.

df unchanged after xfs_growfs.

You passed a device where XFS wanted a mount point. Use /, not /dev/nvme0n1p1.

The instance will not boot after the resize.

This is what the snapshot was for. Detach the volume, attach it to another instance as a secondary disk, inspect and repair, reattach. Nothing in this procedure normally touches the bootloader — the usual cause is an unrelated /etc/fstab edit made in the same session.

The whole thing, condensed

# ---- on your own machine (or the web console) ----

# find the volume
aws ec2 describe-volumes \
  --filters Name=attachment.instance-id,Values=i-0123456789abcdef0 \
  --query 'Volumes[].[VolumeId,Size,Attachments[0].Device]' --output table

# snapshot, then grow it
aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 --description "before resize"
aws ec2 modify-volume  --volume-id vol-0123456789abcdef0 --size 100
aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0

# ---- now log in ----
ssh -i ~/.ssh/my-key.pem ubuntu@203.0.113.42     # ec2-user@ on Amazon Linux

# ---- on the instance ----

# 2 — partition
lsblk
sudo growpart /dev/nvme0n1 1

# 3 — filesystem
df -hT /
sudo resize2fs /dev/nvme0n1p1    # ext4  (Ubuntu)
sudo xfs_growfs -d /             # xfs   (Amazon Linux)

# verify
df -hT

Not having to do this again

Resizing by hand every few months is a smell. Two things help:

Alarm before it is urgent. Disk usage is not in CloudWatch by default — the hypervisor cannot see inside your filesystem. Install the CloudWatch agent and alarm on disk_used_percent at 80%.

Automate the OS side. growpart and resize2fs are idempotent: running them when there is nothing to grow is a no-op. A boot-time unit, or a line in user data, means enlarging the EBS volume is the only manual step from then on.

And the wider point: if a volume needs growing repeatedly, the question is usually not the volume. Logs that never rotate, Docker images that are never pruned, and backups written to the root volume account for most “we ran out of disk” incidents I have seen.

# Worth running before you decide you need more disk
sudo du -xh / | sort -rh | head -20
docker system df
journalctl --disk-usage

Comments

Corrections, additions and "this broke on my machine" reports are all welcome. You can post anonymously — no account needed.