Inside an MP4: the ISOBMFF box (atom) structure explained

Almost every video you watch (on your phone, on YouTube, in a group chat) is an MP4. It is one of the most common file types on Earth, yet most people have no idea what is inside one. The answer is surprisingly tidy: not a wall of pixels, but a tree of nested boxes. Learn to read that tree and you can open almost any modern media file and see exactly what is inside.

This is the first post in a series on the core media container formats. We start with the structure that quietly underpins most of them: the ISO Base Media File Format (ISOBMFF), the shared skeleton behind MP4, Apple's .mov, and even HEIF photos. Learn it once and you can read them all.

A file is a tree of boxes

Open an MP4 and you do not find a stream of pixels; you find boxes. The box is the one and only building block of the format, and it could not be simpler: every box starts with its size and its type, then carries a payload.

The size counts the whole box, header included, so a reader always knows where the next box begins. The type is a four-letter code (moov, trak, mdat) that says what the box is. Read the size, jump ahead by that many bytes, read the next box: that single rule walks the entire file.

Boxes nest. Some boxes are containers whose job is to hold other boxes; others are leaves that hold actual data. Some boxes do both: a field or two of their own, then more boxes nested inside. A container's children simply live inside its size span; there is no end marker, just sizes within sizes.

Box or atom? Apple's QuickTime, the format MP4 grew out of, called these units atoms. ISO renamed them boxes. You will see both words online; they mean exactly the same thing.

A real file, top to bottom

Take a real file with one video track and one audio track. Ignore the nesting for a moment: at the very top level it is just three boxes, one after another on disk.

The diagram shows moov ahead of mdat, but the order of those two actually varies from file to file; we will come back to why it matters.

• ftyp: a short stamp at the very start that declares which flavor of the format this is (plain MP4, QuickTime, HEIF, and so on).
• moov: the movie box, all of the metadata but not a single frame of actual video or audio.
• mdat: the media data box, the raw coded video and audio bytes and nothing else.

That split (moov holds the description, mdat holds the media) is the key to the whole format. The rest of this post is really about how the boxes inside moov describe the bytes inside mdat.

The questions the boxes answer

mdat is just bytes, so all the interesting structure is inside moov. Rather than dump the whole tree at once, we will walk down it one level at a time, and each box we open answers a practical question a player has to ask. Open moov and you find just two kinds of thing: one movie-wide header, mvhd, and one trak box per track (here, one video and one audio; play them together and you have a movie).

mvhd: the movie header and its clock

Start with the header. Here it is, decoded by the analyzer:

It is tiny, and it is almost all about time. A timescale of 1000 with a duration of 32744 means the movie runs 32744 / 1000 = 32.7 seconds; the timescale is simply the unit of time the track counts in, ticks per second, so durations stay exact without fractions. The creation_time keeps time too, and in a way worth pausing on: MP4 stores it as a plain count of seconds from midnight on 1 January 1904, a fixed zero point inherited from the original Macintosh clock. Here that count works out to 5 May 2023, when the file was made; a file that never sets the field leaves it at 0, which is why so many clips report a creation_time of exactly 1904.

Why 1904, and not 1900? Because 1904 is the first leap year of the century. Starting the count there let early date routines treat every fourth year as a leap year with no exceptions, sidestepping the awkward fact that 1900, divisible by 100 but not 400, is not one.

trak: one track

A track is one continuous stream: the video, or the audio. (A file can carry more: extra audio languages, subtitles.) A track's own metadata all lives here inside moov, never in mdat. Open a trak and its headers are right there: the track's ID and display size (tkhd), and one level deeper its own clock, the timescale (mdhd), and whether it carries video or audio (hdlr, which labels the track vide or soun). At the very bottom sits the stbl sample table.

stbl: the sample table

Now open stbl. This is the busiest box in the file, and it is what lets a player actually reach the media. It answers two questions about every sample (one unit of media: a single video frame, or a short slice of audio).

Where are a sample's bytes? In a set of small lookup tables: one records how big each sample is (stsz), one records which group, a chunk, each sample belongs to (stsc), and one records where each chunk begins in the file (stco). Combine them and the player gets an exact byte range to read out of mdat: no scanning, just arithmetic.

When does a sample play? One table (stts) lists how long each sample lasts; add those up and every sample has a precise decode time. A second, optional table (ctts) shifts the display time when frames are stored out of decode order, which modern video does on purpose because reordering frames compresses better. So each sample ends up with a precise place in time and a precise place in the file, all from tables, never by touching mdat.

Why the split is clever

Now the payoff. Because moov is a pure index and mdat is pure data, a player can jump to any moment instantly: look up the sample at, say, 2.4 seconds, read its byte range from the tables, fetch just those bytes. No searching the file. There is one wrinkle: video cannot start decoding from an arbitrary frame, so one more table (stss) marks the keyframes, the frames a decoder can start from cold, and a seek quietly begins at the nearest one.

Does moov come first or last?

The split also has a catch, and it shows up as the order of these two boxes. The standard allows either one first, and which you get is usually an accident of how the file was made. An encoder recording in one pass streams the media out as it captures it, but it cannot finish the index until the very last sample is written; only then does it know every sample's size and position. So the natural result is mdat first, with moov tacked on at the end. Our sample file has exactly this shape: look back at the mvhd screenshot and its file offset, 0x37b2fa0, sitting nearly at the end of the 58 MB file, because the whole moov lives behind the media.

For a file sitting on your disk this makes no difference: the player jumps straight to the end, reads moov, and plays. Over the network it matters a lot. A player cannot decode a single byte of mdat without the index, so if moov is at the end it has to reach the end of the file before anything can start. If the server cannot serve partial ranges, that means downloading the whole file first just to find the table of contents.

Moving moov to the front fixes this. It is a second pass over the finished file, the well-known faststart step (ffmpeg -movflags +faststart), that relocates moov ahead of mdat and rewrites the stored byte offsets so they still point at the right place. Now the player reads the index immediately and begins playback while the rest streams in. (Live streams sidestep the question entirely with a fragmented variant that interleaves many small index-and-data pairs; that is a topic for a later post.)

Look inside your own file

That is the whole format. A box has a size and a type; boxes nest; moov describes and mdat holds. With just that, you can open any MP4, MOV, or HEIF in a tool like the VTCLab Media Analyzer ↗ (opens in new tab) and read it at a glance: how many tracks, how long, video or audio, and whether it is laid out to stream.

Later posts in this series go deeper (the smaller header fields, the special cases real files throw at you, and the fragmented layout behind live streaming) before moving on to the other big containers, MPEG-TS and Matroska/WebM. And if you want to see where all this metadata gets used, Decoding media metadata ↗ (opens in new tab) is a good next read.