Parquet format versions

This page describes how features are added to the Parquet format specification and how they affect reader and writer compatibility. See the Implementation status page for which implementations (arrow, parquet-java, arrow-rs, etc.) support each feature.

Note: If you find out-of-date information, please open an issue or pull request.

Feature compatibility

The Parquet format spec classifies changes by their effect on reader and writer compatibility. Changes differ in their forward compatibility — whether an older reader can read files that use a newer feature.

Forward compatible features remain readable by older readers, with a possibly degraded experience: some metadata may be missing or performance may suffer, but the reader does not fail. Examples:

• Bloom filters: a reader that ignores them skips the pruning metadata but still reads the data correctly.
• Logical type annotations such as VARIANT: an older reader reads the underlying physical column (e.g. BYTE_ARRAY) as raw bytes without applying the logical type.

Forward incompatible features make the data unreadable to older software. Examples:

• New encodings (e.g. the DELTA_* encodings, BYTE_STREAM_SPLIT, RLE_DICTIONARY): a reader that does not implement them cannot decode the column values.
• Data Page V2 headers: a reader that only understands DataPageHeader cannot parse DataPageHeaderV2 pages.

FileMetadata version field

Each Parquet file has a version field in the thrift FileMetadata. This field has historically been used inconsistently: writers populate 1 or 2 without a consistent relationship to the features actually used. See the note in parquet.thrift and this discussion for details.

parquet-format release versions

The Thrift definition is released independently of implementations such as parquet-java or arrow-rs, following the Apache release process. This release version is not recorded in the FileMetaData. Note that release numbering DOES NOT FOLLOW semantic versioning: minor releases (e.g. 2.10.0 to 2.11.0) sometimes contain forward incompatible features.

Adding new features

New features are added by discussion and voting on the parquet dev mailing list (full process in the contributing guide). Once approved, a feature is added to the spec and ships in the next parquet-format release.

Forward incompatible features by version

Forward incompatible features and the format version each became available in. Columns include:

• Release: The parquet-format version the feature was released in.
• Source: Link to all changes in the parquet-format release.
• PRs: The specific pull requests that added the feature to the specification (features added before Parquet moved to Apache in 2014 have no entry, as the original pull request records no longer exist).
• Approved: The mailing list thread where the feature was approved.

Note: Files with an encrypted footer use different magic bytes (PARE instead of PAR1), making it clear to readers they must support modular encryption to read the file; plaintext footer files use PAR1 so legacy readers can still read their unencrypted columns.

Forward compatible additions

Older readers can read files that use these features but may not understand the new information.