Parquet and Feather Files, Explained

After reading this you will know how Parquet and Feather store your data on disk, why a 2 GB file can preview in a fraction of a second, and how to read the schema, row count and metadata without touching Python or Spark.

What these formats are, and one quick look

Parquet and Feather are binary file formats for tables. A CSV file stores your data as text, row by row, with commas between cells. Parquet and Feather store the same table as compact bytes, grouped by column, with the schema (column names and types) written into the file itself.

Here is the hook. Suppose you have a Parquet file with 50 million rows and 40 columns, and you only want to know what the columns are called and see the first 20 rows. With CSV you would stream a lot of text. With Parquet the tool reads a small footer that lists the schema and the layout, then decodes only the first row group. The rest of the file stays untouched on disk.

The preview tool opens both formats in the browser. It shows the schema with types, the total row count, any key/value metadata, and the first rows, and it exports that preview to CSV. Nothing is uploaded.

How the bytes are arranged

The single most useful idea is columnar storage. Take a tiny table of three rows:

A CSV writes this in row order: 1,Paris,12.5 then 2,Paris,7.0 and so on. Parquet writes it in column order: all three id values together, then all three city values, then all three amount values.

Two consequences follow. First, a column of similar values compresses well. The city column is Paris, Paris, Berlin, which a dictionary can encode as 0, 0, 1 plus a lookup table of two strings. Second, a reader that wants only id and amount can skip the city bytes entirely.

Parquet groups rows into row groups (often 100,000 to 1,000,000 rows each), and inside each row group it stores each column as a column chunk made of pages. A footer at the end of the file records where every chunk lives and how many rows each group holds. Feather (the Arrow IPC format) uses a similar column layout but is tuned for fast read and write rather than maximum compression, so it is common for temporary files and interprocess handoff.

Why the preview is fast even on huge files

The speed comes from reading only what you asked for. To show a schema and 20 rows, the tool does roughly this:

1. Read the last few kilobytes of the file to find the footer.
2. Parse the footer to get column names, types, total row count, and the byte offset of the first row group.
3. Read and decode just that first row group, then stop at the first 20 rows.

Put numbers on it. A file with 50,000,000 rows split into row groups of 1,000,000 rows has 50 row groups. Previewing the first 20 rows touches 1 row group, which is 1/50 = 2\% of the data pages plus a small footer. If each row group is about 40 MB, the tool reads roughly 40 MB out of a 2 GB file to answer your question.

The size math, with intuition

You can estimate a Parquet file's size from its columns. For a column with N values, the stored size is roughly the raw size divided by a compression factor:

Here N is the number of rows, b is the average bytes per value before compression, and r is the compression ratio for that column. A column of low-cardinality strings like country codes might have b = 3 and r = 20 because dictionary encoding replaces repeated strings with small integers. A column of random 64-bit floats has b = 8 and r \approx 1 because random bits do not compress.

The intuition: repetition and sorting raise r. If you sort your rows by a column before writing, runs of identical values in that column get longer, and run-length or dictionary encoding shrinks it further. A random column gains nothing from sorting.

Reading the schema and types correctly

The schema is where most confusion starts, so read the type names literally. A few pairings matter:

Physical vs logical type

Parquet stores a physical type (like INT32) and often a logical type (like DATE or DECIMAL) on top. A date is physically an integer counting days from 1970-01-01. The value 19723 is 2024-01-01, not the year 19723.

Timestamp units

Timestamps carry a unit: milliseconds, microseconds or nanoseconds. A microsecond timestamp of 1704067200000000 is 2024-01-01 00:00:00 UTC. Read the wrong unit and you land 1000 times too early or too late.

Nullability

Most Parquet and Arrow columns allow nulls, tracked separately from the values. An empty preview cell can mean null, not an empty string. Those are different, and CSV cannot always tell them apart.

Nested types

Parquet supports lists and structs. A column of type list<int64> holds an array per row. The preview flattens or stringifies these for display, so treat the CSV export of a nested column as a rendering, not the exact structure.

Common mistakes

Three more traps come up often. First, treating the row count as expensive to get: it is read from the footer, so it is instant regardless of file size. Second, assuming Feather and Parquet are interchangeable: Feather (Arrow IPC) usually skips heavy compression and is larger on disk but faster to read, while Parquet trades some read speed for smaller files. Third, reading timestamps and dates as raw integers because you ignored the logical type. Always check the type column before you interpret a number.

A subtler one: dictionary encoding does not guarantee small files. If a string column has as many distinct values as rows (say a unique ID per row), the dictionary is as large as the data, so the ratio r falls toward 1. The widget above shows this: push distinct values up to the row count and the bar stops shrinking.

Related tools

Once you have previewed a file and understood its schema, you often want to work with the data as CSV. Convert your preview or a small extract, then open it in the CSV Viewer & Editor to sort and filter, or send it through the CSV ↔ JSON Converter if a program downstream expects JSON. If your source was a spreadsheet, the Excel → CSV Converter turns each sheet into clean CSV first. When a Parquet column holds names, emails or IDs you must not share, run the CSV through the Data Anonymizer before you pass it on.

Related tools

Auto Analyst: Instant Statistical Report

Drop a CSV or Excel file and get a statistician’s report: data problems, distributions, every real relationship, key drivers and Simpson’s paradox warnings, in plain English.

CSV Viewer & Editor

View, edit, sort, filter and extract columns from CSV files.

CSV ↔ JSON Converter

Convert CSV to JSON and JSON back to CSV.

Data Visualizer

29 chart types — heatmap, treemap, dumbbell, slope, candlestick — with rich styling and annotations.

Excel → CSV Converter

Convert XLSX/XLS sheets to clean CSV, per sheet.

JSON Table Viewer

Paste JSON, get a sortable, searchable table.