The Open Data Stack: Parquet, Iceberg, Polaris, Ossie

Something happened in the data infrastructure world between November 2025 and July 2026 that nobody has drawn clearly yet.

For the first time, you can build an open data stack from raw bytes on object storage all the way up to governed business semantics using only Apache-governed, vendor-neutral standards. Not a proprietary warehouse. Not a closed catalog. Not a BI vendor's modeling layer. The entire stack, from how your data is stored on disk to what that data means to a business analyst or an AI agent, now has an open specification at every layer.

The four projects that make this possible are Apache Parquet, Apache Iceberg, Apache Polaris, and Apache Ossie. Each one solves a different layer of the same problem. Together they compose into something the data industry has been trying to build for two decades: a complete, open, vendor-neutral data foundation.

Let me walk through what each layer does, what problem it solved when it arrived, how the layers connect, and what it means to run this open data stack in practice.

Layer 1: Apache Parquet — How Data Is Stored

Apache Parquet is a columnar file format. It defines how your data is written to disk.

Before Parquet, the dominant formats were row-oriented: CSV, JSON, Avro. Those formats are natural for writing records one at a time. They are terrible for analytical queries that need to read one column across billions of rows. To calculate total revenue across a hundred million order records, a row-oriented format forces you to read every field in every row even though you only need the amount column. Parquet inverts that. Each column is stored together, compressed together, and read together. The query only touches what it needs.

Parquet was developed at Twitter and Cloudera and became an Apache Top-Level Project in 2013. What happened next followed the same pattern that has played out at every layer since: once the format was open and governed by a neutral foundation, every engine adopted it. Spark, Trino, Dremio, Snowflake, BigQuery, DuckDB, and dozens of others all read and write Parquet. The format stopped being a competitive advantage for any single vendor and became infrastructure everyone builds on.

Parquet does not know what a table is. It knows how to store columns efficiently. That is its job. Anything above that, schema evolution, partitioning, time travel, transaction management, is not Parquet's concern.

Layer 2: Apache Iceberg — How Tables Are Structured

Apache Iceberg is an open table format. It sits on top of Parquet and turns a pile of columnar files into something that behaves like a database table.

The problem Iceberg solved is the problem that breaks data lakes in practice. A data lake stores files in object storage. Object storage is cheap and scalable. But it has no concept of a transaction, no concept of a schema, no concept of "current state" versus "old state." When you run a pipeline that overwrites data in S3, there is a window of time when reads from that bucket are undefined. When your schema changes, old files break new queries. When you want to roll back to yesterday's data, you have no mechanism to do it.

Iceberg provides a metadata layer on top of Parquet files that gives you all of this. Every write creates a new snapshot. The snapshot contains a manifest of which files represent the current state of the table. Schema changes are recorded in the metadata without rewriting existing files. Partition strategies can evolve without breaking historical data. Any query engine that understands the Iceberg spec can read the snapshot metadata and know exactly which Parquet files to scan, in what order, with what schema.

Iceberg was developed at Netflix and became an Apache Top-Level Project in 2020. The same dynamics as Parquet played out: once the spec was open and neutral, the format became universal. The table format competition between Iceberg, Delta Lake, and Hudi has effectively converged on Iceberg as the shared standard. Snowflake, Databricks, AWS, Google, and Microsoft all support it.

But Iceberg is still just a spec and a set of file metadata. It does not run a server. It does not manage access control. It does not tell other engines where your tables are or who is allowed to read them. That requires a catalog.

Layer 3: Apache Polaris — How the Catalog Is Governed

Apache Polaris is an open catalog for Apache Iceberg. It implements the Iceberg REST catalog API and acts as the central service that manages metadata, governs access, and serves table locations to any compatible query engine.

The problem Polaris solved is one that was growing invisible in the background of every data platform conversation. Iceberg made the table format open. But if the catalog tracking which Iceberg files are the current version of each table lives inside a proprietary system, the openness is partially an illusion. A Snowflake-managed catalog, an AWS Glue catalog, a Databricks Unity Catalog: each of these does the catalog job well. Each also creates a dependency on that vendor's ecosystem. Query engines from other vendors work less smoothly. Migration between platforms becomes a catalog migration problem, not just a file format problem.

Polaris was co-created by Snowflake and Dremio, donated to the Apache Software Foundation in August 2024, and graduated from incubation to a Top-Level Project in February 2026. That graduation matters: Polaris is not incubating anymore. It is production-ready, community-governed, and already running in production at organizations that want a catalog that belongs to no single vendor.

What Polaris actually does in the open data stack: any engine that speaks the Iceberg REST catalog protocol can connect to Polaris and ask, "where is the current version of the orders table, and am I allowed to read it?" Polaris answers both questions. It tracks the snapshot metadata, enforces role-based access control, and vends temporary storage credentials to the engine so it can read the underlying Parquet files directly from object storage without the engine ever holding your cloud storage keys permanently.

Spark can write a table. Trino can query it. Flink can stream records into it. DuckDB can read it for local analysis. All of them see the same table, with the same schema, governed by the same policies, because Polaris is the single source of truth for what the table is and who can see it.

Polaris does not know what the table means to the business. It knows where the table is, what version is current, and who has permission to access it. What the table means, what "revenue" represents in your business context, what filters apply, what the relationship is to the customers table, that requires a fourth layer.

Layer 4: Apache Ossie — What Data Means

Apache Ossie (incubating) is the open semantic layer standard. It sits on top of the catalog and defines the business meaning of your data: the metric definitions, the dimensional relationships, the governed vocabulary that turns technical table columns into trusted business concepts.

The problem Ossie solves is the one that breaks AI and BI in practice. Your Polaris catalog knows that a table called orders exists with a column called amount. It does not know that amount should be summed after filtering out refunds and internal test orders, and that the result should be called "Net Revenue" rather than "Amount" when displayed in dashboards or consumed by AI agents. That business logic lives somewhere, but where it lives differs across every organization and usually differs across every tool within a single organization.

Ossie defines a YAML-based specification for expressing that meaning once, in a format any tool can read. A semantic model in Ossie contains the dataset definition (orders, joined to customers), the dimensions (order_date, region, product_category), the metrics (total_revenue = SUM of amount with the appropriate filters), and an ai_context block with instructions for how AI agents should interpret the model. That definition is portable. Any BI tool, any query engine, any AI agent that implements the Ossie spec reads the same semantic model and produces the same answer.

The relationship between Ossie and Polaris is by design. Ossie defines the format of semantic models. Polaris is building toward being the governed catalog home where those models live and are served. A converter between Ossie and Polaris already exists in the apache/ossie repository. When the integration matures, a data engineer defines a metric in Ossie YAML, the model is stored in Polaris alongside the tables it describes, and any consuming tool retrieves both the table location and the semantic definition through the same catalog API call.

Ossie was accepted into the Apache Incubator in July 2026, nine months after the repository opened under the name Open Semantic Interchange. More than 50 organizations have joined the working groups, including Snowflake, Databricks, dbt Labs, Salesforce, Oracle, Cloudera, and BlackRock. We covered the full coalition and what each organization brings in an earlier guide.

What the Composed Stack Actually Gives You

Put the four layers of this open data stack together and the picture becomes concrete.

Your data lives in Parquet files on object storage. S3, GCS, or Azure Blob Storage. You control it. No vendor charges you a proprietary storage fee. No vendor can hold your data hostage by locking the format.

Apache Iceberg gives those files table structure. Schema changes happen without rewriting files. You can query your data as it existed three months ago. Multiple engines write to the same tables concurrently without corrupting each other.

Apache Polaris governs access. Every engine connects to the same catalog. Permissions are defined once and enforced everywhere. A data scientist using Spark sees the same tables under the same access rules as an analyst using Trino or a BI tool querying over the REST API. Credential vending means no engine gets permanent cloud storage credentials. Access is scoped, auditable, and centrally managed.

Apache Ossie defines what the tables mean. The metric definitions, the dimension hierarchies, the AI context instructions. An AI agent querying the orders table through this stack does not have to infer what "revenue" means from the column names. It reads the Ossie semantic model and knows: revenue means this calculation, with these filters, at this grain, calculated this way. The agent's answer matches the CFO's dashboard because both consumed the same definition from the same source.

No single vendor owns any of these four layers. Every specification is Apache-licensed. The governance of each layer runs through the Apache Software Foundation. The roadmap of each project is decided by public vote, not by a product management team with a commercial agenda.

The Pattern That Keeps Repeating

These four layers did not arrive simultaneously. They followed a sequence driven by the same market dynamic each time.

Parquet arrived when the row-versus-column format debate was making analytical queries on large datasets unnecessarily expensive. The open format resolved the debate and moved competition up: vendors now compete on query engine performance, not file format.

Iceberg arrived when table format fragmentation was threatening to balkanize the data lake. The open table format resolved the fragmentation and moved competition up: vendors now compete on catalog features and query optimization, not who owns your snapshot metadata.

Polaris arrived when proprietary catalogs were quietly re-creating the vendor lock-in that open table formats had removed. The open catalog resolved that layer and moved competition up: vendors now compete on query performance, managed operations, and developer experience, not who controls your table metadata.

Ossie is arriving because semantic fragmentation is the new battleground. Every BI vendor, every semantic layer tool, every data warehouse has its own proprietary format for expressing business logic. LookML, MetricFlow YAML, Snowflake Semantic Views, Cube schema, AtScale SML: all of these solve the same problem with incompatible formats. Ossie is the open format that, if it achieves the same adoption trajectory as Parquet and Iceberg, resolves that fragmentation and moves competition up again.

The arc is one sentence, repeated at every layer of the open data stack: the community standardizes the format, no vendor owns it, competition moves to the next layer up, users win because their data and logic are portable.

What This Means for AI Agents Right Now

The reason this four-layer open data stack matters most in 2026 is AI agents.

An AI agent querying enterprise data without this stack has to figure out, from scratch, where the data lives, whether it has permission to access it, what the current version is, and what the business terms mean. Every one of those questions is a potential failure point. The agent may query stale data, access data it should not see, or interpret "revenue" using a definition that contradicts the finance team's official number.

With the full Apache stack, three of those four problems have open solutions. Polaris tells the agent where the data is and verifies it has access. Iceberg ensures it reads the current snapshot. Ossie tells it what the data means in business terms, with AI context instructions embedded directly in the semantic model.

The architecture the agentic AI use case has been demanding is now expressible entirely in open standards. An agent reaches the lakehouse through a governed interface, the Polaris catalog serves it scoped credentials and the Iceberg table location, the Ossie semantic model tells it what to calculate and how, and the answer it returns can be traced back to a definition that your data team owns and can audit.

That is not theoretical. The converters exist today. The working groups are active. The pieces are composable now.

The Gaps That Remain

The open data stack is not complete in every sense. A few gaps are worth naming honestly.

Ossie is still incubating. The specification is v0.1. The right posture for most teams in 2026 is to align and experiment, not to migrate production semantic models. The Parquet and Iceberg precedent suggests that incubating projects with this level of ecosystem momentum graduate into production-ready standards relatively quickly, but the timeline is the community's to determine.

The query execution layer is not standardized at this level. Parquet, Iceberg, Polaris, and Ossie define how data is stored, structured, catalogued, and interpreted. They do not define how queries are executed. You still choose an engine: Spark, Trino, Flink, DuckDB, Dremio, or something else. That is where vendors still meaningfully differentiate on performance and developer experience.

Microsoft is still outside the Ossie coalition. The Microsoft semantic model, which powers Power BI and is the most widely deployed semantic layer in enterprise BI, is not participating in the standard. Until that gap closes or Ossie proves its value without it, a significant portion of enterprise semantic workloads remain in a proprietary format.

Start at Layer One

I have been writing for two years that companies fail at AI because they allocate intelligence to the wrong layer of the architecture. They invest in the AI layer before building the semantic layer. They build the semantic layer on top of a poorly governed data foundation. They add data foundation tooling on top of storage that nobody controls consistently.

The Apache open data stack is the expression of what it looks like to start at Layer 1 and build up. Parquet is the floor. Iceberg is the structure. Polaris is the governance. Ossie is the meaning. Only then do you deploy the AI agents on top.

For every dollar you spend on the AI layer, six should go to the layers underneath it. The layers underneath now have an open specification at every level. The excuse for building on proprietary formats is getting harder to sustain, one Apache graduation at a time.