Write-Audit-Publish for Knowledge Lakes in Pure Python (no JVM) | by Ciro Greco | Apr, 2024

On this weblog submit we offer a no-nonsense, reference implementation for Write-Audit-Publish (WAP) patterns on an information lake, utilizing Apache Iceberg as an open desk format, and Challenge Nessie as an information catalog supporting git-like semantics.

We selected Nessie as a result of its branching capabilities present an excellent abstraction to implement a WAP design. Most significantly, we selected to construct on PyIceberg to remove the necessity for the JVM when it comes to developer expertise. In truth, to run all the challenge, together with the built-in purposes we’ll solely want Python and AWS.

Whereas Nessie is technically in-built Java, the information catalog is run as a container by AWS Lightsail on this challenge, we’re going to work together with it solely by means of its endpoint. Consequently, we are able to specific all the WAP logic, together with the queries downstream, in Python solely!

As a result of PyIceberg is pretty new, a bunch of issues are literally not supported out of the field. Particularly, writing continues to be in early days, and branching Iceberg tables continues to be not supported. So what you’ll discover right here is the results of some unique work we did ourselves to make branching Iceberg tables in Nessie attainable straight from Python.

So all this occurred, kind of.

Again in 2017, Michelle Winters from Netflix talked a few design sample known as Write-Audit-Publish (WAP) in knowledge. Basically, WAP is a practical design geared toward making knowledge high quality checks simple to implement earlier than the information turn out to be obtainable to downstream shoppers.

For example, an atypical use case is knowledge high quality at ingestion. The stream will seem like making a staging setting and run high quality checks on freshly ingested knowledge, earlier than making that knowledge obtainable to any downstream software.

Because the title betrays, there are primarily three phases:

1. Write. Put the information in a location that’s not accessible to shoppers downstream (e.g. a staging setting or a department).
2. Audit. Rework and take a look at the information to verify it meets the specs (e.g. examine whether or not the schema abruptly modified, or whether or not there are surprising values, equivalent to NULLs).
3. Publish. Put the information within the place the place shoppers can learn it from (e.g. the manufacturing knowledge lake).

This is just one instance of the attainable purposes of WAP patterns. It’s simple to see how it may be utilized at totally different levels of the information life-cycle, from ETL and knowledge ingestion, to advanced knowledge pipelines supporting analytics and ML purposes.

Regardless of being so helpful, WAP continues to be not very widespread, and solely not too long ago corporations have began eager about it extra systematically. The rise of open desk codecs and tasks like Nessie and LakeFS is accelerating the method, but it surely nonetheless a bit avant garde.

In any case, it’s a superb mind-set about knowledge and this can be very helpful in taming a few of the most widespread issues maintaining engineers up at evening. So let’s see how we are able to implement it.

We aren’t going to have a theoretical dialogue about WAP nor will we offer an exhaustive survey of the alternative ways to implement it (Alex Merced from Dremio and Einat Orr from LakeFs are already doing an exceptional job at that). As an alternative, we’ll present a reference implementation for WAP on an information lake.

👉 So buckle up, clone the Repo, and provides it a spin!

📌 For extra particulars, please consult with the README of the challenge.

The thought right here is to simulate an ingestion workflow and implement a WAP sample by branching the information lake and operating an information high quality take a look at earlier than deciding whether or not to place the information into the ultimate desk into the information lake.

We use Nessie branching capabilities to get our sandboxed setting the place knowledge can’t be learn by downstream shoppers and AWS Lambda to run the WAP logic.

Basically, every time a brand new parquet file is uploaded, a Lambda will go up, create a department within the knowledge catalog and append the information into an Iceberg desk. Then, a easy a easy knowledge high quality take a look at is carried out with PyIceberg to examine whether or not a sure column within the desk incorporates some NULL values.

If the reply is sure, the information high quality take a look at fails. The brand new department is not going to be merged into the principle department of the information catalog, making the information not possible to be learn in the principle department of information lake. As an alternative, an alert message goes to be despatched to Slack.

If the reply is not any, and the information doesn’t include any NULLs, the information high quality take a look at is handed. The brand new department will thus be merged into the fundamental department of the information catalog and the information can be appended within the Iceberg desk within the knowledge lake for different processes to learn.

All knowledge is totally artificial and is generated mechanically by merely operating the challenge. After all, we offer the potential of selecting whether or not to generate knowledge that complies with the information high quality specs or to generate knowledge that embody some NULL values.

To implement the entire end-to-end stream, we’re going to use the next parts:

This challenge is fairly self-contained and comes with scripts to arrange all the infrastructure, so it requires solely introductory-level familiarity with AWS and Python.

It’s additionally not supposed to be a production-ready answer, however relatively a reference implementation, a place to begin for extra advanced eventualities: the code is verbose and closely commented, making it simple to switch and lengthen the fundamental ideas to higher go well with anybody’s use circumstances.

To visualise the outcomes of the information high quality take a look at, we offer a quite simple Streamlit app that can be utilized to see what occurs when some new knowledge is uploaded to first location on S3 — the one that’s not obtainable to downstream shoppers.

We are able to use the app to examine what number of rows are within the desk throughout the totally different branches, and for the branches aside from fundamental, it’s simple to see in what column the information high quality take a look at failed and in what number of rows.

As soon as we now have a WAP stream based mostly on Iceberg, we are able to leverage it to implement a composable design for our downstream shoppers. In our repo we offer directions for a Snowflake integration as a method to discover this architectural risk.

This is likely one of the fundamental tenet of the Lakehouse structure, conceived to be extra versatile than trendy knowledge warehouses and extra usable than conventional knowledge lakes.

On the one hand, the Lakehouse hinges on leveraging object retailer to remove knowledge redundancy and on the similar time decrease storage value. On the opposite, it’s supposed to supply extra flexibility in selecting totally different compute engines for various functions.

All this sounds very fascinating in principle, but it surely additionally sounds very difficult to engineer at scale. Even a easy integration between Snowflake and an S3 bucket as an exterior quantity is frankly fairly tedious.

And actually, we can not stress this sufficient, shifting to a full Lakehouse structure is a whole lot of work. Like rather a lot!

Having stated that, even a journey of a thousand miles begins with a single step, so why don’t we begin by reaching out the bottom hanging fruits with easy however very tangible sensible penalties?

The instance within the repo showcases considered one of these easy use case: WAP and knowledge high quality checks. The WAP sample here’s a likelihood to maneuver the computation required for knowledge high quality checks (and probably for some ingestion ETL) exterior the information warehouse, whereas nonetheless sustaining the potential of making the most of Snowflake for extra excessive worth analyitcs workloads on licensed artifacts. We hope that this submit might help builders to construct their very own proof of ideas and use the

The reference implementation right here proposed has a number of benefits:

Tables are higher than recordsdata

Knowledge lakes are traditionally exhausting to develop in opposition to, because the knowledge abstractions are very totally different from these usually adopted in good outdated databases. Massive Knowledge frameworks like Spark first offered the capabilities to course of massive quantities of uncooked knowledge saved as recordsdata in several codecs (e.g. parquet, csv, and many others), however individuals typically don’t assume when it comes to recordsdata: they assume it phrases of tables.

We use an open desk format because of this. Iceberg turns the principle knowledge lake abstraction into tables relatively than recordsdata which makes issues significantly extra intuitive. We are able to now use SQL question engines natively to discover the information and we are able to rely on Iceberg to deal with offering appropriate schema evolution.

Interoperability is sweet for ya

Iceberg additionally permits for higher interoperability from an architectural standpoint. One of many fundamental advantages of utilizing open desk codecs is that knowledge may be stored in object retailer whereas high-performance SQL engines (Spark, Trino, Dremio) and Warehouses (Snowflake, Redshift) can be utilized to question it. The truth that Iceberg is supported by the vast majority of computational engines on the market has profound penalties for the way in which we are able to architect our knowledge platform.

As described above, our advised integration with Snowflake is supposed to point out that one can intentionally transfer the computation wanted for the ingestion ETL and the information high quality checks exterior of the Warehouse, and preserve the the latter for giant scale analytics jobs and final mile querying that require excessive efficiency. At scale, this concept can translate into considerably decrease prices.

Branches are helpful abstractions

WAP sample requires a method to write knowledge in a location the place shoppers can not by accident learn it. Branching semantics naturally gives a method to implement this, which is why we use Nessie to leverage branching semantics on the knowledge catalog stage. Nessie builds on Iceberg and on its time journey and desk branching functionalities. Lots of the work achieved in our repo is to make Nessie work straight with Python. The result’s that one can work together with the Nessie catalog and write Iceberg tables in several branches of the information catalog with out a JVM based mostly course of to put in writing.

Easier developer expertise

Lastly, making the end-to-end expertise utterly Python-based simplifies remarkably the arrange fo the system and the interplay with it. Some other system we’re conscious of would require a JVM or an extra hosted service to put in writing again into Iceberg tables into totally different branches, whereas on this implementation all the WAP logic can run inside one single lambda perform.

There may be nothing inherently mistaken with the JVM. It’s a basic element of many Massive Knowledge frameworks, offering a typical API to work with platform-specific sources, whereas guaranteeing safety and correctness. Nonetheless, the JVM takes a toll from a developer expertise perspective. Anyone who labored with Spark is aware of that JVM-based methods are typically finicky and fail with mysterious errors. For many individuals who work in knowledge and take into account Python as their lingua franca the benefit of the JVM is paid within the coin of usability.

We hope extra persons are enthusiastic about composable designs like we’re, we hope open requirements like Iceberg and Arrow will turn out to be the norm, however most of all we hope that is helpful.

So it goes.