The landscape of big data management has been transformed by the rising popularity of open table formats such as Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake. These formats, designed to address the limitations of traditional data storage systems, have become essential in modern data architectures. As organizations adopt various open table formats to suit their specific needs, the demand for interoperability between these formats has grown significantly. This interoperability is crucial for enabling seamless data access, reducing data silos, and fostering a more flexible and efficient data ecosystem.
Delta Lake UniForm is an open table format extension designed to provide a universal data representation that can be efficiently read by different processing engines. It aims to bridge the gap between various data formats and processing systems, offering a standardized approach to data storage and retrieval. With UniForm, you can read Delta Lake tables as Apache Iceberg tables. This expands data access to broader options of analytics engines.
This post explores how to start using Delta Lake UniForm on Amazon Web Services (AWS). You can learn how to query Delta Lake native tables through UniForm from different data warehouses or engines such as Amazon Redshift as an example of expanding data access to more engines.
How Delta Lake UniForm works UniForm allows other table format clients such as Apache Iceberg to access Delta Lake tables. Under the hood, UniForm generates Iceberg metadata files (including metadata and manifest files) that are required for Iceberg clients to access the underlying data files in Delta Lake tables. Both Delta Lake and Iceberg metadata files reference the same data files. UniForm generates multiple table format metadata without duplicating the actual data files. When an Iceberg client reads a UniForm table, it first accesses the Iceberg metadata files for the UniForm table, which then allows the Iceberg client to read the underlying data files.
There are two options to use UniForm:
First, to create a new Delta Lake table enabling UniForm, you configure table properties for UniForm in a CREATE TABLE DDL query. The table properties are 'delta.universalFormat.enabledFormats'='iceberg' and 'delta.enableIcebergCompatV2'='true'. When these options are set to the CREATE TABLE query, Iceberg metadata files are generated along with Delta Lake metadata files. In addition to these options, Delta Lake table protocol versions that define supported features by the table such as delta.minReaderVersion and delta.minWriterVersion are required to be set to 2 and 7 or more respectively. For more information about the table protocol versions, refer to What is a table protocol specification? in Delta Lake public document. Appendix 1. Create a new Delta Lake table with UniForm shows an example query to create a new Delta Lake UniForm table.
You can also enable UniForm on an existing Delta Lake table. This option is suitable if you have Delta Lake tables in your environment. Enabling UniForm doesn’t affect your current operations on the Delta Lake tables. To enable UniForm on a Delta Lake table, run REORG TABLE db.existing_delta_lake_table APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2)). After running this query, Delta Lake automatically generates Iceberg metadata files for the Iceberg client. In the example in this post, you run this option and enable UniForm after you create a Delta Lake table.
For the information about enabling UniForm, refer to Enable Delta Lake UniForm in the Delta Lake public document. Note that the extra package (delta-iceberg) is required to create a UniForm table in AWS Glue Data Catalog. The extra package is also required to generate Iceberg metadata along with Delta Lake metadata for the UniForm table.
Example use case A fictional company built a data lake with Delta Lake on Amazon Simple Storage Service (Amazon S3) that’s mainly used through Amazon Athena. According to its usage expansion, this company wants to expand data access to cloud-based data warehouses such as Amazon Redshift for flexible analytics use cases.
There are a few challenges to achieve this requirement. Delta Lake isn’t natively supported in Amazon Redshift. For those data warehouses, Delta Lake tables need to be converted to manifest tables, which requires additional operational overhead. You need to run the GENERATE command on Spark or use a crawler in AWS Glue to generate manifest tables, and you need to sync those manifest tables every time the Delta tables are updated.
Delta Lake UniForm can be a solution to meet this requirement. With Delta Lake UniForm, you can make the Delta Table compatible with the other open table formats such as Apache Iceberg, which is natively supported in Amazon Redshift. Users can query those Delta Lake tables as Iceberg tables through UniForm.
The following diagram describes the architectural overview to achieve that requirement.
In this tutorial, you create a Delta Lake table with a synthetic review dataset that includes different products and customer reviews and enable UniForm on that Delta Lake table to make it accessible from Amazon Redshift. Each component works as follows in this scenario:
Set up resources In this section, you complete the following resource setup:
Launch a CloudFormation template to configure basic resources You use a provided CloudFormation template to set up resources to build Delta Lake UniForm environments. The template creates the following resources.
Complete the following steps to deploy the resources.
Choose Launch stack:
For Stack name, enter delta-lake-uniform-on-aws. For the Parameters, DeltaDatabaseName, PublicSubnetForEMRonEC2, and VpcCIDRForEMRonEC2 are set by default. You can also change the default values. Then, choose Next.
Launch an Amazon EMR on EC2 cluster Complete the following steps to create an Amazon EMR on EC2 cluster.
delta-lake-uniform-blog-post in Name and confirm choosing emr-7.3.0 as its release label.s3://<DeltaLakeS3Bucket>/emr-cluster-logs as the Amazon S3 location. Replace <DeltaLakeS3Bucket> with the S3 bucket from the CloudFormation stack Outputs tab.s3://aws-blogs-artifacts-public/artifacts/BDB-4538/config.json as the Amazon S3 location.Create an Amazon EMR Studio Workspace Complete the following steps to create an Amazon EMR Studio Workspace to use Delta Lake UniForm on Amazon EMR on EC2.
delta-lake-uniform-studio as the Studio name.s3://aws-emr-studio-<ACCOUNT_ID>-<REGION>-delta-lake-uniform-on-aws) can be obtained from EMRStudioS3Bucket on the CloudFormation Outputs tab. Then, choose EMRStudioRole as the IAM role (you can find the IAM Role name on the CloudFormation Outputs tab).delta-lake-workspace as the Workspace name.VpcForEMR and PublicSubnetForEMR keys on the CloudFormation Outputs tab respectively.Upload Jupyter Notebook Complete the following steps to configure a Jupyter Notebook to use Delta Lake UniForm with Amazon EMR.
Choose the arrow icon at the top of the page and upload the Notebook you just downloaded.
Choose and open the notebook (delta-lake-uniform-on-aws.ipynb) you uploaded in the left pane.
After the notebook is opened, choose EMR Compute in the navigation pane.
Attach the Amazon EMR on EC2 cluster you created in the previous section. Choose EMR on EC2 cluster and set the cluster you created previously to EMR on EC2 cluster, then choose Attach.
Create a workgroup and a namespace for Amazon Redshift Serverless For this step, you configure a workgroup and a namespace for Amazon Redshift Serverless to run queries on a Delta Lake UniForm table. You also configure two subnets in the same VPC created by the CloudFormation stack delta-lake-uniform-on-aws. To deploy the resources, complete the following steps:
Choose Launch stack:
For Stack name, enter redshift-serverless-for-delta-lake-uniform.
Now you’re ready to use Delta Lake UniForm on Amazon EMR.
Enable Delta Lake UniForm Start by creating a Delta Lake table that contains the customer review dataset. After creating the table, run REORG query to enable UniForm on the Delta Lake table.
Create a Delta Lake table Complete the following steps to create a Delta Lake table based on a customer review dataset and review the table metadata.
Return to the Jupyter Notebook connected to the Amazon EMR on EC2 cluster and run the following cell to add delta-iceberg.jar to use UniForm and configure the spark extension.
Initialize the SparkSession. The following configuration is necessary to use Iceberg through UniForm. Before running the code, replace <DeltaLakeS3Bucket> with the name of the S3 bucket for Delta Lake, which you can find on the CloudFormation stack Outputs tab.
Create a Spark DataFrame from customer reviews.
Create a Delta Lake table with the customer reviews dataset. This step takes approximately 5 minutes.
Run DESCRIBE EXTENDED {DB_TBL} in the next cell to review the table. The output includes the table schema, location, table properties, and so on.
The Delta Lake table creation is complete. Next, enable UniForm on this Delta Lake table.
Run REORG query to enable UniForm To allow an Iceberg client to access the Delta Lake table you created, enable UniForm on the table. You can also create a new Delta Lake table with UniForm enabled. For more information, see Appendix 1at the end of this post. To enable UniForm and review the table metadata, complete the following steps.
Run the following query to enable UniForm on the Delta Lake table. To enable UniForm on an existing Delta Lake table, you run REORG query against the table.
Run DESCRIBE EXTENDED {DB_TBL} in the next cell to review the table metadata and compare it from before and after enabling UniForm. The new properties, such as delta.enableIcebergCompatV2=true and delta.universalFormat.enabledFormats=iceberg, are added to the table properties.
Run aws s3 ls s3://<DeltaLakeS3Bucket>/warehouse/ --recursive to confirm if the Iceberg table metadata is created. Replace <DeltaLakeS3Bucket> with the S3 bucket from the CloudFormation Outputs tab. The following screenshot shows the command output of table metadata and data files. You can confirm that Delta Lake UniForm generates both Iceberg metadata and Delta Lake metadata files as indicated by the red rectangles below.
Before querying the Delta Lake UniForm table from an Iceberg client, run the following analytic query for the Delta Lake UniForm table from Amazon EMR on EC2 side, and review the reviews count by each product category.
The query result shows the output of the reviews count by product_category:
Enabling UniForm on the Delta Lake table is complete, and now you can query the Delta Lake table from an Iceberg client. Next, you query the Delta Lake table as an Iceberg table from Amazon Redshift.
Run queries against the UniForm table from Amazon Redshift In the previous section, you enabled UniForm on your existing Delta Lake table. This allows you to run queries on a Delta Lake table as if it were an Iceberg table from Amazon Redshift. In this section, you run an analytic query on the UniForm table using Amazon Redshift Serverless and add records with a new product category to the UniForm table through the Jupyter Notebook connected to the Amazon EMR on EC2 cluster. Then, you verify the added records with another analytic query from Amazon Redshift. You can confirm that Delta Lake UniForm enables Amazon Redshift to query the Delta Lake table through this section.
Query the UniForm table from Amazon Redshift Serverless 1. Open Amazon Redshift Serverless console
2. In Namespaces/Workgroups, select the delta-lake-uniform-namespace that you created using the CloudFormation stack.
3. Choose Query data on the right top corner to open the Amazon Redshift query editor.
4. After opening the editor, select the delta-lake-uniform-workgroup workgroup in the left pane.
5. Choose Create connection.
6. After you successfully create a connection, you can see the delta_uniform_db database and customer_review table you created in the left pane of the editor.
7. Copy and paste the following analytic query to the editor and choose Run.
SELECT product_category, count(*) as count_by_product_category FROM "awsdatacatalog"."delta_uniform_db"."customer_reviews"GROUP BY product_category ORDER BY count_by_product_category DESC
8. The editor shows the same result of the review count by product_category as you obtained from Jupyter Notebook in Run REORG query to enable UniForm.
Add new product category records into Delta Lake UniForm table from Amazon EMR Go back to the Jupyter Notebook on Amazon EMR Workspace to add new records with a new product category (Books) into the Delta Lake UniForm table. After adding the records, query the UniForm table again from Amazon Redshift Serverless.
On the Jupyter Notebook, go to Add new product category records into the UniForm table and run the following cell to load new records.
Run the following cell and review the five records with Books as the product category. The following screenshot shows the output of this code.
Add the new reviews with Books product category. This takes around 2 minutes.
In the next section, you run a query on the UniForm table from Amazon Redshift Serverless to check if the new records with the Books product category have been added.
Review the added records in Delta Lake UniForm table from Amazon Redshift Serverless To check if the result output includes the records of Books product category:
Books product category.SELECT product_category, count(*) as count_by_product_category FROM "awsdatacatalog"."delta_uniform_db"."customer_reviews"GROUP BY product_category ORDER BY count_by_product_category DESC
2. The following screenshot shows the output of the query you ran in the previous step. You can confirm the new product category Books has been added to the table from Amazon Redshift side.
Now you can query from Amazon Redshift against the Delta Lake table by enabling Delta Lake UniForm.
Clean up resources To clean up your resources, complete the following steps:
aws-emr-studio-<ACCOUNT_ID>-<REGION>-delta-lake-uniform-on-aws. Replace <ACCOUNT_ID> and <REGION> with your account ID and the bucket’s region.delta-lake-uniform-on-aws-deltalakes3bucket-abcdefghijk.redshift-serverless-for-delta-lake-uniform.delta-lake-uniform-on-aws.Conclusion Delta Lake UniForm on AWS represents an advancement in addressing the challenges of data interoperability and accessibility in modern big data architectures. By enabling Delta Lake tables to be read as Apache Iceberg tables, UniForm expands data access capabilities, allowing organizations to use a broader range of analytics engines and data warehouses such as Amazon Redshift.
The practical implications of this technology are substantial, offering new possibilities for data analysis and insights across diverse platforms. As organizations continue to navigate the complexities of big data, solutions like Delta Lake UniForm that promote interoperability and reduce data silos will become increasingly valuable.
By adopting these advanced open table formats and using cloud platforms such as AWS, organizations can build more robust and efficient data ecosystems. This approach not only enhances the value of existing data assets but also fosters a more agile and adaptable data strategy, ultimately driving innovation and improving decision-making processes in our data-driven world.
Appendix 1: Create a new Delta Lake table with UniForm You can create a Delta Lake table with UniForm enabled using the following DDL.
CREATE TABLE IF NOT EXISTS delta_uniform_db.customer_reviews_create ( marketplace string, customer_id string, review_id string, product_id string, product_title string, star_rating bigint, helpful_votes bigint, total_votes bigint, insight string, review_headline string, review_body string, review_date timestamp, review_year bigint, product_category string)USING deltaTBLPROPERTIES ( 'delta.universalFormat.enabledFormats'='iceberg', 'delta.enableIcebergCompatV2'='true', 'delta.minReaderVersion'='2', 'delta.minWriterVersion'='7')
Appendix 2: Run queries from Snowflake against the UniForm table Delta Lake UniForm also allows you to run queries on a Delta Lake table from Snowflake. In this section, you run the same analytic query on the UniForm table using Snowflake as you previously did using Amazon Redshift Serverless in Run queries from Amazon Redshift against the UniForm table. Then you confirm that the query results from Snowflake match the results obtained from the Amazon Redshift Serverless query.
Configure IAM roles for Snowflake to access AWS Glue Data Catalog and Amazon S3 To query the Delta Lake UniForm table in Data Catalog from Snowflake, the following configurations are required.
Create IAM roles for Snowflake to access AWS Glue Data Catalog and Amazon S3 Create the following two IAM roles for Snowflake to access AWS Glue Data Catalog and Amazon S3.
To configure the IAM roles, complete the following steps:
Choose Launch stack:
Enter snowflake-iceberg as the stack name and choose Next.
Create an AWS Glue Data Catalog Integration Create a catalog integration for AWS Glue Data Catalog. For more information about the catalog integration for AWS Glue Data Catalog, refer to Configure a catalog integration for AWS Glue in the Snowflake public documentation. To configure the catalog integration, complete the following steps:
<YOUR_ACCOUNT_ID> with the IAM role ARN from the snowflake-iceberg CloudFormation Ouputs tab, and replace <REGION> with the region of AWS Glue Data Catalog.CREATE CATALOG INTEGRATION glue_catalog_integration CATALOG_SOURCE=GLUE CATALOG_NAMESPACE='delta_uniform_db' TABLE_FORMAT=ICEBERG GLUE_AWS_ROLE_ARN='arn:aws:iam::<YOUR_ACCOUNT_ID>:role/SnowflakeIcebergGlueCatalogRole' GLUE_CATALOG_ID='<YOUR_ACCOUNT_ID>' GLUE_REGION='<REGION>' ENABLED=TRUE;
3. Retrieve GLUE_AWS_IAM_USER_ARN and GLUE_AWS_EXTERNAL_ID by using DESCRIBE CATALOG INTEGRATION glue_catalog_integration in the editor. The output is similar to the following:
+------------------------------------------------------------------------------------------------------------------------------+| property | property_type | property_value | property_default ||--------------------------+---------------+----------------------------------------------------------------+------------------|| ENABLED | Boolean | true | false || CATALOG_SOURCE | String | GLUE | || CATALOG_NAMESPACE | String | delta_uniform_db | || TABLE_FORMAT | String | ICEBERG | || REFRESH_INTERVAL_SECONDS | Integer | 30 | 30 || GLUE_AWS_ROLE_ARN | String | arn:aws:iam::123456789012:role/SnowflakeIcebergGlueCatalogRole | || GLUE_CATALOG_ID | String | 123456789012 | || GLUE_REGION | String | us-east-1 | || GLUE_AWS_IAM_USER_ARN | String | arn:aws:iam::123456789012:user/<ID> | || GLUE_AWS_EXTERNAL_ID | String | An external ID specified on the IAM Role trust relationships | || COMMENT | String | | |+------------------------------------------------------------------------------------------------------------------------------+
4. Update the IAM role you created using the CloudFormation stack to enable Snowflake to access AWS Glue Data Catalog using that IAM role. Open Trust Relationships of SnowflakeIcebergGlueCatalogRole on the IAM console, choose Edit and update the trust relationship using the following policy. Replace <GLUE_AWS_IAM_USER_ARN> and <GLUE_AWS_EXTERNAL_ID> with the names you obtained in the previous step.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": {"AWS": "<GLUE_AWS_IAM_USER_ARN>" }, "Action": "sts:AssumeRole", "Condition": {"StringEquals": { "sts:ExternalId": "<GLUE_AWS_EXTERNAL_ID>"} } } ]}
You completed setting up the IAM role for Snowflake to access your Data Catalog resources. Next, configure the IAM role for Amazon S3 access.
Register Amazon S3 as an external volume In this section, you configure an external volume for Amazon S3. Snowflake accesses the UniForm table data files in S3 through the external volume. For the configuration of an external volume for S3, refer to Configure an external volume for Amazon S3 in Snowflake public documentation. To configure the external volume, complete the following steps:
<DeltaLakeS3Bucket> with the name of the S3 bucket that you created in Launch a CloudFormation template to configure basic resources from the CloudFormation Outputs tab. Replace <ACCOUNT_ID> with your AWS account ID.CREATE OR REPLACE EXTERNAL VOLUME delta_lake_uniform_s3 STORAGE_LOCATIONS = ( (NAME = 'delta-lake-uniform-on-aws'STORAGE_PROVIDER = 'S3'STORAGE_BASE_URL = 's3://<DeltaLakeS3Bucket>'STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<ACCOUNT_ID>:role/SnowflakeIcebergS3Role' ) );
2. Retrieve STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID by running DESCRIBE EXTERNAL VOLUME delta_lake_uniform_s3 in the editor. The output is similar to the following:
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| parent_property | property | property_type | property_value | property_default ||-------------------+--------------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------+------------------|| | ALLOW_WRITES | Boolean | true | true || STORAGE_LOCATIONS | STORAGE_LOCATION_1 | String | {"NAME":"uniform_s3_location","STORAGE_PROVIDER":"S3","STORAGE_BASE_URL":"s3://<DeltaLakeS3Bucket>","STORAGE_ALLOWED_LOCATIONS":["s3 | || | | | ://<DeltaLakeS3Bucket>/*"],"STORAGE_REGION":"us-east-1","PRIVILEGES_VERIFIED":true,"STORAGE_AWS_ROLE_ARN":"arn:aws:iam::123456789012 | || | | | :role/SnowflakeIcebergS3Role","STORAGE_AWS_IAM_USER_ARN":"arn:aws:iam::123456789012:user/<ID>","STORAGE_AWS_EXTERNAL_ID":"<External | || | | | ID>",... } | || STORAGE_LOCATIONS | ACTIVE | String | uniform_s3_location | |+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2. Update the IAM role you created using the CloudFormation template (in Create IAM roles for Snowflake to access AWS Glue Data Catalog and Amazon S3) to enable Snowflake to use this IAM role. Open Trust Relationships of SnowflakeIcebergS3Role on the IAM console, choose Edit, and update the trust relationship with the following policy. Replace <STORAGE_AWS_IAM_USER_ARN> and <STORAGE_AWS_EXTERNAL_ID> with the values from the previous step.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "<STORAGE_AWS_IAM_USER_ARN&>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<STORAGE_AWS_EXTERNAL_ID>" } } } ]}
The next step is to create an Iceberg table to run queries from Snowflake.
Create an Iceberg table in Snowflake In this section, you create an Iceberg table in Snowflake. The table is an entry point for Snowflake to access the Delta Lake UniForm table in AWS Glue Data Catalog. To create the table, complete the following steps:
CREATE DATABASE <DATABASE_NAME>, replacing <DATABASE_NAME> with a unique database name for the Iceberg table.delta_uniform_snow_db is chosen for the table. Configure the following parameters:EXTERNAL_VOLUME: created by CREATE OR REPLACE EXTERNAL VOLUME query in the previous section, such as delta_lake_uniform_s3.CATALOG: created by the CREATE CATALOG INTEGRATION query in the previous section, such as glue_catalog_integration.CATALOG_TABLE_NAME: the name of Delta Lake UniForm table you created in Data Catalog such as customer_reviews.The complete query is below:
CREATE OR REPLACE ICEBERG TABLE customer_reviews_snow EXTERNAL_VOLUME='delta_lake_uniform_s3' CATALOG='glue_catalog_integration' CATALOG_TABLE_NAME='customer_reviews';
After the table creation is complete, you’re ready to query the UniForm table in AWS Glue Data Catalog from Snowflake.
Query the UniForm table from Snowflake In this step, you query the UniForm table from Snowflake. Paste and run the following analytic query in the Snowflake query editor.
SELECT product_category, count(*) as count_by_product_category FROM customer_reviews_snow GROUP BY product_category ORDER BY count_by_product_category DESC
The query result shows the same output as you saw in Review the added records in Delta Lake UniForm table from Amazon Redshift Serverless section.
+--------------------------------------------------+| PRODUCT_CATEGORY | COUNT_BY_PRODUCT_CATEGORY ||----------------------+---------------------------|| Office_Products | 9673711 || Books | 9672664 || Apparel | 6448747 || Computers | 3224215 || Beauty_Personal_Care | 3223599 |+--------------------------------------------------+
Now you can query from Snowflake against the Delta Lake table by enabling Delta Lake UniForm.
About the Authors Tomohiro Tanaka is a Senior Cloud Support Engineer at Amazon Web Services. He’s passionate about helping customers use Apache Iceberg for their data lakes on AWS. In his free time, he enjoys a coffee break with his colleagues and making coffee at home.
Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team. He works based in Tokyo, Japan. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike.