Storage is decoupled from compute: In practice this means storage and compute use separate clusters, thus these systems are able to scale to many more concurrent users and larger data sizes.This reduces staleness and improves recency, reduces latency, and lowers the cost of having to operationalize two copies of the data in both a data lake and a warehouse. BI support: Lakehouses enable using BI tools directly on the source data.
The system should be able to reason about data integrity, and it should have robust governance and auditing mechanisms. Schema enforcement and governance: The Lakehouse should have a way to support schema enforcement and evolution, supporting DW schema architectures such as star/snowflake-schemas.Support for ACID transactions ensures consistency as multiple parties concurrently read or write data, typically using SQL. Transaction support: In an enterprise lakehouse many data pipelines will often be reading and writing data concurrently.They are what you would get if you had to redesign data warehouses in the modern world, now that cheap and highly reliable storage (in the form of object stores) are available.Ī lakehouse has the following key features: Lakehouses are enabled by a new system design: implementing similar data structures and data management features to those in a data warehouse directly on top of low cost cloud storage in open formats. A lakehouse is a new, open architecture that combines the best elements of data lakes and data warehouses. New systems are beginning to emerge that address the limitations of data lakes. Having a multitude of systems introduces complexity and more importantly, introduces delay as data professionals invariably need to move or copy data between different systems. A common approach is to use multiple systems - a data lake, several data warehouses, and other specialized systems such as streaming, time-series, graph, and image databases. Most of the recent advances in AI have been in better models to process unstructured data (text, images, video, audio), but these are precisely the types of data that a data warehouse is not optimized for. Companies require systems for diverse data applications including SQL analytics, real-time monitoring, data science, and machine learning. The need for a flexible, high-performance system hasn’t abated. For these reasons, many of the promises of the data lakes have not materialized, and in many cases leading to a loss of many of the benefits of data warehouses. While suitable for storing data, data lakes lack some critical features: they do not support transactions, they do not enforce data quality, and their lack of consistency / isolation makes it almost impossible to mix appends and reads, and batch and streaming jobs. About a decade ago companies began building data lakes - repositories for raw data in a variety of formats. Data warehouses are not suited for many of these use cases, and they are certainly not the most cost efficient.Īs companies began to collect large amounts of data from many different sources, architects began envisioning a single system to house data for many different analytic products and workloads. But while warehouses were great for structured data, a lot of modern enterprises have to deal with unstructured data, semi-structured data, and data with high variety, velocity, and volume. Since its inception in the late 1980s, data warehouse technology continued to evolve and MPP architectures led to systems that were able to handle larger data sizes. In this post we describe this new architecture and its advantages over previous approaches.ĭata warehouses have a long history in decision support and business intelligence applications. Over the past few years at Databricks, we've seen a new data management architecture that emerged independently across many customers and use cases: the lakehouse.
0 kommentar(er)
