When we consider how we use the data we work with every day, we should keep in mind that, in most cases, this data serves many purposes and should not be treated as disposable, whether it is older or more recent.
All of this data is information that ultimately becomes strategic for a wide range of decisions and purposes.
In recent years, we have come to value the data companies use every day even more, anticipating a not-so-distant future in which we must optimize processes for ever-faster decisions and use cases.
Along the way, we saw the rise of the data lakehouse, which is simply the combination of data lakes (large repositories of raw data in its original form) and data warehouses (organized sets of structured data), drawing on the best of each architecture to create massive raw data storage at lower cost.
Beyond lower costs, this new approach to storage architecture appeals to companies that use multiple solutions for different day-to-day needs, because they can replace several systems with a single lakehouse. Keep in mind, however, that this architecture model does not provide a single source of information, and it becomes a powerful asset when combined with technology such as Starburst or Starburst Galaxy.
Below, we take a closer look at each of these architectures, with an overview of the benefits and limitations that make data lakehouses a viable architecture for large volumes of information.
Databases:
Databases are the traditional foundation of data technology. Although they take many forms, all databases enable structured data collection. The data is organized and stored systematically to allow efficient retrieval, management and manipulation of information.
Database technologies are generally categorized by functionality, purpose and features, for example:
Database management systems (DBMS): The backbone of data solutions, used primarily to perform create, read, update and delete (CRUD) operations that support many other business applications.
OLTP (Online Transactional Processing): Systems used to create transactional data. They are commonly used to record financial transactions and log data, for example.
ACID compliance: ACID stands for Atomicity, Consistency, Isolation and Durability. This model validates transactional data: operations either complete with 100% accuracy or fail entirely. This level of integrity protects data from any loss or nonconformity.
Data warehouses:
Data warehouses are among the most traditional data storage models, using a centralized, structured and integrated repository designed specifically for queries, reporting and analytics. While these databases were once run on-premises, in recent years this storage has been moving to the cloud.
It is worth noting, however, that on-premises and cloud data warehouses share the same advantages and drawbacks. Below, we take a closer look at data warehouses.
Structured data: For data warehouses to work properly, the stored data must be correctly structured according to predefined standards.
Performance: With structured data specifically, data warehouses deliver high performance, so depending on the workload they are still recommended.
Drawbacks: Data warehouses do not support unstructured data, which performs better in data lakes, for example. In addition, data warehouses demand significant financial and technology resources, making them unfeasible for many companies.
Data lakes:
Data lakes are among today's most modern storage models, used to store structured, unstructured and semi-structured data. They are designed to hold large volumes of information and to support machine learning (ML) and artificial intelligence (AI) workloads. They are typically used for read operations and cost far less than data warehouses.
Multiple data formats: Unlike data warehouses, data lakes support a much wider range of data formats, making them far more versatile. On cost, data lakes also have much lower technology costs than other solutions.
Optimized for machine learning (ML) and AI: Data lakes store vast amounts of raw data. That makes them ideal for feeding machine learning frameworks, often driven by models built in Python, Scala or R. They are also a core technology for training emerging artificial intelligence (AI) models.
Why are data lakes so widely used?
On closer analysis, companies' adoption of data lakes is directly tied to how easily they scale and how little they cost. For large enterprises — and multinationals in particular — it is impossible to store high volumes of information on an architecture that cannot scale.
In terms of storage capacity and cost, data lakes provide a low-cost architecture that can hold large volumes of raw data. In this type of storage, compute and storage are also decoupled, so each can be scaled up or down independently.
As a result, data lakes let architects allocate resources as and when needed. For example, if an organization needs more storage, it can acquire it quickly and efficiently without also buying compute it may not need. Instead, compute and storage needs scale individually, in real time, saving significant time, effort and money.
Finally, looking more closely at costs, data lakes offer much cheaper storage than other options. And as the technology has advanced, their performance has improved to the point where it now rivals traditional data warehouses.
The combination of a scalable, cost-effective and efficient system is a clear win for many organizations, striking an ideal balance across several factors.
What is the difference between a data lake and a data lakehouse?
In recent years, data lakes have matured to the point where modern data lakes include many capabilities that earlier iterations lacked. These modern data lakes are defined by modern, open table formats such as Iceberg, Delta Lake or Hudi. They offer enhanced transactional support that rivals the best OLTP databases, along with capabilities such as schema evolution, partition evolution and time travel. This has further narrowed the gap between data lakes, data warehouses and databases. That convergence gave rise to the term data lakehouse to describe modern data lakes, a blend of data lake and data warehouse.
Data lakehouses include several key features that set them apart from traditional data lakes. These include:
Enhanced governance layer and metadata collection:
Compared with a traditional data lake, a data lakehouse handles data governance far better. This stems from a more advanced approach to metadata collection, which can build a more accurate record of table changes at any point in time. This is the key capability that enables advanced features and functionality that blur the line between data lake, data warehouse and database.
Traditional data lake:
Data lakes, especially those using cloud object storage, typically rely on an immutable storage layer. It can be built on AWS S3, Azure ADLS, Google Cloud Storage or a similar system. In these systems, data can be read and written, but usually cannot be fully deleted or updated. This issue often affects Hive users, who may believe user data has been deleted when it has in fact persisted.
Modern table formats:
All of these capabilities and benefits are made possible by adopting one of the modern lakehouse table formats, such as Iceberg, Delta Lake or Hudi. These formats add a metadata layer on top of the tables in a data lakehouse that enables data warehouse–style capabilities.
Use of modern file formats:
Note that data lakehouses also use modern file formats, just as you would expect from a data lake. These include popular modern formats such as Parquet, ORC and Avro.
Lakehouse enhancements:
In a data lakehouse, enhancements to the governance layer improve metadata collection, allowing data to be fully updated and deleted as needed. This opens up new use cases that were previously out of reach for traditional data lakes. For example, it ensures that regulatory requirements such as GDPR can be fully met using data lake technology. GDPR governance often requires the ability to delete user data on request.
Is this the end of data lakes?
Overall, you should think of the data lakehouse as an evolution of the data lake, not a replacement. That is why lakehouse adoption can be gradual. It is not a technology that forces users to rethink everything they do or dismantle established processes. Instead, a data lakehouse can fit the natural evolution of an organization's data usage patterns while delivering many improvements along the way.
If you enjoyed this content, stay tuned to the Digiage blog — and if you have any questions, we're here to help!