Overview
Data Storage & Management is 20% of the ABDE exam. It covers designing warehouse and lake architectures, optimizing how data is stored, setting up partitioning and indexing, managing lifecycle and retention policies, designing hybrid storage that spans on-premises and cloud, and tuning queries. DASCA names Apache Hive, the SQL layer over distributed storage, which is a strong hint that partition pruning, columnar file formats and the metastore are fair game.
The domain exists because volume, access pattern and cost pull storage in different directions. A layout that suits nightly aggregate scans can be wrong for point lookups, and a retention policy that keeps everything on premium storage quietly multiplies the bill. It sits between ingestion (Domain 1), which decides what lands, and modeling (Domain 5), which decides how tables are shaped for analysis.
How it's tested
Questions describe a dataset, how it is queried and a pain point such as slow queries, high cost or compliance retention. You pick the architecture, partition key, file format, storage tier or tuning step that BEST fits. Read the query filters closely, because the column users filter on most often usually decides the answer.
-
Fit the architecture
Given data variety, users and governance needs, choose a warehouse, a data lake or a lakehouse with an open table format.
-
Fix a slow or costly query
A query scans too much data or reads thousands of small files, and you choose partitioning, columnar format, compaction or statistics as the fix.
-
Apply lifecycle rules
Retention and access-frequency details lead you to the right storage tier, archival rule or deletion policy.
Analysts query a five-year clickstream table in Hive, almost always filtering on event date and sometimes on country. Queries scan the whole table. Partitioning by event date lets the engine skip every partition outside the filter, and storing files as Parquet lets it read only the selected columns. Partitioning by user ID would create millions of tiny partitions and would not match the common filter.
Practical applications 6
Official practical applications from the DASCA outline, in outline order.
2.1 Creating data warehouse/lake architecturesWarehouse, lake and lakehouse, and when each fits.
- What it means
- A data warehouse stores structured, modeled data with schema enforced on write for fast SQL analytics; a data lake stores raw data of any format cheaply in object or distributed storage with schema applied on read; a lakehouse adds table formats such as Delta Lake, Apache Iceberg or Apache Hudi to give lake storage ACID transactions and schema enforcement.
- What to understand
- Most designs organize the lake in layers, such as raw (bronze), cleaned (silver) and curated (gold), so data can be reprocessed from raw. A lake without catalog, ownership and quality controls degrades into a 'data swamp' that nobody trusts. Pick the warehouse for governed BI on structured data, the lake for varied raw data and data science, and a lakehouse when you need both on one copy.
- How it shows up
- A scenario lists data types (logs, images, transactional tables), consumers (BI analysts, data scientists) and needs (ACID updates, low cost), and you choose the architecture.
- Common confusion
- Believing a data lake makes structure unnecessary; it delays schema to read time but still needs layers, metadata and governance.
2.2 Implementing storage optimization strategiesFile formats, compression and file sizing.
- What it means
- Storage optimization means choosing columnar formats such as Parquet or ORC for analytics, applying compression codecs such as Snappy or ZSTD, and keeping files at a healthy size, often in the range of a hundred megabytes to around a gigabyte for distributed engines.
- What to understand
- Columnar formats let a query read only the columns it needs and compress well because similar values sit together, and they store min/max statistics that let engines skip row groups. Many tiny files hurt performance because each file adds open and metadata overhead and loads the NameNode or metastore, so streaming and frequent small writes need periodic compaction.
- How it shows up
- A table written every minute has slowed down over months, and the correct fix is compaction into larger files rather than more compute.
- Common confusion
- Picking row-based formats such as CSV or Avro for analytic scans; Avro suits record-at-a-time writes and schema evolution in pipelines, while Parquet and ORC suit column-oriented reads.
2.3 Setting up partitioning and indexingLetting engines skip data that a query does not need.
- What it means
- Partitioning splits a table into separate directories or segments by a column value, such as date, so queries filtering on that column read only matching partitions (partition pruning); indexing and clustering, including Hive bucketing, sort or hash data within storage to speed lookups and joins.
- What to understand
- Choose a partition column that appears in most query filters and has moderate cardinality; date is the classic choice. Very high-cardinality keys such as user ID create huge numbers of small partitions. Bucketing hashes rows on a column into a fixed number of files, which helps joins and sampling on that column without the explosion that partitioning would cause.
- How it shows up
- You are given the common WHERE clauses and the table's size, and you choose the partition key or explain why a query still scans everything, for example because it wraps the partition column in a function.
- Common confusion
- Partitioning on the most selective column, such as customer ID, instead of the column queries filter on with reasonable cardinality.
2.4 Managing data lifecycle and retention policiesTiers, archival and deletion over time.
- What it means
- Lifecycle management moves data between storage tiers as access frequency drops (hot, warm, cold, archive) and deletes it when retention ends, using rules such as object storage lifecycle policies or dropping expired partitions.
- What to understand
- Retention has two sides: keep data at least as long as legal or business rules require, and no longer than needed, because over-retention adds cost and privacy risk. Archive tiers are cheap to store but slow and costly to retrieve, so they suit data that is rarely read. Date partitioning makes retention easy because expiring a day means dropping one partition rather than running a large DELETE.
- How it shows up
- A scenario states a seven-year compliance requirement and that data older than 90 days is rarely queried, and you choose tiering plus automated deletion.
- Common confusion
- Moving data that dashboards still query every day to an archive tier to save money, which makes those queries slow or expensive.
2.5 Designing hybrid storage solutionsCombining on-premises and cloud, or multiple storage types.
- What it means
- Hybrid storage spreads data across on-premises systems such as HDFS and cloud object storage such as Amazon S3, Azure Data Lake Storage or Google Cloud Storage, or across different storage types chosen per workload.
- What to understand
- Hybrid designs are driven by data residency, existing investment, latency to on-premises applications, and cost. Separating storage from compute, which object storage enables, lets each scale independently, whereas classic HDFS ties storage to cluster nodes. Network egress charges and data movement latency are the hidden costs, so keep compute close to the data it reads most.
- How it shows up
- A company must keep regulated records on premises while bursting analytics to the cloud, and you choose what stays where and how it is synchronized.
- Common confusion
- Assuming moving all data to the cloud is always cheaper, ignoring egress fees, residency rules and the cost of repeatedly copying large datasets.
2.6 Performance tuning for query optimizationMaking large queries scan and shuffle less.
- What it means
- Query tuning reduces the data a query reads and moves, using partition pruning, predicate pushdown, column projection, up-to-date table statistics for the cost-based optimizer, suitable join strategies and pre-aggregated or materialized results.
- What to understand
- Read the execution plan first to see whether the engine is scanning full tables or shuffling huge amounts of data. Broadcast joins send a small table to every node and avoid shuffling the large one. Skewed keys, where one value holds much of the data, overload one task, and techniques such as salting or skew-join handling spread the load.
- How it shows up
- A join between a billion-row fact table and a small lookup table is slow, and the right answer is a broadcast or map-side join rather than adding cluster nodes.
- Common confusion
- Treating more hardware as the first fix, when the plan shows full scans or a skewed key that extra nodes would not cure.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Partition pruning
Partition pruning is the engine skipping entire partitions whose values cannot match a query's filter, so it reads only relevant directories.
- Remember
- Pruning only works when the query filters directly on the partition column; wrapping it in a function or casting it can force a full scan in many engines.
- Tested as
- The most common fix in slow-query scenarios on large date-filtered tables.
- Trap
- Partitioning on a column users rarely filter on, which gives no pruning at all.
-
Columnar formats (Parquet, ORC)
Columnar formats store each column's values together with per-block statistics, so analytic queries read only needed columns and skip blocks by min/max values.
- Remember
- Parquet and ORC are the default choice for analytic tables; ORC is closely associated with Hive and Parquet is the broadest cross-engine standard.
- Tested as
- Questions about reducing scan cost or storage size usually point to converting CSV or JSON to a columnar format.
- Trap
- Choosing Avro for analytics because it supports schema evolution, when Avro is row-based and better for pipeline records.
-
Small files problem
The small files problem is the slowdown caused when a table consists of very many tiny files, each adding open, scheduling and metadata overhead.
- Remember
- In HDFS every file consumes NameNode memory, so the fix is compaction into larger files and avoiding over-partitioning.
- Tested as
- Appears in scenarios about streaming writes or very fine-grained partitions that got slower over time.
- Trap
- Adding compute nodes, which does not reduce per-file overhead.
-
Bucketing
Bucketing (in Hive and Spark) hashes rows on a column into a fixed number of files, keeping matching keys together.
- Remember
- Use it for high-cardinality join or lookup keys where partitioning would create too many partitions; tables bucketed the same way on the join key can be joined with less shuffling.
- Tested as
- Tested as the alternative to partitioning on a high-cardinality column.
- Trap
- Partitioning on user ID instead of bucketing on it.
-
Data lake layers (bronze, silver, gold)
Layering, often called the medallion pattern, keeps raw data (bronze), cleaned and conformed data (silver) and business-ready aggregates (gold) as separate zones.
- Remember
- The raw layer is kept immutable so any downstream layer can be rebuilt after a bug fix.
- Tested as
- Architecture questions ask where a given dataset or consumer belongs.
- Trap
- Letting BI users query the raw layer directly, which exposes unvalidated data.
-
Open table formats (Delta Lake, Iceberg, Hudi)
Open table formats add a transaction log or metadata layer over files in a lake, providing ACID writes, schema enforcement, time travel and row-level updates and deletes.
- Remember
- They are what turn a data lake into a lakehouse and make GDPR-style deletes and upserts practical on lake storage.
- Tested as
- Scenarios needing updates, deletes or consistent reads on lake data point here.
- Trap
- Assuming plain Parquet files in a lake support safe concurrent updates.
-
Storage tiers
Storage tiers trade storage price against access speed and retrieval cost, from hot (frequent access) through cool or cold to archive.
- Remember
- Archive is cheapest to store but has slow and paid retrieval, so match tiers to measured access frequency.
- Tested as
- Lifecycle questions combine access patterns with retention periods.
- Trap
- Archiving data that is still queried regularly.
-
Hive metastore and external tables
The Hive metastore stores table schemas and partition locations; an external table points at data files Hive does not own, while a managed table's data is owned by Hive.
- Remember
- Dropping a managed table deletes its data, while dropping an external table removes only the metadata.
- Tested as
- Tested in questions about sharing lake data across engines and about accidental data loss.
- Trap
- Dropping a managed table expecting the underlying files to remain.
-
Predicate pushdown
Predicate pushdown sends filter conditions down to the storage or file reader so rows and blocks are skipped before data reaches the query engine.
- Remember
- It works best with columnar formats that store statistics, and with sorted or clustered data where min/max ranges are tight.
- Tested as
- Explains why the same query is much faster on Parquet than on CSV.
- Trap
- Confusing it with partition pruning, which skips whole partitions rather than blocks inside files.
-
Broadcast join
A broadcast join copies a small table to every worker so the large table can be joined locally without being shuffled across the network.
- Remember
- It only suits a table small enough to fit in each worker's memory.
- Tested as
- The standard fix for slow fact-to-dimension joins in distributed engines.
- Trap
- Trying to broadcast two large tables, which can exhaust memory.
-
Separation of storage and compute
Separating storage and compute keeps data in object storage while compute clusters are started, resized or stopped independently.
- Remember
- It lowers cost for bursty workloads and lets several engines read the same data, unlike classic HDFS where storage lives on cluster nodes.
- Tested as
- Hybrid and cloud architecture questions test this trade-off.
- Trap
- Assuming HDFS clusters can be shut down without losing access to the data they hold.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Data warehouse vs. data lake vs. lakehouse
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Data warehouse | Structured, modeled data with schema enforced on write | Fast governed SQL analytics, but less suited to raw and unstructured data | Forcing logs, images and raw JSON into it just to keep one platform |
| Data lake | Raw data of any format in cheap object or distributed storage, schema on read | Flexible and low cost, but no ACID updates or enforced schema on its own | Skipping catalog and quality controls, creating a data swamp |
| Lakehouse | Lake storage plus an open table format such as Delta Lake, Iceberg or Hudi | Adds ACID transactions, schema enforcement and row-level changes on one copy of data | Thinking plain Parquet folders already provide these guarantees |
Partitioning vs. bucketing vs. indexing
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Partitioning | Separate directories or segments per value of a column | Skips whole partitions; best for low-to-moderate cardinality filter columns such as date | Using a high-cardinality key and creating millions of small partitions |
| Bucketing | Hash rows on a column into a fixed number of files | Controls file count for high-cardinality keys and helps joins on that key | Expecting it to prune data the way partition filters do |
| Indexing | An auxiliary structure that locates rows by key value | Essential for point lookups in row stores; columnar analytic engines rely more on statistics and sorting | Applying OLTP indexing habits to large distributed scans |
Parquet vs. ORC vs. Avro
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Parquet | Open columnar format with column statistics | Broadest support across engines for analytic reads | Using it for record-at-a-time streaming messages |
| ORC | Columnar format with built-in indexes, closely tied to Hive | Strong compression and Hive ACID table support | Assuming it is row-based because it originated in the Hadoop ecosystem |
| Avro | Row-based binary format with an embedded schema | Strong schema evolution for records in pipelines and message streams | Choosing it for column-heavy analytic queries |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Most selective partition key
- What candidates pick
- Partition a ten-billion-row events table by user ID because lookups by user are common.
- Why it looks right
- In relational databases the most selective column is often the best index, so it feels right.
- Why it's wrong
- Millions of distinct users create millions of tiny partitions, overwhelming the metastore and file system; bucketing on user ID with date partitioning fits better.
Clue to watch for The candidate column has very high cardinality, or the scenario mentions metastore or NameNode strain.
-
Scale out first
- What candidates pick
- Doubling the cluster to fix a query that slowed down as a streaming table grew.
- Why it looks right
- More nodes usually mean more throughput, and it requires no redesign.
- Why it's wrong
- If the table has accumulated thousands of small files, per-file overhead dominates; compaction solves the root cause at a fraction of the cost.
Clue to watch for Frequent small writes, such as micro-batches every minute, and performance degrading gradually over time.
-
Lake equals no structure
- What candidates pick
- Dumping all sources into one lake folder with no layers or catalog to move fast.
- Why it looks right
- Schema-on-read promises flexibility and fast onboarding.
- Why it's wrong
- Without layers, ownership and metadata, users cannot find or trust data; zones and a catalog keep the lake usable.
Clue to watch for Complaints that analysts cannot find datasets or do not trust them.
-
Drop the managed table
- What candidates pick
- Dropping and recreating a Hive managed table to change its definition while keeping the data.
- Why it looks right
- Dropping and recreating is a common way to change a table definition.
- Why it's wrong
- Dropping a managed table deletes its data files; an external table, or ALTER TABLE, keeps the data intact.
Clue to watch for The data must be preserved or is shared with other engines.
-
Cheapest tier for everything
- What candidates pick
- Moving the whole table to archive storage to meet a cost-reduction goal.
- Why it looks right
- Archive pricing is a fraction of hot storage.
- Why it's wrong
- Recent data still feeds daily reports, and archive retrieval is slow and charged; tier by age and access frequency instead.
Clue to watch for The scenario says recent data is read daily while older data is rarely touched.
-
Function on the partition column
- What candidates pick
- Accepting that a query filtering with YEAR(event_date) = 2025 is already optimized because the table is partitioned by date.
- Why it looks right
- The table is partitioned on that column, so pruning seems automatic.
- Why it's wrong
- Many engines cannot prune when the partition column is wrapped in a function; filtering on a direct date range lets pruning work.
Clue to watch for A partitioned table still shows a full scan in the query plan.
Answer framework
When an ABDE question asks how to store or speed up a dataset, work through these steps.
- List the data types, consumers and update needs, then choose warehouse, lake or lakehouse with an open table format.
- Find the columns most queries filter on and partition by the one with moderate cardinality, usually date; bucket high-cardinality join keys instead.
- Store analytic data in a columnar format with compression and keep files reasonably large, scheduling compaction for frequent small writes.
- Apply lifecycle rules: keep data as long as required, tier it by access frequency, and expire it by dropping partitions.
- Before adding hardware, read the query plan for full scans, skew or large shuffles and fix those causes first.
Practice question
Want more like this? The ABDE practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Begin with the architecture comparison, warehouse versus lake versus lakehouse, because it frames every other question in the domain. Be able to say in one sentence what each provides and what each lacks.
Next, spend the most time on physical layout: partitioning, bucketing, columnar formats and file sizing. If you can, create a Hive or Spark SQL table twice, once as CSV partitioned badly and once as Parquet partitioned by date, then compare how much data the same query reads. Seeing pruning and column projection in an execution plan makes the exam scenarios easy to decode. Candidates commonly underestimate the small files problem and the difference between managed and external tables.
Then cover lifecycle and hybrid storage. Practice turning a sentence such as 'keep seven years, rarely read after 90 days' into a tiering and expiry rule, and list the costs of moving data between on-premises and cloud.
Finally, practice query tuning by reading plans for full scans, shuffles and skew. Afterwards, attempt timed questions for this domain and review each miss by naming the filter column, file layout or access pattern you overlooked, then continue to Domain 5 to connect physical storage with data modeling.
FAQ
How much of the ABDE exam covers Data Storage & Management?
DASCA weights it at 20%, second only to ingestion. That suggests about 15 of the 75 questions, though DASCA does not publish per-domain counts.
Do I need to know Apache Hive in depth?
DASCA names Hive for this domain, so expect concepts such as partitions, bucketing, the metastore, and managed versus external tables. Knowing how Hive lays data out matters more than memorizing every configuration setting.
What is the difference between a data lake and a lakehouse?
A data lake is raw files in cheap storage with schema applied on read. A lakehouse adds an open table format such as Delta Lake, Iceberg or Hudi, which brings ACID transactions, schema enforcement and row-level updates to that same storage.
How do I choose a partition column?
Pick the column that appears in most query filters and has moderate cardinality, which is usually a date. Avoid high-cardinality keys such as user ID, which create huge numbers of small partitions; bucket on those instead.
Why are many small files a problem?
Each file adds open, scheduling and metadata overhead, and in HDFS each consumes NameNode memory. Queries over thousands of tiny files spend more time on overhead than on reading data, so compaction into larger files is the fix.
What question style does this domain use?
Mostly scenario questions that describe a dataset, its query patterns and a symptom such as slowness or cost, then ask for the BEST layout or fix. Some questions check terms such as predicate pushdown or external tables.
How does storage relate to data modeling?
Domain 5 decides the logical shape of tables, such as a star schema, while this domain decides how those tables are physically stored, partitioned and kept over time. Exam scenarios often need both views at once.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 2 knowledge
You have covered warehouse, lake and lakehouse designs, partitioning and bucketing, file formats, lifecycle tiers and query tuning. Test whether you can pick the right layout from a scenario's query patterns.