AI infrastructure

Databricks

The lakehouse platform where Spark, Delta Lake and MLflow carry other companies' production data and model workloads.

Overview

Databricks sells an operated data platform: tables in open storage formats through Delta Lake, query and job execution on Spark and the Photon engine, governance in Unity Catalog, and a machine learning and generative AI layer with MLflow at its centre. Much of the foundation is open source and the commercial product is the multi-tenant, multi-cloud version of it, which sets the engineering culture. Correctness under concurrent writers, cost per query, and behaviour when a job meets data ten times larger than the sample it was written against are ordinary daily concerns.

Work splits roughly into the data plane — query optimisation, execution, storage layout — the control plane services that schedule and secure it, and the model side covering serving, retrieval, evaluation and agent tooling. What links them is the customer: a data or machine learning engineer with a pipeline already in production. Developer experience is therefore treated as a real constraint, because an interface change that forces a rewrite of somebody's nightly job has a measurable cost.

Candidates are expected to be fluent in data at scale rather than in models alone. Skew, shuffle, partitioning, file layout, incremental processing and the difference between a slow query and a bad plan come up naturally, and the machine learning questions lean toward lifecycle — tracking, reproducibility, deployment, evaluation — rather than novel architectures.

What Databricks visibly values

Inferred from public material, not from a leaked rubric. Treat these as the themes worth emphasising rather than as a scoring sheet.

Distributed execution reasoning: reading a physical plan, spotting skew or an avoidable shuffle, and knowing which join strategy the optimiser chose and why.
Storage-format literacy — Parquet layout, file sizing and compaction, and what the Delta transaction log gives you when two writers commit at once.
Preferring incremental processing to reprocessing: change data capture, idempotent writes, watermarks, and a backfill plan that does not corrupt the table.
Cost accounting stated in units — bytes scanned, cluster hours, dollars per run — rather than in adjectives about efficiency.
Lifecycle discipline around models: experiment tracking, a registry, reproducible runs, and an evaluation set that exists before the model ships.
Care for the operator on the other side of the API, including migration paths, deprecation windows and defaults that do not silently change a customer's results.
Visible work in the surrounding open source — Spark, Delta Lake, MLflow — where the design discussions are public and readable.

Roles this kind of team hires

Linked titles have a full question bank with model answers. Unlinked ones are titles this employer uses that do not map onto one of our role guides.

How to prepare

Make a slow job fast and be able to narrate it

Take a Spark job that actually runs badly, read its physical plan and stage metrics, find the skew or shuffle causing it, change one thing, and quote the before-and-after runtime. That story is worth more than reciting the optimiser's rules.

Understand the table format, not just the API

Be able to explain what the Delta transaction log stores, how optimistic concurrency resolves two simultaneous writers, why small files hurt, and what compaction and time travel cost you in storage and in query latency.

Bring one end-to-end lifecycle story

Walk through a model or an agent you took from experiment to production: what you tracked, how you versioned data and prompts, what gate it had to pass before release, and how you noticed when it regressed afterwards.

Price the workload out loud

Practise sizing a cluster and estimating the cost of a pipeline run, including how autoscaling and spot capacity change the number. Engineers here are used to customers who read their bill line by line.

Design one interface as if strangers depend on it

Sketch an API for something you have built and then argue against yourself about a breaking change: who it affects, what the deprecation path is, and what you would ship to make the migration mechanical rather than manual.

What to watch for

Common mistakes with this kind of employer

Talking fluently about models while treating the pipeline that feeds them as somebody else's problem. Here the pipeline usually is the problem.
Carrying a single-node mental model into a distributed question. Answers that quietly collect a large dataset onto the driver, or assume the data fits in memory, are noticed immediately.
Knowing the managed product but never having looked at the open formats and engines under it, which is where most of the interesting design questions live.
Answering a data quality or evaluation question with intent rather than a mechanism. Saying you would monitor quality without naming what you would compute and what threshold trips ends the thread.
Assuming a rewrite is free. Proposing to replace a customer-facing behaviour without a migration story reads as inexperience with a platform other people build on.

Sources

This guide is written from primary public material — Databricks's own engineering writing, research pages, published job descriptions, and public talks. Nothing here is taken from a competitor's interview guide, and no specific claim about the loop is made without a source that supports it.

Last reviewed 31 August 2026. Interview processes change without notice and vary by team, level, and location. Databricks's own careers page is the authoritative description of how it hires; where this guide and that page disagree, that page is right.

Rehearse it before it counts.

Knowing what a company values is not the same as demonstrating it under time pressure. Book a 30-minute 1:1 and run the round you are least confident about.

Book a mock interview