Job Board
Consulting

Data Quality Testing with Deequ and Spark Scala

Deequ is an AWS-built library that lets you declare data quality constraints — completeness, uniqueness, value ranges, statistical bounds — as Scala code and run them as Spark jobs. Think of it as unit tests for the rows flowing through your pipeline. It's a genuinely useful tool for Spark Scala teams, with one sharp edge you need to know before you add it to your build: it ships for Scala 2.12 only.

The pitch: unit tests for your data

You already test your transformation logic. You have a suite that feeds a small hand-built DataFrame through a function and asserts on the output. That catches bugs in your code. It does nothing about the data that shows up at runtime — the upstream team that started sending null in a column that was never nullable, the join that silently doubled your row count, the currency field that arrived in cents instead of dollars.

Deequ closes that gap. You declare what "good" looks like — product_id is complete and unique, numViews is non-negative, priority is always one of a known set — and Deequ compiles those declarations into a single optimized Spark pass over the data. The output is a pass/fail report per constraint, computed at full scale on the actual data, not a sample.

The library has three main pieces:

  • Constraint verification — assert that data meets expectations (VerificationSuite + Check).
  • Metrics computation — profile a dataset and compute metrics without pass/fail assertions (AnalysisRunner + analyzers).
  • Constraint suggestion — let Deequ inspect a dataset and propose constraints for you.

On top of those sit a metrics repository for persisting results over time and an anomaly-detection layer that flags metrics drifting outside historical bounds.

Adding Deequ to your build — read this part first

Deequ's version string is unusual: it encodes the Spark minor version it was built against. There is no separate %% Scala-version suffix, because every Deequ artifact is published for Scala 2.12 only. As of August 2026 the latest releases are 3.0.3-spark-3.5 (also 2.0.21-spark-3.5) for Spark 3.5, and 2.0.12-spark-3.4 for Spark 3.4. Pick the one that matches your Spark minor:

// build.sbt — note the SINGLE % , not %%
// Deequ has no _2.12 / _2.13 suffix, so let sbt take the artifact id verbatim.

// For a Spark 3.5 project:
libraryDependencies += "com.amazon.deequ" % "deequ" % "2.0.21-spark-3.5"

// For a Spark 3.4 project (this site's demo stack runs 3.4.1):
libraryDependencies += "com.amazon.deequ" % "deequ" % "2.0.12-spark-3.4"

// Deequ pulls in spark-core/spark-sql/spark-mllib transitively. Exclude them so
// they don't fight the "provided" Spark on your cluster:
libraryDependencies += ("com.amazon.deequ" % "deequ" % "2.0.21-spark-3.5")
  .exclude("org.apache.spark", "spark-core_2.12")
  .exclude("org.apache.spark", "spark-sql_2.12")

Why the single % matters: %% would make sbt look for deequ_2.13 or deequ_2.12, and neither artifact id exists on Maven Central. You'd get an unresolved-dependency error that looks like a typo but is really a naming-convention mismatch.

The Scala 2.12 restriction is not a footnote. Inspecting the POM for even the newest 3.0.3-spark-3.5 release shows scala.major.version = 2.12 and a dependency on spark-core_2.12. That has two consequences worth stating plainly:

  • If your Spark Scala project is already on Scala 2.13, you cannot add Deequ as a normal JVM dependency — the transitive _2.12 Spark artifacts will collide with your _2.13 ones. See Scala 3 and Spark in 2026 for where the language-version lines are drawn.
  • Because Spark 4.0 dropped Scala 2.12 entirely and is 2.13-only, Deequ does not run on a Spark 4.0 cluster as a compiled dependency today. If you're on Spark 4.0, Deequ is effectively a Spark-3.x-only tool for you until AWS publishes a 2.13 build. This is the single most important thing to check before you commit to it.

Your first verification suite

Here is the shape of a real check. A Check groups related constraints and carries a level — Error fails the suite, Warning records the violation without failing.

import com.amazon.deequ.{VerificationSuite, VerificationResult}
import com.amazon.deequ.checks.{Check, CheckLevel, CheckStatus}
import com.amazon.deequ.constraints.ConstraintStatus

// `reviews` is an ordinary Spark DataFrame.
val verificationResult: VerificationResult = VerificationSuite()
  .onData(reviews)
  .addCheck(
    Check(CheckLevel.Error, "core review constraints")
      .hasSize(_ >= 1500000)                                  // at least 1.5M rows
      .isComplete("product_id")                              // no nulls
      .isUnique("product_id")                                // primary key holds
      .isComplete("review_id")
      .isContainedIn("marketplace", Array("US", "UK", "DE", "JP", "FR"))
      .isNonNegative("star_rating")
      .hasMax("star_rating", _ <= 5.0))
  .addCheck(
    Check(CheckLevel.Warning, "soft quality signals")
      .hasCompleteness("review_body", _ >= 0.99)             // 99% non-null is fine
      .hasApproxQuantile("helpful_votes", 0.5, _ <= 10))     // median under 10
  .run()

Everything in that suite compiles down to one Spark job. Deequ shares scans and aggregations across constraints where it can, so adding a tenth check to an existing Check is far cheaper than running ten independent df.filter(...).count() assertions yourself.

The constraint vocabulary is broad: isComplete / hasCompleteness, isUnique / hasUniqueness, isContainedIn, isNonNegative / isPositive, hasMin / hasMax / hasMean / hasSum, hasApproxQuantile, hasPattern (regex), hasDataType, and satisfies for an arbitrary SQL predicate over a fraction of rows.

Reading the results

VerificationResult carries an overall status plus per-constraint detail. The idiomatic pattern is: bail early on success, otherwise print exactly which constraints failed and why.

import com.amazon.deequ.VerificationResult.checkResultsAsDataFrame

if (verificationResult.status == CheckStatus.Success) {
  println("Data passed all checks.")
} else {
  val failures = verificationResult.checkResults
    .flatMap { case (_, checkResult) => checkResult.constraintResults }
    .filter { _.status != ConstraintStatus.Success }

  failures.foreach { r =>
    println(s"FAILED: ${r.constraint} — ${r.message.getOrElse("")}")
  }
}

// Or materialize the whole report as a DataFrame for logging / storage:
val resultDf = checkResultsAsDataFrame(spark, verificationResult)
resultDf.show(truncate = false)

checkResultsAsDataFrame is the hook you want for production: write that DataFrame to a table and you have an auditable history of every quality run, queryable with the same Spark SQL you use for everything else. In a pipeline you typically branch on verificationResult.status — halt the job and page someone on Error, log and continue on Warning.

Profiling with the AnalysisRunner

Sometimes you don't have expectations yet — you have a new dataset and want to characterize it before writing any assertions. That's the AnalysisRunner. It computes raw metrics with no pass/fail attached.

import com.amazon.deequ.analyzers.runners.{AnalysisRunner, AnalyzerContext}
import com.amazon.deequ.analyzers.runners.AnalyzerContext.successMetricsAsDataFrame
import com.amazon.deequ.analyzers.{Size, Completeness, ApproxCountDistinct,
  Mean, StandardDeviation, Compliance, Uniqueness}

val analysis: AnalyzerContext = AnalysisRunner
  .onData(reviews)
  .addAnalyzer(Size())
  .addAnalyzer(Completeness("product_id"))
  .addAnalyzer(ApproxCountDistinct("product_id"))
  .addAnalyzer(Mean("star_rating"))
  .addAnalyzer(StandardDeviation("star_rating"))
  .addAnalyzer(Compliance("top rated", "star_rating >= 4.0"))
  .addAnalyzer(Uniqueness("review_id"))
  .run()

val metrics = successMetricsAsDataFrame(spark, analysis)
metrics.show(truncate = false)

The analyzers map onto statistics you may already be computing by hand — Mean, StandardDeviation, and friends are the same measures covered in the standard deviation and variance and skewness and kurtosis examples, and ApproxCountDistinct is the fast cousin of countDistinct. The difference is that Deequ batches them into one shared pass and hands you a tidy metrics DataFrame instead of a widening .agg(...) call.

Let Deequ write the checks for you

For a large, unfamiliar table, hand-authoring constraints is tedious. ConstraintSuggestionRunner profiles the data and proposes constraints — and, helpfully, emits the Scala code for each one so you can paste the ones you agree with straight into a Check.

import com.amazon.deequ.suggestions.{ConstraintSuggestionRunner, Rules}
import spark.implicits._

val suggestions = ConstraintSuggestionRunner()
  .onData(reviews)
  .addConstraintRules(Rules.DEFAULT)
  .run()

// Each suggestion carries a human description AND the code to implement it.
val asRows = suggestions.constraintSuggestions.flatMap {
  case (column, columnSuggestions) =>
    columnSuggestions.map { s => (column, s.description, s.codeForConstraint) }
}.toSeq.toDS()

asRows.show(truncate = false)

Treat the output as a first draft, not gospel. Suggestion rules infer constraints from whatever sample they see, so they'll cheerfully propose isComplete on a column that just happens to have no nulls today. Review each one against what you actually know about the domain before committing it.

Tracking quality over time

A single run tells you whether the data is good right now. The more valuable question is whether it's drifting. Deequ answers that with a metrics repository plus anomaly checks: persist each run's metrics, then assert that the next run's metrics haven't moved too far from the recent history.

import com.amazon.deequ.analyzers.Size
import com.amazon.deequ.repository.ResultKey
import com.amazon.deequ.repository.memory.InMemoryMetricsRepository
import com.amazon.deequ.anomalydetection.RelativeRateOfChangeStrategy

// In production use FileSystemMetricsRepository backed by S3/HDFS so history
// survives across job runs; InMemory is for illustration.
val repository = new InMemoryMetricsRepository()
val today = ResultKey(runTimestampMillis, Map("dataset" -> "reviews"))

VerificationSuite()
  .onData(reviews)
  .useRepository(repository)
  .saveOrAppendResult(today)
  // Fail if row count grew by more than 2x versus the previous run:
  .addAnomalyCheck(
    RelativeRateOfChangeStrategy(maxRateIncrease = Some(2.0)),
    Size())
  .run()

This is where Deequ earns its keep on a mature pipeline. A row count that doubles overnight, a completeness rate that quietly slips from 99% to 91%, a mean that jumps three standard deviations — these are the failures that pass every static constraint and still ruin a downstream model. Anomaly checks catch the change, not just the absolute value.

Where Deequ fits — and where it doesn't

Deequ is worth reaching for when:

  • You're already a Spark shop and want data quality checks that run at full scale in the same JVM, no extra service.
  • You want quality metrics as data — an auditable, queryable table of every run — rather than log lines.
  • You need statistical and historical checks (quantiles, anomaly detection) that are painful to hand-roll.

Be honest about the constraints:

  • Scala 2.12 only. Restating the headline because it's the deciding factor: no 2.13 build exists, so Deequ is a non-starter on Scala 2.13 projects and on Spark 4.0 clusters as a compiled dependency. On Spark 4.0 your realistic options are staying on a Spark 3.5 job for the quality stage, or reaching for a framework-agnostic tool. Tools like Great Expectations and Soda work against Spark but sit outside the JVM, which trades the Scala-version problem for a Python-in-your-pipeline one.
  • It's a batch tool. Deequ profiles a DataFrame. It's not built to sit inside a low-latency streaming path, though you can run it per micro-batch if you accept the cost.
  • AWS-maintained, community-paced. Deequ is active but not fast-moving; don't expect same-week support for brand-new Spark releases.

For the wider picture of which Spark Scala tools are thriving in 2026, see The State of Spark Scala in 2026.

Checklist: adding Deequ to a Spark Scala project

  • [ ] Confirm your project is on Scala 2.12 and Spark 3.x (not 2.13 / Spark 4.0).
  • [ ] Add "com.amazon.deequ" % "deequ" % "<version>-spark-<your-minor>" with a single %.
  • [ ] Exclude Deequ's transitive spark-core / spark-sql so they don't override your provided Spark.
  • [ ] Start with AnalysisRunner to profile an unfamiliar dataset.
  • [ ] Use ConstraintSuggestionRunner to draft constraints, then review each one by hand.
  • [ ] Encode agreed constraints in a VerificationSuite, splitting Error (halt) from Warning (log).
  • [ ] Persist results with checkResultsAsDataFrame and a metrics repository for history.
  • [ ] Add anomaly checks on the metrics that matter (size, completeness, key means) to catch drift.

The code snippets above are illustrative — adapt the version strings and column names to your project. Start with the Deequ README and examples on GitHub, and read the AWS Big Data blog post Test data quality at scale with Deequ for the design rationale behind the analyzer model.

Article Details
More Spark Scala News and Updates

Created: 2026-08-04

Last Updated: 2026-08-04 10:10:17 PM