Publishing sbt Artifacts to a Private Maven Repository

Once a Spark team extracts shared code — schema definitions, custom UDFs, internal connectors — into its own library, that library has to live somewhere the rest of the org can resolve it from. This tutorial covers the sbt side of publishing to a private Nexus or Artifactory: what sbt publish actually produces, how to set the coordinates and publishTo, how to test the whole thing locally before you touch a real server, and the POM settings that make a published artifact clean to consume.

What `sbt publish` Actually Does

Before configuring anything, it helps to know what a publish produces. When you run sbt publish, sbt uploads a small set of files to a repository under a path derived from your project's coordinates:

com/acme/spark-data-utils_2.13/1.4.0/
  spark-data-utils_2.13-1.4.0.jar          // the compiled classes
  spark-data-utils_2.13-1.4.0.pom          // dependencies + metadata
  spark-data-utils_2.13-1.4.0-sources.jar  // source jar (optional)
  spark-data-utils_2.13-1.4.0-javadoc.jar  // docs jar (optional)
  ...plus .md5 / .sha1 checksums for each

The POM is the important one. It's an XML file listing your library's own dependencies and its identifying metadata. When a downstream project resolves your library, its build tool reads your POM to discover the transitive dependencies it also needs to pull in. A jar with a wrong or missing POM resolves but then fails at compile time with missing symbols, so most of the configuration below is really about producing a correct POM.

The path — com/acme/spark-data-utils_2.13/1.4.0/ — comes directly from three settings: your organization, the project name (with the Scala binary suffix appended), and the version.

Step 1: Set the Coordinates

Maven identifies every artifact by three coordinates: group (organization in sbt), artifact (name), and version. Set them explicitly in build.sbt — the defaults sbt picks for a fresh project are almost never what you want published:

// build.sbt
organization := "com.acme"
name         := "spark-data-utils"
version      := "1.4.0"
scalaVersion := "2.13.11"

A downstream project then declares the dependency with those exact coordinates:

// downstream build.sbt
libraryDependencies += "com.acme" %% "spark-data-utils" % "1.4.0"

The %% operator appends the Scala binary version (_2.13) to the artifact name automatically, which is why the published path contains spark-data-utils_2.13 rather than a bare spark-data-utils. This matters for Spark libraries in particular: a jar compiled against Scala 2.12 is not binary-compatible with a 2.13 cluster, so the suffix is what keeps consumers from resolving the wrong build. Cross-publishing for both versions is covered in the GitHub Actions publishing tutorial; for now, a single scalaVersion is enough to see publishing work end to end.

Step 2: Point publishTo at the Repository

publishTo tells sbt where to send the artifacts. Point it at the deploy path of your internal repository — the same Nexus or Artifactory host you resolve from, but the URL that accepts writes:

// build.sbt
publishTo := {
  val nexus = "https://nexus.acme.internal/repository/"
  if (isSnapshot.value)
    Some("Acme Snapshots" at nexus + "maven-snapshots/")
  else
    Some("Acme Releases" at nexus + "maven-releases/")
}

The isSnapshot.value check routes by version type. A version ending in -SNAPSHOT (like 1.4.0-SNAPSHOT) is mutable — you can overwrite it repeatedly while iterating. A plain version (1.4.0) is a release, which most repositories accept exactly once and then lock. The two live on separate paths, and sending a release to the snapshots path or vice versa usually fails server-side, so let the version itself pick the destination.

The string before at — "Acme Snapshots" — is the resolver name. sbt uses it in log output and, crucially, to match against your credentials. It doesn't have to match the resolver name you use for reading; it just has to be present.

Step 3: Supply Credentials

Publishing writes to the server, so it always needs credentials — and they should come from a service account with write permission, not a personal login. The credentials shape is identical to the read side, covered in depth in the private Maven repositories tutorial. The short version: sbt sends credentials only when the realm and host match what the server expects.

For a developer publishing manually, the file-based form works:

// ~/.sbt/.credentials
realm=Sonatype Nexus Repository Manager
host=nexus.acme.internal
user=svc-spark-publish
password=abc123-write-token

// build.sbt
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

Two constraints trip people up on the write path specifically:

The realm must match the server's WWW-Authenticate header exactly. A mismatch means sbt never sends the credentials, and the upload fails with a 401 that reads like a network error. Find the exact string with curl -v against any artifact path and copy it verbatim.
The account needs write permission. Read-only credentials sail through sbt update but fail at upload time with a 403 on the PUT request. It's easy to misread that as a connectivity problem when it's really a permissions one.

Step 4: Round Out the POM

The defaults sbt uses for publish lean toward publishing to Maven Central, and a few of them are wrong for an internal repo. Set these once and forget them:

// build.sbt
publishMavenStyle      := true
Test / publishArtifact := false
pomIncludeRepository   := { _ => false }

publishMavenStyle := true writes a Maven-layout POM, which is the format every Nexus/Artifactory/CodeArtifact instance expects. Without it, sbt publishes Ivy-style metadata that Maven-based consumers can't read.
Test / publishArtifact := false stops sbt from publishing your test jar. Internal libraries almost never expose their tests, and publishing them just adds noise and drags test-only dependencies into the POM.
pomIncludeRepository := { _ => false } keeps your internal resolver URLs out of the published POM. Otherwise consumers inherit references to nexus.acme.internal and get confusing resolution failures if they don't have access to the same hosts.

For a Spark library, one more setting belongs here: keep Spark itself on the provided scope so it doesn't leak into your POM as a hard dependency. Every cluster already ships Spark; baking a specific Spark version into your library's dependency graph only causes version conflicts downstream.

// build.sbt
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.4.1" % "provided"

provided dependencies are available when you compile and test your library, but sbt omits them from the published POM — exactly the behavior you want for Spark.

Step 5: Publish Sources and Docs (Optional but Recommended)

By default sbt publishes only the main jar and the POM. For an internal library that other engineers will read and step into with a debugger, the sources jar is worth its weight — it's what lets an IDE show your library's source instead of decompiled bytecode. sbt can attach both a sources and a docs jar:

// build.sbt
Compile / packageSrc / publishArtifact := true   // sources jar
Compile / packageDoc / publishArtifact := true   // scaladoc jar

The sources jar is cheap and high-value. The docs jar is slower to build (it runs Scaladoc) and often skipped for internal-only libraries; turn it off if it's slowing your publishes down and nobody consumes the docs.

Step 6: Test It Locally First

You don't need a running Nexus to shake out publishing problems. sbt can publish into your local caches, which is the fastest way to verify the coordinates, POM, and artifacts are right before pointing at a real server.

publishLocal writes to the local Ivy cache (~/.ivy2/local), where other sbt projects on the same machine can resolve it:

sbt publishLocal

publishM2 writes to the local Maven cache (~/.m2/repository) instead — useful when the consuming project is Maven-based, or when you want to inspect the produced files by hand:

sbt publishM2

After either command, look at what was actually produced:

ls ~/.ivy2/local/com.acme/spark-data-utils_2.13/1.4.0/
// ivys/  jars/  poms/  srcs/

Open the POM and confirm it lists the dependencies you expect — and that Spark is absent (because it's provided) and your test framework is absent (because Test / publishArtifact := false and test-scoped deps stay out of the compile POM). Getting this right locally means the real publish is just a change of destination.

To prove resolution end to end, add the dependency to a throwaway project and run sbt update against the local cache. If it resolves, the artifact is well-formed.

Step 7: Publish for Real

With publishTo and credentials in place, publishing is one command:

sbt publish

sbt compiles, packages the jar (plus sources/docs if enabled), generates the POM, and PUTs all of it to the resolver publishTo selected. Watch the log for the destination host — if it uploads to nexus.acme.internal, you're done. Confirm the artifact landed by fetching its POM directly:

curl -u "$NEXUS_USER:$NEXUS_PASSWORD" \
  https://nexus.acme.internal/repository/maven-releases/com/acme/spark-data-utils_2.13/1.4.0/spark-data-utils_2.13-1.4.0.pom

A 200 with the POM XML means the publish worked. A 404 means the upload didn't land where you expected — usually a release/snapshot path mismatch or a wrong Scala suffix.

Snapshot vs Release Discipline

The isSnapshot.value routing in Step 2 only helps if you actually move the version between snapshot and release states. The convention that keeps things sane:

While developing, keep the version at 1.4.0-SNAPSHOT. Publish freely; each sbt publish overwrites the previous snapshot. Downstream projects that opt into the snapshot get your latest work.
When you cut a release, drop the -SNAPSHOT suffix, publish 1.4.0 once, then bump to the next snapshot (1.5.0-SNAPSHOT). The release is now immutable — anyone resolving 1.4.0 gets exactly the bytes you published, forever.

One caveat that bites consumers, not publishers: Ivy caches snapshots by their version string. If you overwrite 1.4.0-SNAPSHOT repeatedly, a downstream project may keep resolving a stale copy from its cache. That's a strong argument for embedding a commit SHA or timestamp in snapshot versions once you automate this — which is exactly what the CI setup does.

Common Mistakes

A release version that already exists. Most repos reject a second PUT to an existing release path with a 409 Conflict or 400. Bump the version instead of trying to overwrite a release — that immutability is the point.
Spark baked into the POM. Forgetting % "provided" on Spark publishes a library that drags a specific Spark version into every consumer's dependency tree, causing conflicts on clusters running a different build. See the provided scope tutorial.
Read-only credentials. They pass sbt update and fail at upload with a 403. The error names the failing PUT, but it's easy to mistake for a network issue.
Wrong publishMavenStyle. Left at its Ivy default, the artifact publishes but Maven-based consumers can't read the metadata. Set it to true for any Maven-compatible repository.
Publishing the test jar. Without Test / publishArtifact := false, your test-only dependencies (utest, scalatest) leak into the picture and the registry fills with jars nobody consumes.

Next Steps

Publishing manually from a laptop is the right way to learn the mechanics, but it's the wrong way to operate. Once the config above works, the natural progression is:

Move publishing into CI. Laptop publishes cause version drift and credential sprawl. The GitHub Actions publishing tutorial shows how to wire publishTo, env-based credentials, git-derived versioning, and cross-Scala publishing into a reproducible workflow.
Verify the consumer side. Publishing is only half the contract. The private Maven repositories tutorial covers the resolver and credential setup a downstream Spark project needs to pull your library back down.
Get it onto the cluster. If you run jobs with spark-submit --packages, the cluster resolves your library through Ivy and needs its own credentials — see using spark-submit with private Maven dependencies. If you bundle everything into a fat jar instead, assembly merge strategies are what you'll configure next.

The build.sbt settings here are a one-time investment. After that, publishing an internal Spark library is no different from publishing any other Scala artifact — a single command that lands a clean, resolvable jar in your team's repository.