Spark Scala Shift Functions: shiftleft, shiftright, and shiftrightunsigned

The bitwise shift functions move the bits of an integer column left or right by a fixed number of positions. They're useful for packing and unpacking flags, fast multiplication or division by powers of two, and working with binary protocol data.

The shiftleft, shiftright, and shiftrightunsigned functions first appeared in version 3.2.0 and are defined as:

def shiftleft(e: Column, numBits: Int): Column

def shiftright(e: Column, numBits: Int): Column

def shiftrightunsigned(e: Column, numBits: Int): Column

The first argument is the column to shift and the second is the number of bit positions to shift by — a plain Scala Int, not a Column.

Shifting Left

shiftleft moves every bit one position to the left for each bit in numBits, filling the new low-order bits with zeros. Each shift left by one position is equivalent to multiplying by two.

val df = Seq(1, 3, 8, 16, 255).toDF("value")

val df2 = df
  .withColumn("shift_1", shiftleft(col("value"), 1))
  .withColumn("shift_2", shiftleft(col("value"), 2))
  .withColumn("shift_4", shiftleft(col("value"), 4))

df2.show(false)
// +-----+-------+-------+-------+
// |value|shift_1|shift_2|shift_4|
// +-----+-------+-------+-------+
// |1    |2      |4      |16     |
// |3    |6      |12     |48     |
// |8    |16     |32     |128    |
// |16   |32     |64     |256    |
// |255  |510    |1020   |4080   |
// +-----+-------+-------+-------+

Shifting 1 left by 4 gives 16 (2⁴), and shifting 255 left by 2 gives 1020 (255 × 4). The return type matches the input type, so very large shifts on Int columns can overflow — cast to Long first if you need more headroom.

Shifting Right

shiftright performs a signed (arithmetic) right shift. The low-order bits are discarded and the sign bit is replicated into the new high-order positions, so negative numbers stay negative. Each shift right by one position is equivalent to integer division by two, rounded toward negative infinity.

val df = Seq(64, 100, 255, -8, -64).toDF("value")

val df2 = df
  .withColumn("shift_1", shiftright(col("value"), 1))
  .withColumn("shift_2", shiftright(col("value"), 2))
  .withColumn("shift_3", shiftright(col("value"), 3))

df2.show(false)
// +-----+-------+-------+-------+
// |value|shift_1|shift_2|shift_3|
// +-----+-------+-------+-------+
// |64   |32     |16     |8      |
// |100  |50     |25     |12     |
// |255  |127    |63     |31     |
// |-8   |-4     |-2     |-1     |
// |-64  |-32    |-16    |-8     |
// +-----+-------+-------+-------+

Notice that 100 >> 3 returns 12, not 12.5 — the fractional bits are dropped. For negative values, -8 >> 3 returns -1 because the sign bit is preserved through each shift.

Signed vs Unsigned Right Shift

shiftrightunsigned is identical to shiftright for non-negative values but treats the bits as unsigned for negative inputs — the new high-order bits are filled with zeros instead of the sign bit. The result is a large positive number rather than a small negative one.

val df = Seq(8, 64, -1, -8, -64).toDF("value")

val df2 = df
  .withColumn("signed", shiftright(col("value"), 2))
  .withColumn("unsigned", shiftrightunsigned(col("value"), 2))

df2.show(false)
// +-----+------+----------+
// |value|signed|unsigned  |
// +-----+------+----------+
// |8    |2     |2         |
// |64   |16    |16        |
// |-1   |-1    |1073741823|
// |-8   |-2    |1073741822|
// |-64  |-16   |1073741808|
// +-----+------+----------+

For -1, the 32-bit two's-complement representation is all ones (0xFFFFFFFF). A signed shift right by 2 keeps it all ones (still -1), while an unsigned shift right by 2 produces 0x3FFFFFFF — the maximum positive Int shifted down, which is 1073741823. Use shiftrightunsigned when you're treating the column as a raw bit pattern rather than a signed number, such as when parsing packed binary fields.

Related Functions

For viewing or producing the binary representation of integers, see bin and conv. For converting between hexadecimal strings and bytes, see hex and unhex.