It’s a thing you commonly hear in programming communities dedicated to languages that have macros. “Macros are the tool of last resort”, they say. “You never need a macro”.

I think that macros are misunderstood as a result of having been misused by misguided souls who used macros to solve problems that could have been solved by normal language constructs. Programmer communities are surprisingly susceptible to the proliferation of memes, and so the anti-macro sentiment became a meme as a result of a few bad experiences at some point in the past.

My friend and former colleague Brendan McAdams inspired me recently to learn Scala macros. As is common, Brendan gave a thought-provoking talk (PDF) that, well, provoked some thought. You see, previously I was one of those people who unthinkingly propagated the anti-macro meme, despite having had zero personal experience with macros — in Scala or anywhere. So, as Brendan’s slides rolled by on my screen, I decided to set out on a macro adventure of my own, in search of truth regarding macros in Scala.

After a couple of weeks spent writing, debugging and reading macro code, I’m coming to the conclusion that there are only two categories of problems that can benefit from macros:

This conclusion crystallized in my mind while I wrote two macro-based projects. One — a better enumeration library for Scala called numerato — falls mostly in the Class A space. The other — a event sourcing/CQRS library that I haven’t formally published yet — is squarely within Class B.

In this post we’ll take a look at numerato in an attempt to understand the problem it tries to solve, as well as the role of Scala macros in implementing the solution.

A grievous hole

Anyone coming to Scala from Java will be unpleasantly surprised by the absence of a good enumeration facility in Scala. Java has one, yet Scala offers “only” ADTs, which are powerful enough, but are a poor tool to use in emulating a proper enumeration. ADTs are great for creating hierarchies of values that can be sealed against intrusion from outside (i.e., a new value cannot be introduced post-definition), and provide a small degree of safety in verifying exhaustiveness of matches. Scala’s compiler will helpfully emit warnings when a match involving a sealed type hierarchy omits some of the possible values.

Java enumerations, however, go a bit further than this. A Java enum provides facilities for enumerating (no pun intended) the enum’s values at runtime by calling the static value() method. Another useful feature is being able to convert an enum value to and from its integer ordinal index. Java enums even edge into ADT-like territory by allowing enum values to carry custom fields and methods. I think we can all agree that — used properly — a Java enum can be quite powerful.

Scala offers a much more limited enumeration as part of its standard library. The astute reader will already notice a crucial aspect of Scala enumerations, one that makes them a lot less useful than Java’s: in Java, enumerations are part of the language, whereas in Scala, a mere stdlib feature implements an enumeration-like half-solution that is awkward to use and is not very useful to begin with.

Critically speaking, my main beef with both Scala ADTs and enumerations is that there is no way to guarantee — and I use this word in the strongest possible sense — at compile time that a match involving an enumeration or ADT will absolutely, positively be exhaustive. I say guarantee because I have found through painful experience that a mere warning is often woefully insufficient. It is my firm belief that discipline is great, but enforcement of discipline that itself relies on more discipline is doomed to fail.

Filling the hole

So, I set out to dream up a better Scala enumeration facility. I started with the syntax:

class Status {
  val Enabled, Disabled = Value
}

Here we’ve defined an enumeration named Status, with two possible values: Enabled and Disabled. To ease transition from traditional Scala enumerations, I’ve decided to keep the declaration syntax.

In case you’re wondering, a Java equivalent of the above idealized enum would look something like this:

public enum Status {
    ENABLED, DISABLED
}

Considering these two code snippets side by side, I think it’s pretty obvious that we’re attempting to create a missing language feature, making our quest fall squarely in Class A territory.

Our new enum implementation must meet the following goals:

With this in mind we can start laying down the foundation.

Filler material

What would it take to hit the proposed goals? Let’s start with the first: we already know that Scala’s own exhaustiveness checking relies on a sealed type hierarchy. Armed with this knowledge we can rewrite our idealized enum syntax in plain Scala:

// @enum class Status {
sealed abstract class Status
//  val Enabled, Disabled = Value
case object Enabled extends Status
case object Disabled extends Status
// }

Next, we promised to provide an ordinal index and readable name on each enum value. This is also easy to achieve:

// @enum class Status {
sealed abstract class Status(val index: Int, val name: String)
//  val Enabled, Disabled = Value
case object Enabled extends Status(index = 0, name = "Enabled")
case object Disabled extends Status(index = 1, name = "Disabled")
// }

Finally - and before we tackle exhaustive matches - we can add the promised iteration and lookup functionality:

// @enum class Status {
sealed abstract class Status(val index: Int, val name: String)
//  val Enabled, Disabled = Value
case object Enabled extends Status(index = 0, name = "Enabled")
case object Disabled extends Status(index = 1, name = "Disabled")
// }
object Status {
  val values = List[Status](Enabled, Disabled)
  def fromIndex(index: Int): Status = index match {
    case 0 => Enabled
    case 1 => Disabled
  }
  def fromName(name: String): Status = name match {
    case "Enabled" => Enabled
    case "Disabled" => Disabled
  }
}

We’ve gradually translated a sort of idealized syntax implementing a missing Scala feature into a naive-but-workable implementation. There’s only one problem with this: we’ve done the work manually, while Java does this automatically on the programmer’s behalf.

Thus our goal becomes to automate the above process of implementing a language feature. This is where macros come in.

Abstract syntax tree hugger

In technical terms, the transformation of code we’ve effected above is just another AST transform: we’ve taken one Scala syntax tree and produced another. This is exactly what macros are for; in fact, macro inputs and outputs are all in the form of Scala ASTs.

Our task is thus to write code which will transform the above class Status { … } declaration into some AST representing the final result we’ve produced manually. If this sounds familiar, that’s because it is: we need to write a “function” of sorts, which will take a Scala AST representing class Status { … }, and will return another AST representing the transformed result.

Such a function can take the form of a macro annotation. Our idealized syntax thus becomes:

@enum class Status {
  val Enabled, Disabled = Value
}

Let’s go ahead and lay down the boilerplate foundation for this @enum annotation:

import scala.annotation.StaticAnnotation
import scala.reflect.macros.whitebox

class enum(debug: Boolean = false) extends StaticAnnotation {
  def macroTransform(annottees: Any*): Any = macro EnumMacros.decl
}

class EnumMacros(val c: whitebox.Context) {
  import c.universe._

  def decl(annottees: Tree*): Tree = {
    ??? // HALP I'M LOST
  }
}

See that def decl(annottees: Tree\*): Tree? This is our AST transformation function: that’s where we’ll do all of the work.

Let’s think back to our idealized syntax: on some level, we can think of it as just a bunch of text containing pieces we want to extract, then re-arrange and otherwise re-use in making some new piece of text, er, I mean code. It’s almost like turning some text into a template-ized version of itself. This intuition is very helpful in writing Scala macros because of one killer feature: quasiquotes.

Here I encourage you to view (or review) the slides from Brendan’s talk. He gives a good explanation of quasiquotes there, which boils down to “AST templating”, and wouldn’t be as clear coming from me.

So, armed to the teeth with quasiquotes, we can write down a template-like structure that will allow us to extract useful pieces from the input AST, and then construct a new AST containing all the new generated goodness:

import scala.annotation.StaticAnnotation
import scala.reflect.macros.whitebox

class enum(debug: Boolean = false) extends StaticAnnotation {
  def macroTransform(annottees: Any*): Any = macro EnumMacros.decl
}

class EnumMacros(val c: whitebox.Context) {
  import c.universe._

  def decl(annottees: Tree*): Tree = {
    annottees match {
      case List(q"class $enumType { ..$body }") =>
        val values = body.collect {
          case q"""val $value = Value""" => value
        }
        val bodyParts = values.zipWithIndex.map {
          case (value, index) =>
            q"""case object $value extends $enumType($index, ${s"$value"})"""
        }
        q"""
          sealed abstract class $enumType(val index: Int, val name: String)
          object ${enumType.toTermName} {
            ..$bodyParts
            val values: List[$enumType] =
              List(..$values)
            val fromIndex: Int => $enumType =
              Map(..${values.map(value => q"$value.index -> $value")})
            val fromName: String => $enumType =
              Map(..${values.map(value => q"$value.name -> $value")})
          }
        """
    }
  }
}

Quasiquotes allow us to use the exact same syntax to both pattern-match against an existing AST, and produce a new AST afterwards. The upshot is that we can write macro code that’s easy on the eyes & doesn’t hide any treachery behind a veil of simplicity.

Better still is the fact that whatever AST we produce will be passed through the typer phase, which means that mistakes we make in generated code will be caught by the compiler, almost as if the code was written manually to begin with.

Without further ado I present the generated code:

sealed abstract class Status(val index: Int, val name: String);
object Status {
  case object Enabled extends Status(0, "Enabled") with scala.Product with scala.Serializable;
  case object Disabled extends Status(1, "Disabled") with scala.Product with scala.Serializable;
  val values: List[Status] = List(Enabled, Disabled);
  val fromIndex: _root_.scala.Function1[Int, Status] = Map(Enabled.index.->(Enabled), Disabled.index.->(Disabled));
  val fromName: _root_.scala.Function1[String, Status] = Map(Enabled.name.->(Enabled), Disabled.name.->(Disabled))
};

Well, it looks kind of ugly, but I’m glad I don’t have to type this in by hand for EVERY. SINGLE. ENUM.

Fill the hole, seal the leaks

Emboldened by this newfound power of creation^W code generation, we can now correct some problems with our first generated enum. While we had the presence of mind to mark the base enum type sealed to prevent rogue derivation from this type in other compilation units (aka files), nothing prevents the intrepid explorers among our users from doing all sorts of silly things, such as:

@enum class Status {
  val Enabled, Disabled = Value
}

// rogue joker
case object JokeIsOnYou extends Status(-1, "ha ha ha")

This will, unfortunately, compile. As you can imagine, JokeIsOnYou shouldn’t exist, yet it does, and makes our enum essentially useless, since neither Status.values() nor the lookup methods know anything about JokeIsOnYou. What can we do?

Since we’re simply writing Scala “with templates”, all we have to do is create a protected abstract type within the companion object we generate for each enum type, then require a value of this protected type to be passed in every time an enum value derives from the base generated enum type. This is best illustrated like so:

q"""
  sealed abstract class $enumType(
    val index: Int,
    val name: String
  )(implicit sealant: ${enumType.toTermName}.Sealant)

  object ${enumType.toTermName} {
    @scala.annotation.implicitNotFound(
      msg = "Enum types annotated with @enum can not be extended directly. " +
      "To add another value to the enum, please adjust your " +
      "`def ... = Value` declaration.")
    protected sealed abstract class Sealant

    protected implicit object Sealant extends Sealant

    ..$bodyParts

    val values: List[$enumType] = List(..$values)
    val fromIndex: Int => $enumType = Map(..${values.map(value => q"$value.index -> $value")})
    val fromName: String => $enumType = Map(..${values.map(value => q"$value.name -> $value")})
  }
"""

Here, Sealant is the type we hide within our generated companion. The only possible instance of Sealant exists inside the generated companion, which also happens to be the only place that subtypes of our enum - i.e. all of its values - can be instantiated. The result is what we expect; deriving from Status outside of the macro-generated companion is no longer possible:

[error] Status.scala:9: Enum types annotated with @enum can not be extended directly.
[error] To add another value to the enum, please adjust your `def ... = Value` declaration.
[error] case object JokeIsOnYou extends Status(-1, "ha ha ha")
[error]                                 ^
[error] one error found

This is kind of fun! What else can we do?

Switching gears

We’ve sealed up our enum as tightly as we could, ensuring that Scala’s own exhaustiveness check will emit a warning in case we forget to account for every enum value in a pattern match. However, as mentioned previously, warnings are way too easy to ignore. Can we create an optional syntax that will emit an error instead of a warning? Let’s shoot for this:

val color =
  Status switch {
    case Enabled => "green"
    case Disabled => "red"
  }

This looks pretty good, and it’s easy to implement with another macro. To start, let’s position the switch method to hang off our generated companion:

q"""
  object ${enumType.toTermName} {
    // ... everything as before plus this:
    def switch[A](pf: PartialFunction[Status, A]): Status => A =
      macro numerato.SwitchMacros.switch_impl[$enumType, A]
  }
"""

Here we’re taking a partial function capable of maybe converting Status to some A, and we turn it into a total Status => A function using a macro. This one’s a bit more involved than straight-up quasiquotin’, so feel free to study its implementation on your own time.

Let’s illustrate the switch syntax at work; suppose we had gone ahead and added another value - say, Unknown - to our Status enum. The compiler now emits an error when it sees that we’ve forgotten to include the newly added enum value in our pattern match:

@enum class Status {
  val Enabled, Disabled, Unknown = Value
}

import Status._

val color =
  Status switch {
    case Enabled => "green"
    case Disabled => "red"
  }
// Exiting paste mode, now interpreting.

// <console>:24: error: not all values of Status are covered: Unknown
//          Status switch {
//                        ^

Yes! It blows up at compile time. Take that, Java!

Parting thoughts

We’ve written a couple of macros that implement a saner, safer, easier to use enumeration for the Scala language. Whether or not you agree that this is a missing language feature is unimportant. The purpose of this exercise was to illustrate a valid use case for macros as a tool for creating new syntactical constructs.

As mentioned previously, material for this post was taken from a library called numerato, which makes it easy to take advantage of the @enum annotation and switch syntax.

Stay frosty & happy quasiquotin’!