Antoine Kalmbach

In Scala, Akka actors, as in the traditional Actor model, may modify private state. The accepted convention is to have a mutable object (e.g. a Map), a var, and mutate it like so:

class Library extends Actor {
  var books = scala.collection.mutable.Map.empty[String, String]
  
  def receive: Receive = {
    case AddBook(isbn, title) => books += (isbn -> title)
  }
}

object Library {
  case class AddBook(isbn: String, title: String)
}

This is a bad idea. There are several reasons for this. First, Scala eschews vars, they should only be used when absolutely necessary (read: never). There is also an additional need for thread-safety for the collection, not because of the receive method itself. The receive method is guaranteed to run inside a single thread. However, an unsuspecting user might still launch a Future and modify the collection, leading to unpredictable behaviour. Such concurrent mutations on a var put strain on the garbage collector, in fact, it often necessitates the existence of a garbage collector.1 Lastly, as with any mutable state and possible lack of referential transparency, the code can become hard to reason about.

Thankfully, Akka actors offer a possibility to do this completely functionally. The function context.become allows an Actor to change its receive method on-the-fly. In other words, it lets the Actor change its state and communication model. Here’s the above implemented using this paradigm:

class Library extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => {
      // for immutable maps, += returns a new collection
      context.become(active(books += (isbn -> title))) 
    }
  }
}

The active function returns a new Receive, receiving the current actor state as its parameter. Adding logic to it is now easy:

class Library extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn title) => {
      if (books.size < 10) {
        context.become(active(books += (isbn -> title)))
      } else {
        sender() ! "Too many books"
      }
    }
  }
}

The above code is now thread-safe and doesn’t use mutable collections, but what if our logic gets more complicated? What if we need to talk to another Actor, or talk to the sender of the message? This is where we stumble upon a design feature of Akka: all of its Actors are actually compiled down into a callback-based implementation. There is no guarantee that a Future launched in a receive case will be running in the same thread as the next! One could argue that this is not a feature but a flaw, but I won’t go that far. Hence, code dealing with Futures in Akka actors needs to deal with the unforgiving reality that there is no guarantee of thread safety. Case in point:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case AskForBook(isbn) => {
      popReservation(isbn) foreach { i =>
        // AAH!!!
        context.become(active(books - i))
        sender() ! s"Here you go: $i"
      }
    }
  }
}

Why am I screaming in the comments? First, as calling map for our Future launches a new thread, we have no idea whether sender() returns the same value in the new thread, and second, we may be modifying the books collection concurrently with other threads - leaving the garbage collector to collect our mess. So we strain the GC and risk giving the book to the wrong caller!

Since the actual execution of a Future is left to the execution context, which in the case of Actors is the ActorSystems dispatcher, we may or may not be invoking sender() in the right thread — there is simply no guarantee. We can’t reason about it, it has been hidden from us.

To deal with this, Akka has introduced the pipe pattern, which is an implicit given to Futures which solves this:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case AskForBook(isbn) => {
      // launch another thread
      val reservation: Future[String] = popReservation(isbn) map { i =>
        s"Here you go: $i"
        context.become(active(books - isbn)) // AAH!
      }
      // but sender() is still the same
      reservation pipeTo sender
    }
  }
}

Another option is to fix the reference of sender:

val s = sender()
val reservation: Future[String] = popReservation(isbn) map { i =>
  s ! s"Here you go: $i"
  context.become(active(books - isbn)) // AAH!
}

Ok, now we’ve fixed sender(), but what about the books collection? Let’s add a PopBook(isbn: String) case class, and handle that for removals:

class Library(popReservation: String => Future[String]) extends Actor {
  def receive = active(Map.empty)
  
  def active(books: Map[String, String]): Receive = {
    case AddBook(isbn, title) => { ... } // as before
    case PopBook(isbn) => context.become(active(books - isbn))
    case AskForBook(isbn) => {
      // launch another thread
      val reservation: Future[String] = popReservation(isbn) map { i =>
        s"Here you go: $i"
        self ! PopBook(i)
      }
      // but sender() is still the same
      reservation pipeTo sender
    }
  }
}

Sending messages to self is always thread-safe - the reference does not change over time. So, at this point, it seems clear that making actor code thread-sane involves the use of:

  • immutable state - call context.become with a closure over the new actor state,
  • converting asynchronous state modifications as messages to be handled later, and
  • making sure the sender() reference is consistent

What about complicated states? What if we need to react differently to these messages, e.g., when the library is closed? I sense that you’re about to mention Akka’s FSM construct, which builds a state machine, encapsulating state and transitions to what is essentially syntactic sugar, and on the surface, seems like a good idea.

Enter Akka FSMs

At a closer look, it essentially leads us to repeat the same mistakes as above, and the arguments against it are argumented here. In summary, it boils down to:

  1. Akka FSM’s is too restrictive. You cannot handle multi-step or complicated state transitions, and modeling undeterministic behaviour is impossible.
  2. You are tied to Akka completely, you must use Akka testkit for your tests. Anyone who has worked with testkit knows this to be a burden.
  3. State transitions have identity instead of being truly functional, that is, FSMs alter the current state instead of producing a new one.

Moreover, and I think this is the biggest shortcoming, the Akka FSM are finite-state automata — they are characterised by the state transition function (Input, State) => State. Since we know actors are more about communication than anything else, this model is insufficient, and what we need is a state machine that can produce output: a finite state transducer. Its state transition function has the signature (Input, State) => (Output, State) - every transition produces an output, and Scala can model this efficiently:

trait FSA[State, Input, Output] {
  def transition(s: State, i: Input): (Option[Output], State)
}

With all these flaws, despite being a nice idea at a glance, it’s obvious that for any complicated logic Akka FSM’s aren’t sufficient.

Let’s envision a radical version of actors, accounting for all the flaws described above:

  • State transitions should be about producing a new state, i.e. (Input, State) => (Output, State)
  • Actor computations will deal with asynchronous code, we must deal with this intelligently
  • Keep I/O logic out of actors - the actor only communicates with the external world
  • Actors should only mutate their state with with context.become

The last bullet point is especially important, as it constrains state changes to be entirely functional, as you can simply make a function def foo(state: State): Receive, and keep calling it recursively, by transitioning states thusly:

def active(state: State): Receive = {
  case someInput: Input => context become active(state)
}

This idea is not new. Erlang actors have worked like this for actual decades, and arguments for using this method in Scala can be found left and right, summarized particularly well in Alexandru Nedelcu’s Scala best practices.

active(Sum) ->
  receive 
    {From, GetValue} -> From ! Sum;
    {n} -> active(Sum + n)
  end.

Putting emphasis on the last point, I’ve come up with a moniker called communicators.

Actor, meet communicator

Let’s define the Communicator trait first independently:

trait Communicator[State, Input, Output] extends Actor {
  /** This is the initial actor state */
  def initial: State

  /** The state transition function */
  def process(state: State, input: Input): Future[(Option[Output], State)]

  /** The output processing function */
  def handle(state: State, output: Output, origin: ActorRef): Future[Unit]
}

initial is simply the initial state machine state, process is the state transition function and handle is the function that will deal with dispatching the result of process. Because we’re producing content in another thread, we want to make sure the reference of sender is fixed, and by using this with the pipeTo pattern, we get thread safety. Let’s extend the Actor trait to get receive

trait Communicator[State, Input, Output] extends Actor {
  /** This is the initial actor state */
  def initial: State

  /** The state transition function */
  def handle(state: State, product: Output, origin: ActorRef): Future[Unit]

  /** The output processing function */
  def process(state: State, input: Input): Future[(Option[Output], State)]
  
  def receive = active(initial)
  
  /** I/O handling which the deriving class must implement */
  def active(newState: State): Receive
}

The active function is the actual output-producing function. The user is left to define three things:

  • the initial actor state in initial
  • the output dispatch function handle
  • the state transition function process
  • the active function which handles input and output

To see this in action, first, let’s define the application states.

object Library {
  // Library state
  case class LibraryState(open: Boolean, books: Map[String, String])

  // Input alphabet
  sealed trait LibraryInput
  case class SetOpen(o: Boolean)                  extends Input
  case class AddBook(isbn: String, title: String) extends Input
  case class GetBook(isbn: String)                extends Input

  // Output alphabet
  sealed trait LibraryOutput
  case object SorryWeAreClosed                        extends Output
  case object DoNotHaveIt                             extends Output
  case object SorryReserved                           extends Output
  case class Book(isbn: String, title: String)        extends Output
  case class Reservation(isbn: String, title: String) extends Output
}

The actual state is just a case class: this gives us the nice copy function for easy updates. Then we use polymorphism to implement the input and output alphabets. Then we implement the actor itself:

class Library(getReservation: String => Future[Boolean])
    extends Communicator[LibraryState, LibraryInput, LibraryOutput] {

  import Library._

  def initial = State(false, scala.collection.immutable.Map.empty)

  override def active(newState: LibraryState): Receive = {
    case (output: LibraryOutput, origin: ActorRef) => handle(output, origin)

    case input: LibraryInput => {
      val origin = sender()
      process(newState, input) map {
        case (output, state) =>
          output foreach { o =>
            self ! (o, origin)
          }
          self ! state
      }
    }
  }

  override def process(state: State, input: Input): Future[(Option[Output], State)] =
    input match {
      case SetOpen(o) => Future.successful((None, state.copy(open = o)))

      case (GetBook(_) | AddBook(_, _)) if !state.open =>
        Future.successful((Some(SorryWeAreClosed), state))

      case GetBook(isbn) => {
        val book =
          for {
            (isbn, title) <- state.books.get(isbn)
          } yield {
            getReservation(isbn) map { reserved =>
              if (!reserved) {
                (Some(Book(isbn, title)), state.copy(books = state.books - isbn))
              } else {
                (Some(SorryReserved), state)
              }
            }
          }

        book getOrElse Future.successful((Some(DoNotHaveIt), state))
      }

      case AddBook(isbn, title) =>
        Future.successful((None, state.copy(books = state.books + (isbn -> title))))
    }

  override def handle(state: State, output: Output, origin: ActorRef): Future[Unit] = {
    Future {
      origin ! output
    }
  }
}

Decoupling Akka

So, now we’ve made a very thin actor, with little I/O logic inside it, but it’s still an actor. Let’s decouple it entirely from actor semantics. First, we define a StateMachine[I, O] trait:

trait StateMachine[I, O] {
  def process(input: I): Future[(Option[O], StateMachine[I, O])]
}

And excise the state logic from the Communicator, moving it to the State case class:

case class LibraryState(open: Boolean, books: Map[String, String], getReservation: String => Future[Boolean])(
    implicit ec: ExecutionContext)
    extends StateMachine[LibraryInput, LibraryOutput] {
    
  def process(input: LibraryInput): Future[(Option[LibraryOutput], LibraryState)] = {
    input match {
      case SetOpen(o) => Future.successful((None, copy(open = o)))

      case (GetBook(_) | AddBook(_, _)) if !open =>
        Future.successful((Some(SorryWeAreClosed), copy()))

      case GetBook(isbn) => {
        val book =
          for {
            title <- books.get(isbn)
          } yield {
            getReservation(isbn) map { reserved =>
              if (!reserved) {
                (Some(Book(isbn, title)), copy(books = books - isbn))
              } else {
                (Some(SorryReserved), copy())
              }
            }
          }

        book getOrElse Future.successful((Some(DoNotHaveIt), copy()))
      }

      case AddBook(isbn, title) =>
        Future.successful((None, copy(books = books + (isbn -> title))))
    }
  }
}

You may be wondering: wait, where’s the handle implementation? We kept that out from the state machine class since it’s not its responsibility - so we keep that in the Communicator:

class Library(getReservation: String => Future[Boolean])
    extends Communicator[LibraryInput, LibraryOutput, LibraryState] {
  import context.dispatcher

  def initial = LibraryState(false, scala.collection.immutable.Map.empty, getReservation)

  override def handle(output: LibraryOutput, origin: ActorRef): Unit = origin ! output

  override def active(newState: LibraryState): Receive = {
    case (output: LibraryOutput, origin: ActorRef) => handle(output, origin)

    case state: LibraryState => context become active(state)

    case input: LibraryInput => {
      val origin = sender()
      newState.process(input) map {
        case (output, state) => {
          output foreach { o => 
            self ! (o, origin)
          }
          self ! state
        }
      }
    }
  }
}

So, all state is kept neatly in a separate entity that’s entirely unit testable in its own right without having to rely on Akka testkit or the like – input and output dispatch and state transitions are done in the active method.

I know the state case class manipulation introduces more boilerplate, but as long as that boilerplate isn’t complicated, I think this is a fair compromise. Plus, one can use lenses to remove some of the boilerplate, e.g., by defining handy update functions. One could cook up something doggedly interesting using Cats and StateT - as long as you provide a function of the kind (I, S) => (Option[O], S), the sky is the limit.

Thanks to Jaakko Pallari (@jkpl) for previewing this.

  1. This is actually false, as Aaron Turon, a core Rust developer, proves in his article about getting lock-free structures without garbage collection


Focus is a design element in programming languages that I think deserves more attention than it gets.

A focused language puts emphasis on a set of coherent idioms. Multi-paradigm languages like C++ or C# are unfocused because they lack a certain principle.

Take C, for instance. You can do OOP in C, but it’s awkward. You need structures full of function pointers and the language wasn’t designed for it: it’s not a good idea to do it. The point is that you can but you shouldn’t.

Focus is not so much of what a language has but something that it embodies. A single-paradigm language can still be unfocused, because there can be several ways to wield the singular paradigm. A multi-paradigm language can be focused if the multi-paradigm languages connect at a higher level. Focus can be implemented as a coding standard or it can be something that everybody understands as the idiomatic way of doing things.

Focus is not always a positive trait but it rarely is a negative trait. On the other hand, a lack of focus is more often negative than positive.

Take Haskell, a pure functional language, effectively single-paradigm; it is a very special case. The language itself is absolutely focused to the point of extreme autism, but its flexible type system and vibrant community, there are many ways to program Haskell. Do you absolutely need state? Use state, but be careful. Do you want IO without monads? Well, sure, but be careful.

At a high level, Haskell code is pure. It permits some inconsistencies with its principal paradigm but it eschews them and this is the key difference.

A bigger problem with focus is that it often is intangible. It’s easier to point out languages that are unfocused than those that are. Focus is about philosophy. Some language are very philosophical. For instance, Clojure is just as much about its particular approach to concurrency, state and identity, that is a language implementing those ideas. The language caught on because Rich Hickey, the author, did not market it as the tool that would have solved everybody’s problems, but because he marketed the ideas that Clojure represented as a solution to common programming problems.

“If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.”

Antoine de Saint-Exupéry

In this context, Clojure can be seen as a focused language. These core philosophies are what constitute the language, the fact that Clojure happens to be a Lisp dialect implementing the philosophies is secondary in my mind. With that in mind, I acknowledge being a Lisp is also a core part of Clojure, but its principles about state and identity can be implemented in any language. Clojure does let you do OOP but it feels awkward. When you grok Clojure you understand what that means: the language can be bent for that purpose, but it doesn’t want to be. Its philosophy is like a memory-foam, if you tamper with it, it will coalesce back into its original form. When you see that, it’s the moment you understand what Clojure—or any other language—is about.

Some languages double down on philosophy by making it a part of a coding standard and enforcing it: Go. Go embodies simplicity and intuition, intentionally eschewing things that are not modern, but complex, opting to keep the core language simple. Some chalk this down as a negative trait, others love it; I find it to be both good and bad. Good, because I can jump into any Go codebase and get its purpose in minutes; bad, because sometimes I want to use abstractions for which Go is unsuitable. I respect its design philosophy, because it has one, and absolutely flaunts it. It’s not just a structural type system, it’s an idea.

Scala is another beast. It began as an experiment, trying to augment and fix the deficiencies of Java. It was designed by brilliant PLT theorists and the language is a beautiful and magnanimous behemoth. Scala has so many features that it eschews focus either intentionally or unintentionally. On the other hand, Scala is capable of many great things. But if you ask two Scala programmers what Scala represents to them, you may get different answers.

It can be a technical aspect. To some, it might be about Shapeless or all the cool things that go with it. Macros. DSLs. Code generation. Or it could be how Akka or Spark are amazing tools. It could also be a philosophical difference. Some people want an advanced type system and don’t want to be constrained by the laziness and purity of Haskell. Others want the JVM. Some just want a better Java. Some just happen to use it for Spark.

I would choose the simpler Scala, the better Java. Trait-based generics, sum types, implicits, and functional programming, to name a few. This is not just because it’s less complicated, from a business perspective, it makes it easier to hire new programmers.

As a professional Scala developer and a long-time functional programming enthusiast, I fear that I may never comfortably jump to another company, confident that since I’ve written Scala, I can understand their Scala. That, or years of experience, but who knows what’s enough? Their, whoever they may be, Scala might not be the simple Scala I and my colleagues prefer.

This is scary. For the future of the language, this is an untenable position. While I absolutely enjoy working with the language, I’m afraid that it is fated to be like Macbeth from Shakespeare: “thou shalt get kings, though thou be none”. Thus, Scala will inspire a great language and then die. Maybe it already did, and the clock is ticking. Some purport Kotlin as the successor, but I wouldn’t bet on it just yet.

“Ah, but a man’s reach should exceed his grasp, or what’s a heaven for?”

Robert Browning

The thing about Scala is that this is a conscious design decision. The language is meant to have everything and the kitchen sink. Programming languages don’t have to be simple. Powerful languages are powerful tools. Use them well, you can achieve greatness. You have to choose your tool set and hone it.

But for Haskell, Go, and Clojure, you’re using them, and you’re thinking, what is the natural way to do this? Once you find it, you find yourself implementing ideas using that philosophy, that natural way, and you’re no longer just using a tool. You’re using an idea.


What is the point of abstractions?

We want to hide things. We want to generalize things. We want to extend things.

Why are mathematical abstractions so intractable? Why is the Wikipedia page on functors incomprehensible to someone not used to mathematical formalisms? Why does it sound so vague?

When approaching abstractions, for educational purposes, it is sometimes easier to think of analogies or similes. We can conceptualize the idea of functors of “procedures” that operate on things inside “boxes”, or we can study relational algebra using Venn diagrams.

These analogies are dangerous, because they are vague. Formalisms leave no room for interpretation because they are exact not whimsically, but because of the pervasive imprecision of the human mind.

Let’s take an analogy that’s very approachable but quite dangerous. Explaining modular arithmetic can done with clocks. The analogy would go like this:

You see, when you have a number modulo 12, think of it as a clock. If x is over 12, think of like like the hand of a clock looping over, and you’re back where you started.

The problem with such an analogy is that not everybody uses 12-hour clocks. Most of Europe uses a 24-hour clock with no distinction between AM and PM. Of course, they are also taught to understand that “sixteen” means “four” since nobody builds 24-hour analog clocks (yet). That aside, it’s still very possible, that when explaining the above analogy to someone accustomed to 24-hour clocks, they’ll get confused since what comes after 12, is 13, not 0.

This is a basic but fundamental example: things like functors, semigroups, monads, and categories, are a bit intractable for a reason: there’s no room left for interpretation.

Mathematical formalisms pare fundamental ideas into pure forms. Your intuition can’t get in the way and corrupt them.

The obvious downside is that these formalisms are harder to understand. I wager that this is for the better because down the road there are concepts so high-level one can’t even begin to think in analogies, and it will only slow one down.

There was a turning point in my math studies when I stopped trying to grok things using analogies. My approach to topology was geometrical. I tried to visualize limit points in my mind and in vain, because the mind can’t bend itself around more than three spatial dimensions. Granted, visualizing hypercubes was possible (“like a cube is made of sides, a hypercube is made of cubes”)… kind of.

Stopping this perilous habit, I started to memorize laws instead. That changed the language of maths for me, forever. I wasn’t understanding relations via shapes or arrows, but by basic axioms and mathematical laws. It wasn’t too long before I started to visualize concepts using these laws.

I stopped staring concepts in the eye, looking for hidden meanings behind bounded sets. I simply read the definition, thought “huh”, and memorized it. Slowly, by building towards other related concepts and set theory I quickly understood what the law meant, without trying to grok the hidden meaning.

Once that became a habit, it became easy and changed my approach forever: let go of intuition. Abstract ideas are hard by definition and they need to be understood piece by piece, from the ground up.

This is why any explanation of a precise thought, like a mathematical formalism, using something imprecise like an analogy, is a fallacy doomed to fail.

When functional programmers try to explain basic ideas like semigroups or functors, they often find themselves in an apologetic mire of simplifications. This is doomed to fail. Give concrete examples of how a functor works. Don’t illustrate them as operations that can map stuff in boxes to other boxes. Invent a problem and solve it using a functor. After all, they’re such a basic concept, even those writing non-functional code end up doing them all the time.

Let the abstraction sink in, it’s the only thing that will survive.