Easiest Way to Recover Akka Actor for Remote Application JVM Failure Scenario in Scala

In this blog, I would explain how easily you can recover Akka actor for Remote Application JVM failure scenario in Scala.

I have implemented this in one of my project and it is working fine.

Suppose there is a remote application, which is being used by multi JVMs, and JVMs are passing message to each other through Akka actor, then
there must be an Akka actor recovery mechanism to prevent any application crash.

For ex:
There are three JVMs, accessing a remote application. All of them are using same Akka actor reference to pass message to each other.
That Actor could be created by any JVM. Other JVM would access that actor by its actor path.

If JVM1, who is the creator of actor, shuts down for any reason then application would be crashed because other JVMs are using same actor unless we don’t have any Akka actor recovery mechanism.

I am giving here just a scenario. There are many other scenarios as well for crashing a remote application.

Now let’s come back to scenario which I have explained above:

We have one controller, which creates an actor on the request of any JVM, if actor does not exist, or gives reference of existing actor on the request of any JVM.

def getOrCreateActor(): ActorRef={
// Write here logic for 
// creating an actor, if actor does not exist
// Or
// giving reference of actor, if actor exists. 

}

/**
* Retry support for Akka Actor
*/
def retry(func: => ActorRef, message: Any, maxAttempts: Int, attempt: Int): Option[Any] = {
    try {
      val future = akka.pattern.ask(func, message)
       // Use Await.result, if blocking is necessary
       // Use mapTo for non-blocking scenario
      val result = Await.result(future, timeout.duration) 
      Some(result)
    } catch {
      case e: TimeoutException =>
        if (attempt <= maxAttempts) retry(func, message, maxAttempts, attempt + 1)
        else None
    }
  }

When Akka future fails to respond, this retry logic would start automatically and would work until result is obtained or until number of retry limit is finished.
If we would not provide any limit, this would work indefinitely until result is obtained.

A sample call to this method :-

retry(getOrCreateActor, message, 10, 0)

As you can see, in the above “ask” method, we are passing a function, to create Akka actor, as first parameter instead of passing particular actor.

The reason is:

If an actor dies for any reason, this logic would still work. It would create a new actor to prevent the application being crashed and try to receive result from newly created actor.

Note:- We should always avoid Await.result unless we have any blocking scenario. We could use mapTo for non-blocking scenario.

In my case I am using Await.result. Because there are some chances for failed JVM to get repaired after some time, so that we might not need to create new actor.

If you have tried any better approach in terms of coding and performance, please suggest here.

I would try to implement the same.

Your suggestion would be highly appreciated.