Alpakka: Process big CSV files using Akka Streams

Table of contents
Reading Time: 3 minutes

In this big data world, we have lots of data, and generally, that data comes in CSV files. So as the data is huge, so the CSV files will also be huge. It will not be feasible to load that CSV file into memory in one go. So here the challenge comes, that how should we read those big CSVs.

I got the same use case in one of my projects. I got an assignment where I needed to process the CSV file and do some transformation.

The first thing came in my mind that I should load the CSV file in memory and then should do processing, but as the file was big, so this option was not the feasible one, so I dropped that idea. Then I thought of reading the file line by line. By doing this, the whole file will not be loaded in memory as only one line at a time will be loaded, so this option was looking perfect.

So I decided to go with this approach only. Now I needed to decide what technology should I used to read the CSV file line by line.

First I thought of using which reads the file and returns the iterator. Using iterator, we can process the file line by line as follows:

object Application {
  def main(args: Array[String]): Unit = {
    val source = Source.fromFile("/home/rishi/files/emp.csv")
    val lines = source.getLines()
    while (lines.hasNext) {

The above code looks pretty simple and will process the CSV file line by line easily. But it was not fulfilling my use case completely. I wanted to access each line of CSV with its header but in the above code, it is not happening, and to fulfill my use case, I needed to create a Map explicitly by taking CSV’s first line as a header. Doing all these things by myself will take some effort, therefore I was looking for some library which will do it for me.

So I moved to Alpakka which uses Akka stream under the hood and provides simple APIs to process CSV. Akka stream does the processing in a non-blocking way and provides the features of backpressure. Non-blocking processing was the extra advantage that I was getting with Akka stream. ( processes the file in a synchronous way.)

Let’s take a look of the code below:

First, we need to add the below dependencies in the pom.xml file:



then, create a CsvUtility file which reads the CSV file using Akka streams:


class CsvUtility(implicit val system: ActorSystem) {

  def processCsvFile(inputFile: File): Future[Boolean] = {
    val inputFileName = inputFile.getName
    val csvFile: Path = Paths.get(inputFile.getPath)
    val source = FileIO.fromPath(csvFile)

      .filter(row => row.getOrElse("dept", "") == "bowler")
      .runForeach(x => println(x))
      .map { _ =>
        println(s"Successfully processed the file $inputFileName")
      .recover {
        case _: Exception => println(s"Error in processing the file $inputFileName")

As the CSV file is having tilde (~) as the separator, so we have provided that separator explicitly as


To convert each line into a Map where the key will be the CSV header and the value will be the row value, we have used the following code :


Akka stream returns the future as it processes the files in a non-blocking way.

Now, create a main class that will start the process.


object Application {

  def main(args: Array[String]): Unit = {
    implicit val system: ActorSystem = ActorSystem()

    val file = new File("/home/rishi/files/cricketer.csv")
    new CsvUtility().processCsvFile(file)
      .map { x =>
        if (x) println("File successfully processed") else println("File processing failed")
      .recover {
        case _: Exception => println("Error in file processing")

Let’s look at the CSV file.



That’s it. It’s quite simple. There are many other ways to use Alpakka for CSV processing which you can find out here.

I hope, this blog will be useful for everyone.



Written by 

Rishi is a tech enthusiast with having around 10 years of experience who loves to solve complex problems with pure quality. He is a functional programmer and loves to learn new trending technologies. His leadership skill is well prooven and has delivered multiple distributed applications with high scalability and availability by keeping the Reactive principles in mind. He is well versed with Scala, Akka, Akka HTTP, Akka Streams, Java8, Reactive principles, Microservice architecture, Async programming, functional programming, distributed systems, AWS, docker.