How the Interns at Knoldus Caused a Flutter at Scala Days 2015 !

Knoldus is a select Typesafe partner and has been engaged in delivering reactive, scalable products for the last 4.5 years. The company boasts of well rounded full stack developers who engage with the clients and take their reactive product dream to reality. A lot of case studies can be accessed here. An integral part of the Knoldus Internship Program (KIP) is to introduce great minds on Java to the beautiful Scala language. There are 2 months worth of classroom sessions held for engineers who want to leap to Scala after having worked for a few years on Java. These sessions include Scala, Akka, Play framework, Spark, Spray, Design Patterns, Good coding practices, TDD with Scala, Selenium DSL, Angular js, Backbone and much more. The batch of KIP-2015 was no different. We had a set of very capable interns join us and we started brain washing them on Scala.

Interestingly, Knoldus sponsored the Scala Days 2015 both at San Francisco and Amsterdam. 2 weeks before the conference, a group of interns, while enjoying coffee came up with the idea of doing sentiment analysis on the tweets which originate on the during the sessions. The idea was simple, we would tag the tweets to a session if they happen to contain the twitter handle of the speaker. Next the tweet collection for the session would start 2 hours prior to the session and stay active for 8 hours after the session (For people like me who would tweet towards the end of the day something like -> “Had a great day today, learned a lot from @HANDLE on xyz”). We did not assume that we would start getting a lot of traffic but the solution was fun. See the product here.

The architecture of the solution was as follows

As you would notice, all the relevant tweets were collected from GNIP and put into a Kafka queue. A cluster of 2 spark nodes would pick up the tweets from the queue, perform sentiment analysis and store it into MongoDB. The play front end would pick up the sentiment analyzed tweets from Mongo and present them on the browser. All in real time. The team used Bootstrap because we imagined that most of the people at the conference who would view the results would either do it on their phones or their tablets. Tweets were broadly divided into 3 categories. Positive, Negative and Neutral (or no sentiment detected). The frames used for sentiment analysis were a very small (almost infinitely small) subset of the frames that we use for emotion analysis on consumer brands which splits the tweets, blogs, boards, fb posts, tumblr, pinterest pictures into 18 different emotions. The Scala Days 2015 began and so did the tweets.

The flutter

If you have been to the Scala Days you would notice that Typesafe is very very particular about the feedback that each session receives. This formal feedback process is done by volunteers holding an iPhone at the exit gates where all the attendees punch their feedback about the session. I am told that if a speaker gets a red button pressed then the speaker gets a good reason not be considered for the next year. Harsh, I believe but probably it maintains the quality of the conference.

Now, our sentiment detection product started getting a bit popular and we started spotting people checking it on their phones. So far so good.

As soon as the session was done, after a cooling period of around 30 minutes, a tweet went out which described the sentiments for that session. People seemed to be loving it. It was the most immediate feedback on a session and attendees and speakers were elated. Example

Hell broke loose (Ok, it wasn’t that bad, it is just to get your attention) when some of the speakers started getting disturbed by the negative sentiments being detected. The tweet showed the sentiments in red, green, yellow. The official feedback picked up clicks from attendees in red, yellow, green. This is where the problem started. Speakers started flocking back to the conference organizers assuming that the official feedback was being tweeted. Knoldus representatives at the conference were summoned and we needed to clarify that the sentiment analysis was un-official and had no inputs from the official feedback scores. The similarity of colors was another problem.

A couple of solutions were immediately sought.
1. Let us change the color scheme so that it looks different from the official feedback.
2. Let us carry a disclaimer on the top, “Note: This experimental service is not related to in any way to the official Scala Days session feedback. This is designed to detect Twitter sentiment and graphically represent the results in real time.”

We rolled up the changes in a few hours and the site went live again with the changes. Overall inspite of the bump, it was an amazing experience. It showcased how in less than 10 working days a cool system could be built quickly on top of the amazing Typesafe stack. The drill down of any session looked like this.

The team was elated to get the response that they got. Thank you, all attendees at Scala Days. In the next post, we would try to post an analysis of the sentiment analysis. Till then happy functional programming!