Do you want to catch up with your opponents by looking at their product listings or generate your own content from someone else’s website or want to run some analytics or just want to scrap a website for the sake of it? Well, it’s not a rare scenario of website scraping. Though, the question is, how to scrap a website! When we Google for – “How to crawl a website?”, we get a lot of libraries for various programming languages. These libraries are pretty useful but rigid in use. Sure these libraries are fun to use and do the work for you. But don’t you sometimes feel that you can have more control over things. A scraping library might force you to download all web pages to the database and then process them. It increases the workload and those damn web pages sit in your expensive hard disk.
Lately, I have a been thinking to find faster and less resource consuming solution to scrap and process data to get information from a website. And I found one! My target to scrap was a job listing website. I used Akka with JSoup and processed web pages that sum up around 0.5 GB in size, in half an hour at my home (with top internet speed ~400KBps). I wanted to scrap all the jobs listed on that job site. To start scraping I needed the starting point, so I gathered my seed links. The homepage could be the topmost link and work as the seed link, but as I knew what I was going to scrap, unlike web crawlers, I scraped the topmost pages of the website that divide the jobs into different categories. The example of such page(s) could be category page on an e-commerce website like Amazon’s Shop by Category. All the category links of the Amazon can be selected with CSS selector ul.nav_cat li a.nav_a For my target job site, I used the JSoup to scrap the root-category-webpage, without any actors though, to give me the seed links I needed. Once I had my seed links, I saved them in a .csv file along with the next CSS selector page on that seed link page to select the further HTML Elements. For example, if I select the Kid’s Watches from Amazon’s category page, then links of the watches from the listing can be selected with div.a-row div.a-column a.a-link-normal.a-text-normal CSS selector. And once I get these links I’ve access to the actual watch details. The details of the products are themselves stored in web pages in a structured way. Voila! Finally, we have the information we needed.
Now comes the code. Haven’t we been waiting for that? (Aah! I knew that :D)
The seed links in the CSV file look somewhat like this:
https://www.example.com/x-y--z--a,div.multiColumn.colCount_four a
https://www.example.com/x-y--z--b,div.multiColumn.colCount_four a
https://www.example.com/x-y--z--c,div.multiColumn.colCount_four a
https://www.example.com/x-y--z--d,div.multiColumn.colCount_four a
I have replaced the name of the job website with example, and other obvious words with letters such as x, y, and z. I am just being cautious so that one doesn’t put the load on the website with unnecessary scraping. My whole list can be seen on Pastebin.
Below are the first two actors which first extract the seed links from the CSV file and then further hyperlinks available on the seed pages. SeedLinkExtractorActor gets the SeedPageCSVDirectory as the first-most message. Upon this, actor gets the list of seed links and CSS selectors from the CSV file as another message in its queue; it does so by telling itself with self! readSeedPagesFromCSVDiectory(csvFile). Once actor gets the list of seed links, it starts getting the links out of the seed pages with help of the child actors. The child actors keep running on different threads and keep pushing the extracted links to parent actor. That’s SeedLinkExtractorActor delegates the job to extract the links from the seed pages to child actors (made with SeedLinkExtractorActorChild.props). Once a child actor sends the extracted links to parent actor it has completed its work and leaves its thread to be used by some other actor.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.jobllers.service.jobsscrapper.impl.scrapper.actors | |
| import akka.actor.{Actor, PoisonPill, Props} | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.actors.JobLinkExtractorActor.JobPageCrawlInfo | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.actors.SeedLinkExtractorActor.{FileName, SeedPage, SeedPageCSVDirectory, SeedPageOptional} | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.actors.SeedLinkExtractorActorChild.{GetSeedLinksFromPage, SendSeedLinks} | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.naukri.Headers | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.util.RemoteContentPuller | |
| import org.joda.time.DateTime | |
| import org.jsoup.Jsoup | |
| class SeedLinkExtractorActor extends Actor with RemoteContentPuller { | |
| var numberOfSeedLinks = 0 | |
| var startingMillisec: Long = 0L | |
| var jobLinkExtractorActorCounter = 0 | |
| override def receive: Receive = { | |
| case SeedPageCSVDirectory(csvFile: FileName) ⇒ self ! readSeedPagesFromCSVDiectory(csvFile) | |
| case seedPages: List[SeedPageOptional] ⇒ | |
| var counter = 0 | |
| startingMillisec = DateTime.now().getMillis | |
| seedPages.foreach { seedPageOpt ⇒ | |
| counter += 1 | |
| context.actorOf(Props(classOf[SeedLinkExtractorActorChild]), s"childOfSeedLinkExtractorActor$counter") | |
| .tell(GetSeedLinksFromPage(seedPageOpt.optSeedPage.get), self) | |
| } | |
| case sentSeedLinks: SendSeedLinks ⇒ | |
| val seedLinks = sentSeedLinks.linkList | |
| jobLinkExtractorActorCounter += 1 | |
| context.actorOf(JobLinkExtractorActor.props, s"jobLinkExtractorActor$jobLinkExtractorActorCounter") | |
| .tell(JobPageCrawlInfo(seedLinks, "div.srp_container.fl div.row a.content"), self) | |
| numberOfSeedLinks += seedLinks.size | |
| println(s"$numberOfSeedLinks links extracted in ${DateTime.now().getMillis – startingMillisec} milliseconds") | |
| } | |
| private def readSeedPagesFromCSVDiectory(fileName: FileName): List[SeedPageOptional] = { | |
| import scala.io.Source | |
| val csvDirectorySource = Source.fromFile(fileName.name) | |
| val linesFromDirectory = csvDirectorySource.getLines().toList | |
| csvDirectorySource.close() | |
| linesFromDirectory.map { line => | |
| val linkSelector = line.split(",") | |
| if (linkSelector.size == 2) { | |
| SeedPageOptional(Some(SeedPage(linkSelector.head, linkSelector(1)))) | |
| } else { | |
| SeedPageOptional(None) | |
| } | |
| } | |
| } | |
| } | |
| object SeedLinkExtractorActor { | |
| val props = Props(classOf[SeedLinkExtractorActor]) | |
| case class FileName(name: String) | |
| case class SeedPage(uri: String, selector: String) | |
| case class SeedPageOptional(optSeedPage: Option[SeedPage]) | |
| case class SeedPageList(seedPages: List[Option[SeedPage]]) | |
| case class SeedPageCSVDirectory(file: FileName) | |
| } | |
| class SeedLinkExtractorActorChild extends Actor with RemoteContentPuller { | |
| import scala.collection.JavaConversions._ | |
| var numberOfSeedLinks = 0 | |
| override def receive: Receive = { | |
| case GetSeedLinksFromPage(seedPage: SeedPage) ⇒ | |
| val seedLinks = Jsoup.connect(seedPage.uri).headers(Headers.naukariHeaders).timeout(60000).get() | |
| .select(seedPage.selector) | |
| .foldLeft(List.empty[String])((listOfLinks, link) ⇒ addVerifiedLinkToList(listOfLinks, link)) | |
| sender() ! SendSeedLinks(seedLinks) | |
| self ! PoisonPill //Use –> context stop self <– instead | |
| } | |
| } | |
| object SeedLinkExtractorActorChild { | |
| case class SendSeedLinks(linkList: List[String]) | |
| case class GetSeedLinksFromPage(seedPage: SeedPage) | |
| } |
There are three more actors which extract the actual job description information – first, extracting the links from the job listings and then, from those individual pages the information can be extracted.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.jobllers.service.jobsscrapper.impl.scrapper.actors | |
| import java.io.IOException | |
| import java.util.concurrent.TimeoutException | |
| import akka.actor.{Actor, Props} | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.actors.JobLinkExtractorActor.JobPageCrawlInfo | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.actors.JobLinkExtractorChildActor.CrawlJobPage | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.naukri.Headers | |
| import com.jobllers.service.jobsscrapper.impl.scrapper.util.RemoteContentPuller | |
| import org.jsoup.Jsoup | |
| import org.jsoup.nodes.Element | |
| import scala.util.Try | |
| import scala.util.control.NonFatal | |
| class JobLinkExtractorActor extends Actor with RemoteContentPuller { | |
| var counter = 0 | |
| override def receive: Receive = { | |
| case jobPagesCrawlInfo: JobPageCrawlInfo ⇒ | |
| jobPagesCrawlInfo.uriList.foreach { jobPageURI ⇒ | |
| counter = counter + 1 | |
| println(s"[~~~~~~~~~~~~~~~~~~~~~~~~~~~] -> Listing will extracted from listing page.") | |
| context.actorOf(JobLinkExtractorChildActor.props, s"JobLinkExtractorActorChild$counter") | |
| .tell(CrawlJobPage(jobPageURI, jobPagesCrawlInfo.selector), self) | |
| } | |
| } | |
| } | |
| object JobLinkExtractorActor { | |
| val props = Props(classOf[JobLinkExtractorActor]) | |
| case class JobPageCrawlInfo(uriList: List[String], selector: String) | |
| } | |
| class JobLinkExtractorChildActor extends Actor with RemoteContentPuller { | |
| var retryCount = 0 | |
| override def receive: Receive = { | |
| case CrawlJobPage(uri: String, selector: String) ⇒ getJobLinksFromPage(uri, selector) | |
| case x: Any ⇒ println(s"Something weird received. $x") | |
| } | |
| private def getJobLinksFromPage(uri: String, selector: String): Unit = Try { | |
| import scala.collection.JavaConversions._ | |
| val page = Jsoup.connect(uri).headers(Headers.naukariHeaders).timeout(60000).get() | |
| val links = page.select(selector) | |
| .foldLeft(List.empty[String])((listOfLinks, link) ⇒ addVerifiedLinkToList(listOfLinks, link)) | |
| val nextPageLinkElement: List[Element] = page.select("div.srp_container.fl div.pagination a:last-child").toList | |
| context.actorOf(JobLinkHandlerActor.props) ! links | |
| //Start crawling the "NEXT" page if there are more links else kill itself. | |
| if (nextPageLinkElement.nonEmpty) { | |
| println(s"-> -> -> -> Next on link found.") | |
| self ! CrawlJobPage(nextPageLinkElement.last.attr("href"), selector) | |
| } else { | |
| println(s". . . . Stopping actor.") | |
| context stop self | |
| } | |
| } recover { | |
| case NonFatal(_: TimeoutException) ⇒ | |
| if (retryCount < 5) { | |
| println(s"Couldn't connect to the $uri while trying for 60 seconds. Retrying.") | |
| self ! CrawlJobPage(uri, selector) | |
| } else { | |
| context stop self | |
| } | |
| retryCount += 1 | |
| case NonFatal(_: IOException) ⇒ | |
| println(s"Got IOException while extracting job links from $uri") | |
| context stop self | |
| } | |
| } | |
| object JobLinkExtractorChildActor { | |
| val props = Props(classOf[JobLinkExtractorChildActor]) | |
| case class CrawlJobPage(uri: String, selector: String) | |
| } | |
| class JobLinkHandlerActor extends Actor { | |
| override def receive = { | |
| case links: List[String] ⇒ | |
| //Save these links into the database / file | |
| println(s"************************* ${links.size} links received successfully *************************") | |
| } | |
| } | |
| object JobLinkHandlerActor { | |
| val props = Props(classOf[JobLinkHandlerActor]) | |
| } |
The second class does behave more in the same manner. It delegates the work to child actors and when child actors complete their work, they stop themselves, after giving results back to parent actor and/or saving the result to database.
P.S.: The code above was written as out of curiosity. More emphasis was on to put the actors to work and getting juice out of my machine. There are many unused variables/ lines of code, many long println()’s, some CSS selector were used directly in code rather than keeping in a configuration file – no specific attention was paid to keep the code too clean. Next time, I’ll keep that in mind.
Where should we go from here on? Well, it was quite fascinating putting things at work and see the terminal with those println()’s but it’s still just the beginning. We’ve many things to do, in production we need to watch our actors. We need to create a feedback mechanism so that parent actor doesn’t push too much work on child actors (or there could be out of memory exception). Also, we need to create a fail-safe mechanism so that if system crashes or network breaks or job site blocks our IP for some time (due to too much traffic from on IP, this happened to me 😛), we can restart the things where left. This can be done with the help of the Akka Persistence.
Please comment to give any feedback or if anything you want me to do in future with actors.
Always interested in web-scraping projects — I’ve been doing these on and off for twenty years now. Just wanted to comment that (usually) the double consonant gets a short sound — scrapping a web site would mean getting rid of it (e.g. turning it into scrap or junk) — and that you should use the single consonant “scraping” or “scraper” to get the long A sound. Despite being a native speaker, I’m often tripped by these rules and their exceptions, but this one is pretty solid. English is a ridiculous language, to be fair, but this is the difference between “hoping” and “hopping”…
As for the scraping approach itself, you appear to be looking at a breadth-first search by retrieving all of the seed URLs and then adding the child URLs for each. You may prefer a depth-first approach, so that if the job is interrupted, you at least have complete data on the seed URLs or their children that have been completely scraped. Depth-first is also easier to write in a way that won’t exceed memory/connection limits (in my opinion).
Thank you so much Ken. I’ll update the keyword from scrappy to scrapy (I don’t have the access to make changes after publish, need to ask admin 😛 ). Thanks for pointing that out.
For your suggestion about the use of depth first search, my approach isn’t completely breadth first search. Once I’ve the seed links, I start the moving towards the end link (where is my actual information) and once I reach there I process that information right then and there and then that thread(which actor is running on) leaves the CPU resources.
This crawling process can be envisioned as- one branch gets created from seed link to the each information page (which can be labeled as leaf page). This branch has many actors in between and so many points of processing. Once the information has been acquired from leaf page the link from its parent disappears and the parent itself vanishes when all the children of that actor process the leaf pages they are intended to. After the parent actor (one level above the leaf page) is done with its responsibility it’s also stopped. And in this way all branches goes poof from leaf to seed page links.
If we run this same logic on single thread it’d be depth first search. But I should, indeed, make it complete depth first search scraping that way I’d be sure with the categories which I’ve processed already. Hmmm… I’ll make some changes in my code.
Thank you so much for guiding me 🙂
Reblogged this on Coding, Unix & Other Hackeresque Things.