Parsing HTML Table to CSV Through Shell Script

A few days back, we got the requirement that we need to parse an HTML table through shell script and generate a report based on the contents of the table. I tried many things and after much brainstorming created a shell script that parses the HTML tables into a CSV file. I will try to explain each and every step of this script.

The Script:

As we can see that this script is divided into four aptly functions. Let’s discuss the functions one by one.

First, we come across the html_clean function. This function removes all the HTML syntax from the HTML page and it creates a text file which contains the data which was originally in the table which needed to be parsed. We can change the values in the sed command to get various desired (also undesired) effects in the output text file.

Then, there is clean_text function. This function needs some knowledge about the tables that are being utilized here. As I knew that there are 6 rows in my table, so I created a loop which will automatically format the text file in such a way that each row is in a separate line. Reading and manipulating this text file is much easier than the output from the previous function.

Now, we come across create_csv() function. As the name suggests, this function is responsible to read the file from the previous function and convert it into a CSV file. This function has been optimized to utilize the output of the previous function.

The last function check_status is just to create a report which in this case is all of the records lying in a particular time range. We can put any logic in here and in the end, we get a CSV file which can be used for further manipulation. In my case, the CSV file was used in an automation pipeline to check the status of some finished jobs of some application.

Hope, this blog helps. Happy Reading !!!


Written by 

Sudeep James Tirkey is a software consultant having more than 0.5 year of experience. He likes to explore new technologies and trends in the IT world. His hobbies include playing football and badminton, reading and he also loves travelling a lot. Sudeep is familiar with programming languages such as Java, Scala, C, C++ and he is currently working on DevOps and reactive technologies like Jenkins, DC/OS, Ansible, Scala, Java 8, Lagom and Kafka.

Knoldus Pune Careers - Hiring Freshers

Get a head start on your career at Knoldus. Join us!