Knime: Accessing a REST API with dynamic query param

Reading Time: 3 minutes

Nowadays Rest API is the most widely used way to share data, In which many API returns a subset of complete data in form of page.

Sometimes we need to append multiple query param in the URL to get some specific and filtered data.

In this blog, we will learn how to generate dynamic URLs by adding query param and get data.

Knime platform supports Rest interface with Get-Request and Post-Request Node.

Accessing Rest Interface in Knime

Below is the Knime Analytic workflow which include all the Knime Nodes required to access Rest API in this blog are as below:

Explaining the Configuration of each Node

Table Creator

This node is with default configuration, It read the table from the native Knime format, The Performance is

Counting Loop Start

The Loop Start is the node that starts a loop that is executed a predefined number of times. At the end of the loop you need LoopEnd, which collects the results from all loop iterations. All nodes in between are executed as many times as you specify in the dialog of LoopStart.

In this node configuration we give the number of time the loop will run.

Math Formula

This node evaluates a mathematical expression based on the values in a row or given expression. The computed results can be either appended as a new column or a variable.

In this node we configure the increment on the loop index, in this case, it is increasing by 1.

String Manipulation

This node Manipulates strings like Search and Replace, to Uppercase remove leading and trailing white spaces.

Here this node will append the page number to the fix part of URL

Get Request

This node is actually responsible for fetching the data from the Rest URL, Here we can hard code the URL if there is no dynamic query param.

Or we can take the URL from the variable that we created in the last node with query param as shown in the below screenshot.

Also, there are other different configurations we can do in our case it is not required.

Loop End

This node will be executed which will execute all the node inside Counting loop Start and Loop End, this we can leave with default configuration.

JSON Writer & MongoDB Writer

After we get the data from Rest API we can save it to disk or any database.

JSON Writer: Writes the JSON values to JSON file, In configuration just give folder location to write the file.

MongoDB Writer: The MongoDB Writer node inserts documents in JSON cells into a MongoDB database. It needs just the connection details.

Conclusion:

Source of data can be multiple, Knime support almost all source of data to be analysed and processed.

The above workflow was small example of reading data from rest API with data in multiple pages.

Written by 

A developer trying to learn and share always.