Nowadays Rest API is the most widely used way to share data, In which many API returns a subset of complete data in form of page.
Sometimes we need to append multiple query param in the URL to get some specific and filtered data.
In this blog, we will learn how to generate dynamic URLs by adding query param and get data.
Knime platform supports Rest interface with Get-Request and Post-Request Node.
Accessing Rest Interface in Knime
Below is the Knime Analytic workflow which include all the Knime Nodes required to access Rest API in this blog are as below:
Explaining the Configuration of each Node
Table Creator
This node is with default configuration, It read the table from the native Knime format, The Performance is
Counting Loop Start
The Loop Start is the node that starts a loop that is executed a predefined number of times. At the end of the loop you need LoopEnd, which collects the results from all loop iterations. All nodes in between are executed as many times as you specify in the dialog of LoopStart.
In this node configuration we give the number of time the loop will run.
Math Formula
This node evaluates a mathematical expression based on the values in a row or given expression. The computed results can be either appended as a new column or a variable.
In this node we configure the increment on the loop index, in this case, it is increasing by 1.
String Manipulation
This node Manipulates strings like Search and Replace, to Uppercase remove leading and trailing white spaces.
Here this node will append the page number to the fix part of URL
Get Request
This node is actually responsible for fetching the data from the Rest URL, Here we can hard code the URL if there is no dynamic query param.
Or we can take the URL from the variable that we created in the last node with query param as shown in the below screenshot.
Also, there are other different configurations we can do in our case it is not required.
Loop End
This node will be executed which will execute all the node inside Counting loop Start and Loop End, this we can leave with default configuration.
JSON Writer & MongoDB Writer
After we get the data from Rest API we can save it to disk or any database.
JSON Writer: Writes the JSON values to JSON file, In configuration just give folder location to write the file.
MongoDB Writer: The MongoDB Writer node inserts documents in JSON cells into a MongoDB database. It needs just the connection details.
Conclusion:
Source of data can be multiple, Knime support almost all source of data to be analysed and processed.
The above workflow was small example of reading data from rest API with data in multiple pages.
