Introduction to Jq!

Hello guys, in this blog we will be discussing about what is jq and how to use it for reading and manipulating JSON data and files.

What is JQ?

jq is a lightweight and flexible command-line JSON processor. It is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

jq is a fantastic command-line JSON processor. It plays nice with UNIX pipes and offers extensive functionality for interrogating, manipulating and working with JSON file.

Now let us see same basic uses of jq.

How to pretty print JSON

jq can do a lot but probably the highest frequency use for most users is to pretty print JSON either from a file or after a network call. Suppose that we have a file people.json containing the following json.

[{"id": 1, "name": "Arthur", "age": "21"},{"id": 2, "name": "Richard", "age": "32"}]

jq can pretty print this file using the . filter. This takes the entire input and sends it to standard output. Unless told not to jq will pretty print making JSON readable.

> jq '.' people.json
[
  {
    "id": 1,
    "name": "Arthur",
    "age": "21"
  },
  {
    "id": 2,
    "name": "Richard",
    "age": "32"
  }
]

How to use pipes with jq

Because jq is UNIX friendly it is possible to pipe data in and out of it. This can be useful for using jq as a filter or interacting with other tools. In the following pipeline cat pipes the file into jq and this is piped onto less. This can be very useful for viewing large JSON files.

> cat people.json | jq '.' | less

How to find a key and value

To find a key and value jq can filter based on keys and return the value. Taking the previously used file people.json, we can look how it is done.

> jq '.[].name' people.json
"Arthur"
"Richard"

Multiple keys can by passed separated by commas.

> jq '.[].name,.[].age' people.json
"Arthur"
"Richard"
"21"
"32"

How to find items in an array

To search for items in arrays use bracket syntax with the index starting at 0.

> jq .[0] people.json
{
  "id": 1,
  "name": "Arthur",
  "age": "21"
}

Multiple elements of an array may also be returned.

> echo '["a","b","c","d","e"]' | jq '.[2:4]'
[
  "c",
  "d"
]

How to combine filters

jq can combine filters to search within a selection. For the people.json JSON document suppose that the names need to be filtered..

This can be achieved with a pipe with the jq filter.

> jq '.[] | .name' people.json
"Arthur"
"Richard"

How to transform JSON

jq can be used for more than just reading values from a JSON object. It can also transform JSON into new data structures. Let us use a new JSON named dog.json.

{
  "name": "Buster",
  "breed": "Golden Retriever",
  "age": "4",
  "owner": {
    "name": "Sally"
  },
  "likes": [
    "bones",
    "balls",
    "dog biscuits"
  ]
}

A new array can be created containing the name and likes as follows.

> jq '[.name, .likes[]]' dog.json
[
  "Buster",
  "Bones",
  "balls",
  "dog biscuits"
]

This can be very useful in data transform pipelines to shift JSON data from one structure to another.

How to transform data values within JSON

jq can also operate on data within JSON objects. Suppose the following JSON file exists and is saved as inventory.json.

{ "eggs": 2, "cheese": 1, "milk": 1 }

jq can be used to perform basic arithmetic on number values.

> jq '.eggs + 1' inventory.json
3

How to remove keys from JSON

jq can remove keys from JSON objects. This outputs the JSON object without the deleted key. Suppose the following JSON is saved as cheeses.json.

{ "maroilles": "stinky", "goat": "mild" }

jq can remove keys as follows leaving just wonderful stinky cheese.

> jq 'del(.goat)' cheeses.json
{ "maroilles": "stinky" }

How to map values

jq can map values and perform an operation on each one. In the following example each item in an array is mapped and has two subtracted.

> echo '[12,14,15]' | jq 'map(.-2)'
[
  10,
  12,
  13
]

Conclusion

In this blog, we looked at the basics of jq and their usage. We will be further exploring more about jq in more blogs to follow.

Thanks for reading!

References

https://stedolan.github.io/jq/
https://stedolan.github.io/jq/manual/
https://thoughtbot.com/blog/jq-is-sed-for-json
https://blog.appoptics.com/jq-json/
http://www.compciv.org/recipes/cli/jq-for-parsing-json/