• basic
  • syntax for jq
  • dealing with json objects
  • slicing and filtering
  • mapping and transforming
• join
• split
• replacing
• builtin operators
  • debug
  • select
  • contains
  • inside
  • toUpperCase : ascii_upcase
  • to_entries
  • try to_entries
  • from_entries
  • with_entries
  • as
• tricky
  • space in the key
  • get urlencode
  • string to json
  • map

basic

syntax for jq

dealing with json objects

slicing and filtering

[!NOTE|label:references:]

• select by type: with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

mapping and transforming

join

[!TIP] references:

• to output multiple values on a single line
• join multiple values

$ echo '{ "some": "thing", "json": "like" }' |
       jq -r '[.some, .json] | join(":")'
thing:like

• or

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | "\(.[0]) \(.[1])"'
  thing like
  

• or

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | "\(.[0]): \(.[1])"'
  thing: like
  

• or with reduce

  $ echo '{ "some": "thing", "json": "like" }' |
         jq -r '[.some, .json] | reduce .[1:][] as $i ("\(.[0])"; . + ",\($i)")'
  thing,like
  

• or

  $ echo '{ "k1": "v1", "k2": "v2", "k3": "v3", "k4": "v4" }' |
         jq -r '[.k1, .k2, .k3] | reduce .[1:][] as $i ("\(.[0])"; . + ",\($i)")'
  v1,v2,v3
  

• or with .first and .last

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder" }, { "first": "Michael", "last": "Jackson" } ] }' |
         jq -r '.users[] | .first + " " + .last'
  Stevie Wonder
  Michael Jackson
  

• or

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder" }, { "first": "Michael", "last": "Jackson" } ] }' |
         jq -r '.users[] | .first + " " + (.last|tostring)'
  

• or

  $ echo '{ "users": [ { "first": "Stevie", "last": "Wonder", "number": 1 }, { "first": "Michael", "last": "Jackson", "number": 2 } ] }' |
         jq -r '.users[] | .first + " " + (.number|tostring)'
  Stevie 1
  Michael 2
  

split

[!TIP] references:

• remove a substring from a string
• split(str))
  • Splits an input string on the separator argument.
  • example&j=%22a%2C%20b%2Cc%2Cd%2C%20e%2C%20%22)

    jq 'split(", ")'
    Input "a, b,c,d, e, "
    Output  ["a","b,c,d","e",""]
    

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri'
/1
/2
/3

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri | split("/")[1]'
1
2
3

• try online

  $ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
          jq '.[].uri | split("/")[]'
  ""
  "1"
  ""
  "2"
  ""
  "3"
  $ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
          jq -r '.[].uri | split("/")'
  [
    "",
    "1"
  ]
  [
    "",
    "2"
  ]
  [
    "",
    "3"
  ]
  

replacing

[!TIP|label:references:]

• Replacing substrings in a string
• sub(regex; tostring), sub(regex; string; flags)
  • Emit the string obtained by replacing the first match of regex in the input string with tostring, after interpolation. tostring should be a jq string, and may contain references to named captures. The named captures are, in effect, presented as a JSON object (as constructed by capture) to tostring, so a reference to a captured variable named "x" would take the form: "(.x)".

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri'
/1
/2
/3

$ echo '[{"uri" : "/1" }, {"uri" : "/2"}, {"uri" : "/3"}]' |
        jq -r '.[].uri | sub("/"; "")'
1
2
3

builtin operators

[!NOTE|label:references:]

• jq manual - Builtin operators and functions

debug

• without debug

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) )'
  

• with debug:

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) | debug )'
  ["DEBUG:",false]
  ["DEBUG:",false]
  ["DEBUG:",true]
  {
    "id": "second",
    "val": 2
  }
  ["DEBUG:",false]
  ["DEBUG:",false]
  ["DEBUG:",true]
  {
    "id": "SECOND",
    "val": 3
  }
  

select

[!NOTE|label:refrences:]

• imarslo: example on jenkins api analysis
• imarslo: example on gerrit api analysis
• jq tips : remove emtpy line
• Filter empty and/or null values with jq

  to_entries[]
  | select(.value | . == null or . == "")
  | if .value == "" then .value |= "\"\(.)\"" else . end
  | "\(.key): \(.value)"
  

$ echo "[1,5,3,0,7]" |
       jq 'map(select(. >= 2))'
[
  5,
  3,
  7
]

• or

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2}]''' |
         jq '.[] | select(.id == "second")'
  {
    "id": "second",
    "val": 2
  }
  

• select with any

  [!NOTE|label:references:]

  • any(.attributes; .enabled == true) == any(.attributes; .enabled) == .attributes.enabled == true == .attributes.enabled
  • try online

  $ cat sample.json
  [
    {
      "attributes": {
        "created": "2021-10-18T12:02:39+00:00",
        "enabled": true,
        "expires": null,
        "notBefore": null
      },
      "contentType": null,
      "id": "https://kjkljk./secrets/-/1",
      "managed": null,
      "name": "pw",
      "tags": {}
    },
    {
      "attributes": {
        "created": "2021-10-18T12:06:16+00:00",
        "enabled": false,
        "expires": null,
        "notBefore": null
      },
      "contentType": "",
      "id": "https://kjklj./secrets/-/2",
      "managed": null,
      "name": "pw",
      "tags": {}
    }
  ]
  
  # get [{.id}] format
  $ cat sample.json |
        jq -r 'map(select(any(.attributes; .enabled)) | {id})'
  [
    {
      "id": "https://kjkljk./secrets/-/1"
    }
  ]
  
  # get [.id] format
  $ cat sample.json | jq -r 'map(select(any(.attributes; .enabled)) | .id)'
  [
    "https://kjkljk./secrets/-/1"
  ]
  
  # get .id format ( without `map()` )
  $ cat sample.json | jq -r '.[] | select(any(.attributes; .enabled)) | .id'
  https://kjkljk./secrets/-/1
  
  # re-format
  $ cat sample.json | jq -r '{"ids": .[] | select(any(.attributes; .enabled)) | .id}'
  {
    "ids": "https://kjkljk./secrets/-/1"
  }
  

contains

[!NOTE|label:reference:]

• imarslo: example on artifactory api analysis
• imarslo: example on list Error pods in kuberetnes

$ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second-one", "val": 3}]''' |
       jq '.[] | select( .id | contains("second") )'
{
  "id": "second",
  "val": 2
}
{
  "id": "second-one",
  "val": 3
}

• by using test

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second.one", "val": 3}]'''  |
         jq '.[] | select( .id | test("sec*") )'
  {
    "id": "second",
    "val": 2
  }
  {
    "id": "second.one",
    "val": 3
  }
  

inside

$ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "second-one", "val": 3}]'''  |
       jq '.[] | select( [.val] | inside([2,3]) )'
{
  "id": "second",
  "val": 2
}
{
  "id": "second-one",
  "val": 3
}

• or

  $ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
         jq '.[] | select( .val == (2, 3) )'
  {
    "id": "second",
    "val": 2
  }
  {
    "id": "SECOND",
    "val": 3
  }
  

toUpperCase : ascii_upcase

$ echo '''[{"id": "first", "val": 1}, {"id": "second", "val": 2},  {"id": "SECOND", "val": 3}]'''  |
       jq '.[] | select(.id | ascii_upcase == "SECOND")'
{
  "id": "second",
  "val": 2
}
{
  "id": "SECOND",
  "val": 3
}

to_entries

[!NOTE] references:

• jq: filter input based on if key ends with specified string

# original
$ echo '{ "name" : "marslo" }' | jq -r
{
  "name": "marslo"
}

# `to_entries[]`
$ echo '{ "name" : "marslo" }' | jq -r 'to_entries[]'
{
  "key": "name",
  "value": "marslo"
}

# or `to_entries`
$ echo '{"a": 1, "b": 2}' | jq -r to_entries
[
  {
    "key": "a",
    "value": 1
  },
  {
    "key": "b",
    "value": 2
  }
]

• example to get key and value

  $ echo '{ "some": "thing", "json": "like" }' \
         | jq -r 'to_entries[] | "\(.key)\t\(.value)"'
  some    thing
  json    like
  

  • or output format to @csv

    $ echo '{ "some": "thing", "json": "like" }' |
           jq -r '[.some, .json] | @csv'
    "thing","like"
    

  • or output format to @tsv

    $ echo '{ "some": "thing", "json": "like" }' |
           jq -r '[.some, .json] | @tsv'
    thing like
    

• to_entries and select

  # orignal
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' | jq -r
  {
    "name/": "marslo",                 # wants value if key ends with '/'
    "age/": "18",                      # wants value if key ends with '/'
    "citizenship": "china"
  }
  
  # select
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
         jq -r 'to_entries[] | select(.key|endswith("/")) '
  {
    "key": "name/",
    "value": "marslo"
  }
  {
    "key": "age/",
    "value": "18"
  }
  
  # get `.value` after selected
  $ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
         jq -r 'to_entries[] | select(.key|endswith("/")) | .value'
  marslo
  18
  

try to_entries

$ cat sample.json
[
  {
    "name": "x",
    "hobby": [                     // < has hobby
      "baseball",
      "baseketball"
     ]
  },
  {
    "name": "y"                    // < no hobby
  }
]

# to_entries[] directly will cause issue
$ cat sample.json |
      jq '.[] | .name as $n | .hobby | to_entries[] | [$n, .value]'
[
  "x",
  "baseball"
]
[
  "x",
  "baseketball"
]
jq: error (at <stdin>:12): null (null) has no keys

# try to_entries[]
#                                      try
#                                       v
$ cat sample.json |
      jq '.[] | .name as $n | .hobby | try to_entries[] | [$n, .value]'
[
  "x",
  "baseball"
]
[
  "x",
  "baseketball"
]

from_entries

$ echo '[{"key":"a", "value":1}, {"key":"b", "value":2}]' |
       jq -r from_entries
{
  "a": 1,
  "b": 2
}

with_entries

[!NOTE|label:references:]

• JQ: Filtering for keys

$ echo '{"a": 1, "b": 2}' |
       jq 'with_entries(.key |= "KEY_" + .)'
{
  "KEY_a": 1,
  "KEY_b": 2
}

• to get particular branch permissions from Gerrit

  # with_entries
  $ curl -fsSL https://gerrit.domain.com/a/projects/$(printf %s "path/to/sandbox" | jq -sRr @uri)/access |
         tail -n+2 |
         jq -r '.local | with_entries( if(.key | test("^.+sandbox/marslo.+$")) then ( {key: .key, value: .value } ) else empty end )'
  
  # try to_entries[]
  $ curl -fsSL https://gerrit.domain.com/a/projects/$(printf %s "path/to/sandbox" | jq -sRr @uri)/access |
         tail -n+2 |
         jq -r '.local | try to_entries[] | select(.key | test("^.+sandbox/marslo.+$")) | .value'
  

as

[!NOTE|label:references]

• jq: filter input based on if key ends with specified string

$ echo '{ "name/" : "marslo", "age/" : "18", "citizenship" : "china" }' |
       jq -r '. as $o | keys_unsorted[] | select(endswith("/")) | $o[.]'
marslo
18

tricky

space in the key

$ echo '{ "k1 name": "v1", "k2 name": "v2", "k3": "v3", "k4": "v4" }' |
       jq -r '."k1 name"'
v1

get urlencode

[!TIP|label:references:]

• Percent-encoding

$ printf %s 'input text' | jq -sRr @uri
input%20text

$ printf %s 'input=(text)' | jq -sRr @uri
input%3D%28text%29

$ printf %s '(=)&' | jq -sRr @uri
%28%3D%29%26

string to json

$ echo [
  "foo1:         xxxxxx",
  "foo2:    xxxxxx",
  "foo3:     xxxxxx",
  "foo4:         xxxxxx",
  "foo5:   xxxxxx",
  "foo6:       xxxxxx"
] | jq -r 'map(capture("^(?<key>.*):\\s*(?<value>.*)$")) | from_entries'
{
  "foo1": "xxxxxx",
  "foo2": "xxxxxx",
  "foo3": "xxxxxx",
  "foo4": "xxxxxx",
  "foo5": "xxxxxx",
  "foo6": "xxxxxx"
}

map

[!NOTE|label:references:]

• semantics of map on a sequence of objects in jq
  • map( ... ) ≡ [ .[] | ... ], which map( . ) ≡ [ .[] | . ] ≡ [ .[] ]
  • for array: map( . ) ≡ [ .[0], .[1], ... ] ≡ .
  • for object: map( . ) ≡ [ .["key1"], .["key2"], ... ]