Ansible, YAML, and JSON

Ansible is very flexible with data types, and easily transforms primitives and even JSON strings, but from time to time you need to process and transform something a bit more complex than String to Boolean conversions.

Ansible is not the first choice for data transformation, but its core functions are Python-based, baked by Jinja2 templates. Let's see how you can leverage them for data processing. For this example, I'm going to use a simple shell script that does nothing but a set of lines similar to the screenshot below.  I should note, that Oracle Opatch 12c utility produces quite a similar patch list report.

My goal is a playbook that receives the raw data and produces a list of objects. Let start with the data capture.

---
- name: Process List of Patches 
  hosts: localhost
  vars: 
    patch_list: []
  tasks:
    - name: Receive External Data
      tags:
       - always
      shell: 
        cmd: |
         ./shell-output.sh
      register:  os_output
    - name:  Raw Output Lines
      debug:
        verbosity: 2 
        var: os_output.stdout_lines

Now, all that we need is to transform the list of strings into a list of objects. I'm going to use YAML multiline definitions  '|' and '>' similar to the shell task above. The new task would generate a JSON-compliant string and convert it into a list.  

    - name: Transformation v1
      tags:
        - v1
      set_fact: 
       patch_list: |-
          [
          {%- for line in os_output.stdout_lines -%}
            { "id": "{{ line.split(';')[0] }}",
              "title":"{{ line.split(';')[1] }}" }
              {{ ", " if not loop.last else "" }}
          {%- endfor -%}
          ]
    - name: Result For V1
      tags: 
        - v1
      debug:
        var: data_list

The code above assigns the Jinja2 template to the variable data list. The template has a 'for' control to iterate over the output lines and generate JSON. A few things that may require an explanation:

• Multi-line starts with '|-'. It keeps new line characters and helping to avoid syntax errors. Some parsers allow new lines before JSON data but throw an error for whitespaces. The dash instructs to remove all extra newlines after the main text.
• Loop controls are surrounded by {%- -%} where '-' mandates left and right trims for the lines. The resulting output would have no empty lines between data elements.
• The end of the line template checks if the current line is the last one. If it's not it adds a comma character to the element definition.

As an output playbook prints structured data - a list of objects with two attributes each. Here is the output.

I have one more conversion task to the final playbook. It looks a bit more complex, but it could be useful if you need to generate nested lists and dictionaries. The full source code is available on Github.