Ansible lookup plugin throws TypeError: encoding without a string argument

I’m trying to use the anatomicjc.passbolt collection in our Ansible setup. We use Ansible in combination with AWX and have the collection as well as py-passbolt installed in the AWX Ansible Execution Environment.
The URL/Private key and pasphrase are set up as a Credential type within AWX, which exposes those strings as environment variables during the run of the playbook.

My playbook looks like this:

 name: Test passbolt lookup plugin
  hosts: localhost
  gather_facts: no
  tasks:
    - name: check environment
      ansible.builtin.debug:
        msg: |
          PASSBOLT_BASE_URL: {{ lookup('ansible.builtin.env', 'PASSBOLT_BASE_URL') }}
          PASSBOLT_PRIVATE_KEY: {{ lookup('ansible.builtin.env', 'PASSBOLT_PRIVATE_KEY') }}
          PASSBOLT_PASSPHRASE: {{ lookup('ansible.builtin.env', 'PASSBOLT_PASSPHRASE') }}
    - name: Lookup predefined resource
      ansible.builtin.debug:
        msg: "Password is: {{ lookup('anatomicjc.passbolt.passbolt', 'Ansible predefined test resource').password }}"

Output:

ansible-playbook [core 2.15.0]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
No config file found; using defaults
Vault password: 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: passbolt-test.yml ****************************************************
1 plays in passbolt-test.yml

PLAY [Test passbolt lookup plugin] *********************************************

TASK [check environment] *******************************************************
task path: /runner/project/passbolt-test.yml:6
ok: [localhost] => {
    "msg": "PASSBOLT_BASE_URL: https://passbolt.domain.name\\nPASSBOLT_PRIVATE_KEY: -----BEGIN PGP PRIVATE KEY BLOCK-----\\\\n\\\\nXXXXXXXXXXXX\\\\nXXXXXXXXXXXX\\\\n....\\\\n-----END PGP PRIVATE KEY BLOCK-----\\nPASSBOLT_PASSPHRASE: XXXXXXX\\n"
}

TASK [Lookup predefined resource] **********************************************
task path: /runner/project/passbolt-test.yml:12
exception during Jinja2 execution: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/ansible/template/__init__.py", line 831, in _lookup
    ran = instance.run(loop_terms, variables=self._available_variables, **kwargs)
  File "/usr/share/ansible/collections/ansible_collections/anatomicjc/passbolt/plugins/lookup/passbolt.py", line 278, in run
    self.passbolt_init(variables, kwargs)
  File "/usr/share/ansible/collections/ansible_collections/anatomicjc/passbolt/plugins/lookup/passbolt.py", line 248, in passbolt_init
    self.p = PassboltAPI(dict_config=self.dict_config)
  File "/usr/local/lib/python3.9/site-packages/passbolt/__init__.py", line 28, in __init__
    self.key, _ = PGPKey.from_blob(self.config.get("private_key"))
  File "/usr/local/lib/python3.9/site-packages/pgpy/types.py", line 195, in from_blob
    po = obj.parse(bytearray(blob, 'latin-1'))
TypeError: encoding without a string argument
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'anatomicjc.passbolt.passbolt'. Error was a <class 'TypeError'>, original message: encoding without a string argument. encoding without a string argument"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

As you can see, the environment variables are set and readable by ansible.builtin.env (replaced sensitive strings here for posting). But somehow it seems not to be picked up correctly by anatomicjc.passbolt or py-passbolt as the error tells me that the passed blob coming from self.config.get("private_key") is not a string?

Any idea what could be going wrong here ?

Hi @Robinr :wave:

Thanks for trying out this lookup plugin.

Can you try to add an environment section in your playbook like this:

 name: Test passbolt lookup plugin
  environment:
    PASSBOLT_BASE_URL: "{{ lookup('ansible.builtin.env', 'PASSBOLT_BASE_URL') }}"
    PASSBOLT_PRIVATE_KEY: "{{ lookup('ansible.builtin.env', 'PASSBOLT_PRIVATE_KEY') }}"
    PASSBOLT_PASSPHRASE: "{{ lookup('ansible.builtin.env', 'PASSBOLT_PASSPHRASE') }}"
  hosts: localhost
  gather_facts: no
  tasks:
...

I’m not sure it will solve your problem as the error message is encoding without a string argument.

If your PASSBOLT_PRIVATE_KEY is encrypted with ansible-vault, maybe could you add a | string Jinja2 filter, like this:

 name: Test passbolt lookup plugin
  environment:
    PASSBOLT_BASE_URL: "{{ lookup('ansible.builtin.env', 'PASSBOLT_BASE_URL') }}"
    PASSBOLT_PRIVATE_KEY: "{{ lookup('ansible.builtin.env', 'PASSBOLT_PRIVATE_KEY') | string }}"
    PASSBOLT_PASSPHRASE: "{{ lookup('ansible.builtin.env', 'PASSBOLT_PASSPHRASE') }}"
  hosts: localhost
  gather_facts: no
  tasks:
...

I usually add this | string filter to vaulted variables.

Here is an example:

users_pwd:
  user1:
    pwd: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      64656436313631363838646665643338316363386534636663313338396539613036343232336361
      6265663639386233303731373733353163633166396633640a393033393538323131393061323964
      37666138336134633839353966663336313466343133313431396165363564646636636465333964
      6230343130363636390a313462613865653838643530613931236431666234363962653337323138
      35626232323437636130376130316466373462653862396631633635613336646639
    pwd_salt: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      66646264346531323166663037326632633234313162646266306666313335613036366235393837
      3135306135626432393935306333353161366634353339310a623734623239326335333563306430
      61313563386538633738653133303866633135623664666661636112333338353265323265323034
      3739343863623163370a306231633630343534343561366465353238323534643339643134666234
      31343232623338383739326533303761316530653337613130386433376434653533
...

user1:
  uid: "1000"
  createhome: "yes"
  group: "user1"
  password: "{{ users_pwd.user1.pwd | string | password_hash('sha512', users_pwd.sas.pwd_salt) }}"
  groups:
    - "docker"

Hope this helps,

Hi @AnatomicJC, Thanks for your reply…
Unfortunately adding the environment section makes no difference, neither does adding the | string filter (despite the values in the OS environment not being vaulted; but I tried to be sure).

I have also tried defining the variables inside Ansible inventory as vaulted strings, instead of using the OS environment variables. But also to no avail; I always keep getting the exact same error: “encoding without a string argument”.

On the other hand, I have been using username/password authentication through OS environment variables already on the community.vmware collection which gives the choice to authenticate using module parameters or OS environment variables VMWARE_HOST, VMWARE_USER, VMWARE_PASSWORD. And there I never had any trouble to have the collection use those OS env variables (also without environment: block in the plays, just the variables present in the OS env). So I’m assuming it has nothing to do with the AWX Ansible Execution Environment setup.

I’m not very experienced with Python, but I checked the code of the community.vmware collection. And they seem to be using ansible.module_utils.common.parameters.env_fallback in vmware_rest_client.py while you seem to implement your own fallback to environment method (while I don’t immediately see where it could be going wrong in your implementation, it may be better to use the built-in method?).

I had a look at your error, I would say you private key is not stored in an expected format for the pgpy library.

The private key is read from there in the passbolt-py library:

And this method from PGPy is invoked:

Some hints:

In /usr/local/lib/python3.9/site-packages/passbolt/init.py near line 28, you can try to add some debug like printing type of self.config.get("private_key"), I am interested by the result here:

print(type(self.config.get("private_key")))

Did you read the last section of the readme about how to format private key for environment variables: GitHub - passbolt/lab-passbolt-py: Python library for Passbolt API

As an alternative to environment variable, maybe can you try to store the private key in ansible inventory and encrypt it with ansible-vault, as I did here: lab-passbolt-ansible-poc/playbooks/example-playbook.yml at main · passbolt/lab-passbolt-ansible-poc · GitHub

On my side, I will try to setup an AWX instance and try to reproduce your issue.

I will let you know.

After adding the debug code you provided I found out the type of the private_key variable is:

<class 'ansible.parsing.yaml.objects.AnsibleVaultEncryptedUnicode'>

Which didn’t add up as there is currently no vaulted string in the play nor the environment. So I investigated further and found out that somehow AWX didn’t overwrite it’s internal list of host/group vars when syncing from the inventory source (while it is set to actually do so) and there was still a leftover definition of those PASSBOLT_ vars from previous tests which where no longer present in the inventory source. Hence my confusion.

Anyway, I have now cleared those definitions from the AWX internal inventory and now I have this as type for the string:

<class 'ansible.utils.unsafe_proxy.NativeJinjaUnsafeText'>

And the error changed into:

Error was a <class 'ValueError'>, original message: Expected: ASCII-armored PGP data.

Not sure about the error, I removed the environment: block from the play hoping that the lookup plugin would just take the OS environment variables as originally intended.
The type now has changed to:

<class 'str'>

which looks perfect to me :slight_smile:

But the play still fails with the error Expected: ASCII-armored PGP data.
So it seems now the problem is actually the formatting of the key.

For that I did follow the procedure and executed sed -z 's/\n/\\n/g' passbolt-recovery-kit.txt to get the required string.
This string is formatted like this:

-----BEGIN PGP PRIVATE KEY BLOCK-----\n\nXXXXXXXX\nXXXXXXXX\n...\n-----END PGP PRIVATE KEY BLOCK-----\n

I finally solved it.
It turned out I had to define the AWX credential field for the private key as a “multiline” field and paste the private key as is (without newline conversion) into the field. AWX then automatically replaces all newlines to “\n” when putting it into an OS environment variable of the Execution Environment container.

As a result I now get this output:

TASK [Lookup predefined resource] **********************************************
task path: /runner/project/passbolt-test.yml:16
<class 'str'>
/usr/local/lib/python3.9/site-packages/pgpy/constants.py:192: CryptographyDeprecationWarning: IDEA has been deprecated
  bs = {SymmetricKeyAlgorithm.IDEA: algorithms.IDEA,
/usr/local/lib/python3.9/site-packages/pgpy/constants.py:194: CryptographyDeprecationWarning: CAST5 has been deprecated
  SymmetricKeyAlgorithm.CAST5: algorithms.CAST5,
/usr/local/lib/python3.9/site-packages/pgpy/constants.py:195: CryptographyDeprecationWarning: Blowfish has been deprecated
  SymmetricKeyAlgorithm.Blowfish: algorithms.Blowfish,
ok: [localhost] => {
    "msg": "Password is: MH6-%M32k=TV=r'-J)"
}

(no wories, password is not used anywhere and is just generated as a test entry)

I assume I can ignore those warnings.

For sake of completeness and in case you want to add this to the documentation of the plugin. To be able to use the plugin with AWX, these are the steps you have to perform:

  • Build a custom AWX Execution Environment using Ansible Builder:
    • add
      py-passbolt
      
      to requirements.txt
    • add
      collections:
          - name: anatomicjc.passbolt
      
      to requirements.yml
  • Add a new Custom Credential Type to AWX:
    • Name: Passbolt Credentials
    • Description: Passbolt credentials for accessing Passbolt
    • Configuration input:
    fields:
      - id: passbolt_url
        type: string
        label: Passbolt Base URL
      - id: passbolt_private_key
        type: string
        label: Passbolt Private GPG Key
        secret: true
        multiline: true
      - id: passbolt_passphrase
        type: string
        label: Passbolt Private GPG Key Passphrase
        secret: true
    required:
      - passbolt_url
      - passbolt_private_key
      - passbolt_passphrase
    
    • Configuration injector:
      env:
        PASSBOLT_BASE_URL: '{{ passbolt_url }}'
        PASSBOLT_PASSPHRASE: '{{ passbolt_passphrase }}'
        PASSBOLT_PRIVATE_KEY: '{{ passbolt_private_key }}'
      
  • Add a new credential of the type Passbolt Credentials to AWX:
    • Set the url and passphrase
    • Upload or paste the contents of the Private key file into the Passbolt Private GPG Key field without any modifications
  • Create or update an AWX template to use the custom EE and add the above defined Passbolt Credentails. The playbook executed by this template will now have access to passbolt using the lookup plugin.

Hi,

Many thanks for this documentation @Robinr

I pushed a new release including your documentation:

Cheers,

2 Likes

@AnatomicJC, thanks, but I noticed a small error/typo. I submitted a new pull request to fix it :slight_smile:

Thanks, I merged it

Regards,