I have not set this up on a Google domain, so these are general thoughts:
It’s more a matter of how email is received than the restriction settings were are reviewing.
A server’s declared hostname is commonly checked against IP addresses. Maybe the Passbolt machine is internal-only and not public-facing and when Google tries to verify the domain it cannot. Hard to know, because this information was not provided yet. Using the IP address with a HELO command still has to be checked against the actual IP address from where the request is being made.
“Step-2” on the link above is to set the on-premises relay to point to Google. This thread however is treating a Passbolt machine as a relay. In the Configuration section of Step-2 it says:
We recommend that you configure your mail server to present a unique identifier (such as your domain name or the name of your mail server) in the HELO or EHLO command in the SMTP relay connections your server makes to Google. Avoid using generic names such as “localhost” or “smtp-relay.gmail.com,” which can occasionally result in issues with DoS limits.
So, my thought is that although the IP address is used for restriction, the declared domain (hostname) of the server is also being considered.
In my own mail server that I run, I have domain restrictions. The connecting servers can have credentials but my mail server will still not accept the connection if they do not present a whitelisted domain. In the case of an internal-only machine, this is even more relevant because my mail server cannot resolve it’s domain through normal means.
So in addition to a whitelisted domain, I also must note the IP address it has or else it gets flagged as incorrectly resolved.