500 internal server error when inserting or updating a record

We host passbolt ourselves with a docker image. In order to automate password changes we are trying to use the API. We have “pro” licenses.

Our script is PHP based, currently we succeed in fetching resources and users but when we try to send data through POST or PUT we get an internal server error. I know it’s not an authentication problem because we had a 403 response before, we managed to fix this.

We are unable to get more information. Is there a way to get a better error message than “Internal server error”.

Hi @bram.goffings Welcome to the forum!

With a 500 error, if it’s related to the passbolt app there should be errors in the error log. Check /var/log/passbolt or even php logs - hard to know for sure what is causing the error.

Known errors will be 400 with a description.

Is the script you are running in the same docker container with passbolt? Or is it from another container/machine?

I am assuming you have reviewed the API endpoint flow guide but if not, here it is Passbolt Help | Passbolt API Documentation and https://api-reference.passbolt.com

Can you provide an example call with PHP and which endpoint? We could help review it for any missing or incorrect parts.

The docker passbolt logs are mapping to stderror channels.
When looking at those logs (with docker logs) we only see the 500 error code requests but not the reason for the fatal error.

I can share you a test project but not in this forum.
The main test script I can share is pasted below this comment.

We tried everything written in the API documentation.

<?php

require(__DIR__ . '/lib/gpg_auth.php');
require(__DIR__ . '/vendor/autoload.php');

if (!extension_loaded('gnupg')) {
    trigger_error('You must enable the gnupg extension.', E_USER_ERROR);
}
if (!extension_loaded('curl')) {
    trigger_error('You must enable the curl extension.', E_USER_ERROR);
}

// Get config.
$config = require(__DIR__ . '/config/config.php');
$gpgAuth = new GpgAuth($config['server_url'], $config['private_key_path'], $config['private_key_passphrase']);

// Login in passbolt. This step will return a cookie that can be used in other guzzle calls.
$gpgAuth->login();

$client = new \GuzzleHttp\Client([
  'base_uri' => $config['server_url'],
  'timeout'  => 2.0,
]);
$request_options = [
  'headers' => ['Cookie' => $gpgAuth->getCookie()]
];

// Fetch info about the user
$response = $client->request('GET', '/users/me.json', $request_options);
$response_body = json_decode($response->getBody()->getContents(), TRUE);

$new_secrets[] = (object) [
  'data' => $gpgAuth->encrypt('Test password!', $response_body['body']['gpgkey']['fingerprint']),
];

$payload = [
  'name' => 'TEST',
  'secrets' => $new_secrets
];

// We had to alter the gpgauth class from the original documentation as it didn't work with post and put requests.
// We think the CSRF handling changed, but the adapted code below allowed us to pass the CSRF checks
preg_match_all('/csrfToken=([^;]*);/mi', $response->getHeader('Set-Cookie')[0], $matches);
$request_options = [
  'headers' => [
    'Cookie' => $gpgAuth->getCookie($matches[0][0]),
    'X-CSRF-Token' => $matches[1][0],
  ],
  'json' => json_encode($payload)
];

// Insert the password
// @todo fix 500 fatal error
$client->request('POST', '/resources.json', $request_options);

And here is our adapted GpgAuth class

<?php
/**
 * Passbolt ~ Open source password manager for teams
 * Copyright (c) Passbolt SA (https://www.passbolt.com)
 *
 * Licensed under GNU Affero General Public License version 3 of the or any later version.
 * For full copyright and license information, please see the LICENSE.txt
 * Redistributions of files must retain the above copyright notice.
 *
 * @copyright     Copyright (c) Passbolt SA (https://www.passbolt.com)
 * @license       https://opensource.org/licenses/AGPL-3.0 AGPL License
 * @link          https://www.passbolt.com Passbolt(tm)
 */

class GpgAuth
{

    private $serverUrl;

    private $privateKey = [
        'keydata' => '',
        'info'    => [],
        'passphrase' => null,
    ];

    private $serverKey = [
        'keydata' => '',
        'info'    => []
    ];

    private $gpg;

    public $sessionId = null;

    public $csrfToken = null;


    public function __construct($serverUrl, $privateKeyPath, $privateKeyPassphrase = null)
    {
        $this->privateKey['keydata'] = file_get_contents($privateKeyPath);
        $this->privateKey['passphrase'] = $privateKeyPassphrase;
        $this->serverUrl             = $serverUrl;
        $this->gpg                   = new \gnupg();
        $this->gpg->seterrormode(\gnupg::ERROR_EXCEPTION);
        $this->sessionId            = null;
        $this->csrfToken            = null;
    }

    public function generateToken()
    {
        return 'gpgauthv1.3.0|36|' . $this->uuid() . '|gpgauthv1.3.0';
    }

    function uuid()
    {
        $data    = random_bytes(16);
        $data[6] = chr(ord($data[6]) & 0x0f | 0x40);
        $data[8] = chr(ord($data[8]) & 0x3f | 0x80);

        return vsprintf('%s%s-%s-%s-%s-%s%s%s', str_split(bin2hex($data), 4));
    }

    private function _curlPost($url, $postParams)
    {
        $c = curl_init();
        curl_setopt($c, CURLOPT_URL, $url);
        curl_setopt($c, CURLOPT_POST, 1);
        curl_setopt($c, CURLOPT_POSTFIELDS, http_build_query($postParams));
        curl_setopt($c, CURLOPT_HEADER, 1);
        curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
        $response   = curl_exec($c);
        $headerSize = curl_getinfo($c, CURLINFO_HEADER_SIZE);
        $header     = substr($response, 0, $headerSize);
        curl_close($c);

        return $header;
    }

    public function initKeyring()
    {
        $this->privateKey['info'] = $this->gpg->import($this->privateKey['keydata']);
    }

    public function getServerKey()
    {
        $c = curl_init();
        curl_setopt($c, CURLOPT_URL, $this->serverUrl . '/auth/verify.json');
        curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
        $response = curl_exec($c);
        curl_close($c);

        $responseJson    = json_decode($response, true);
        $this->serverKey = $responseJson['body'];

        $this->serverKey['info'] = $this->gpg->import($this->serverKey['keydata']);

        return $this->serverKey;
    }

    private function _getHeader($header, $name)
    {
        $headerLines = explode("\n", $header);
        $res         = [];

        foreach ($headerLines as $headerLine) {
            if (strpos(strtolower($headerLine), strtolower($name) . ':') !== false) {
                $res = array_merge($res, explode(';', trim(explode(":", $headerLine)[1])));
            }
        }

        if ($res) {
            return implode(';', $res);
        }

        return false;
    }

    public function stage0()
    {
        $token = $this->generateToken();
        $this->getServerKey();
        $this->gpg->addencryptkey($this->serverKey['info']['fingerprint']);
        $encryptedToken = $this->gpg->encrypt($token);

        $post = [
            'data' => [
                'gpg_auth' => [
                    'keyid'               => $this->privateKey['info']['fingerprint'],
                    'server_verify_token' => $encryptedToken,
                ],
            ],
        ];


        $header         = $this->_curlPost($this->serverUrl . '/auth/verify.json', $post);
        $retrievedToken = $this->_getHeader($header, 'X-GPGAuth-Verify-Response');

        if ($retrievedToken !== $token) {
            throw new Exception('Stage 0: Tokens mismatch');
        }

        return $token;
    }

    public function stage1A()
    {
        $post = [
            'data' => [
                'gpg_auth' => [
                    'keyid' => $this->privateKey['info']['fingerprint'],
                ],
            ],
        ];

        $header         = $this->_curlPost($this->serverUrl . '/auth/login.json', $post);
        $encryptedToken = $this->_getHeader($header, 'X-GPGAuth-User-Auth-Token');
        $encryptedToken = (stripslashes(urldecode($encryptedToken)));

        $this->gpg->adddecryptkey($this->privateKey['info']['fingerprint'], $this->privateKey['passphrase']);
        $verify = $this->gpg->decryptverify($encryptedToken, $decryptedToken);

        if ($verify[0]['fingerprint'] !== $this->serverKey['info']['fingerprint']) {
            throw new Exception('Stage 1A: Signature mismatch');
        }

        return $decryptedToken;
    }

    public function stage1B($token)
    {
        $post = [
            'data' => [
                'gpg_auth' => [
                    'keyid'             => $this->privateKey['info']['fingerprint'],
                    'user_token_result' => $token,
                ],
            ],
        ];

        $header = $this->_curlPost($this->serverUrl . '/auth/login.json', $post);

        $status        = $this->_getHeader($header, 'X-GPGAuth-Progress');
        $authenticated = $this->_getHeader($header, 'X-GPGAuth-Authenticated');

        $authenticated = ($status === 'complete' && $authenticated === 'true');
        if ( ! $authenticated) {
            throw new Exception('Stage 1B: Authentication failure');
        }

        $cookieHeader = $this->_getHeader($header, 'Set-Cookie');

        preg_match_all('/passbolt_session=([^;]*);/mi', $cookieHeader, $matches);
        $this->sessionId = $matches[1][0];
    }

    public function getCookie(bool $csrfToken = false)
    {
        $cookie = "passbolt_session={$this->sessionId}; path=/; HttpOnly;";
        if ($csrfToken) {
            $cookie .= " $csrfToken";
        }

        return $cookie;
    }

    public function login()
    {
        $this->initKeyring();
        $this->stage0();
        $token  = $this->stage1A();
        $this->stage1B($token);
    }

    public function encrypt($token, $key) {
      $this->gpg->addencryptkey($this->serverKey['info']['fingerprint']);
      return $this->gpg->encrypt($token);
    }
}

It is complicated to say without any logs as @garrett mentioned. You could set DEBUG env variable to true but that would require you to install all the development libraries.

Other option would be to spin a passbolt dev container. You would need to clone the api and also install the dev libraries: passbolt_docker/README.md at master · passbolt/passbolt_docker · GitHub

You could set DEBUG env variable to true but that would require you to install all the development libraries.

Is there any documentation how to do this in the docker image?

In my opinion your best option is to follow the dev container docs I linked on my previous post.

I think curl has a verbose output option too?

Yes, that is also true but I think it would be better to get server logs here. But at this point any log would be useful, yes :smiley:

1 Like

Verbose is not enough, will try the debug option

  • Found bundle for host passbolt.DOMAIN_NAME: 0x5559aaca3f90 [serially]
  • Re-using existing connection! (#0) with host passbolt.DOMAIN_NAME
  • Connected to passbolt.DOMAIN_NAME (IP) port 443 (#0) > POST /resources.json HTTP/1.1 Host: passbolt.DOMAIN_NAME User-Agent: GuzzleHttp/7 Content-Type: application/json Cookie: passbolt_session=nj8dvmd0mtiiqpv4cebaa4946h; path=/; HttpOnly; 1 X-CSRF-Token: 10052780da10c53c640fd7e5b2241c5378813629c38f97d0d5cab26eb1bdcb458a84aa8927bf33006c1873f58b0919023b70e881eeab33e1a29290585ed2edaac Content-Length: 2
  • upload completely sent off: 2 out of 2 bytes
  • Mark bundle as not supporting multiuse < HTTP/1.1 500 Internal Server Error < Server: nginx/1.18.0 < Date: Thu, 13 Apr 2023 13:30:11 GMT < Content-Type: application/json < Transfer-Encoding: chunked < Connection: keep-alive < Keep-Alive: timeout=5 < Expires: Thu, 19 Nov 1981 08:52:00 GMT < Cache-Control: no-store, no-cache, must-revalidate < Pragma: no-cache < strict-transport-security: max-age=31536000; includeSubDomains < Content-Security-Policy: default-src ‘self’; script-src ‘self’; style-src ‘self’ ‘unsafe-inline’; img-src ‘self’ data:; frame-src ‘self’ https://*.duosecurity.com; <
  • Connection #0 to host passbolt.DOMAIN_NAME left intact

I didn’t review your changes in the source, but are you piggybacking two requests in one?

No 2 seperate requests.
And the second notice about “mark bundle…” is not a problem, I checked this on the interwebz :slight_smile:

@diego I tried to follow the documentation but I get stuck on step 5, there is no .env.example file and I tried to search PATH_TO_PASSBOLT_API across the entire passbolt_api and docker project but I couldn’t find the constant

  1. Copy the .env.example file into .env and replace the PATH_TO_PASSBOLT_API variable with the path to the passbolt_api repository on your machine

I think it’s calling out the 1.1 standard instead of http2. In NGINX the configuration for websocket connections, multiplexing, etc would be:
listen 443 ssl http2; (default is 1.1)

But I am not aware of passbolt’s need for it, and the setup leaves it at 1.1. Not sure why it would throw this error for you if you are not attempting an additional request on an open connection.

.env.example is part of the docker repository: passbolt_docker/.env.example at master · passbolt/passbolt_docker · GitHub

The PATH_TO_PASSBOLT_API is the path on your host where you have clone the passbolt_api repository.

So the idea is you git clone passbolt_api repository somewhere in your host machine and you also clone passbolt_docker in your host machine too.

Then you build a dev container following the instructions on the readme that mounts the code of the passbolt_api that you have cloned on your host.

Hope this helps a bit

The documentation is not 100% correct.

cd passbolt_docker
docker-compose -f dev/docker-compose-dev.yml up -d

There is no docker compose .yml file, there is a docker-composer-dev.yaml file.

cd passbolt_docker
docker-compose -f dev/docker-compose-ce.yaml exec passbolt /bin/bash -c \
  'su -m -c "/var/www/passbolt/bin/cake passbolt register_user -u myuser@passbolt.local \
   -f name  -l lastname  -r admin" -s /bin/sh www-data'

After this step the documentation states, that I have to copy the output in the browser. But the command doesn’t generate any output.

When visiting passbolt.local and using myuser@passbolt.local I get this error

# Access to this service requires an invitation.

This email is not associated with any approved users on this domain. Please contact your administrator to request an invitation link.

@diego

In the end we noticed that the documentation used the docker-compose-ce.yaml and not the docker-compose-dev.yaml

cd passbolt_docker
docker-compose -f dev/docker-compose-ce.yaml exec passbolt /bin/bash -c \
  'su -m -c "/var/www/passbolt/bin/cake passbolt register_user -u myuser@passbolt.local \
   -f name  -l lastname  -r admin" -s /bin/sh www-data'

So in the end I managed to setup the instance. But next problem… When I use https://passbolt.local I can visit the website (after ignoring ssl problems) but when I try to test the API I get stuck.

When I visit http://passbolt.local/auth/verify.json in the browser I get a clean json response but when we use curl "curl -I https://passbolt.local/auth/verify.json -k -H ‘Content-Type: application/json’
" you get a 404 response. (added the -k to bypass ssl errors and tried the content type header as suggested in curl get json returns 404 - Stack Overflow

Using the API has been a very bumpy ride because of these 500 errors and incorrect/missing docs.