Docker container wont restart after system crach

I have run into a problem where a previously working install of Passbolt in a Docker container on a Synology NAS has stopped working after an unexpected shutdown (due to a power cut) of the server.

Everything was working fine prior to the power cut. I was able to access Passbolt on the desired subdomain and domain. The SSL and port forwarding in the router were all working okay.

Now the container starts but report it is ‘unhealthy’. The last output from the container is “Failed to connect to localhost port 8080 : Connection refused”. I have checked that port 8080 is not being used by another process.

I would like to try to resolve this without a full re-install as there are some users who have stored passwords that I would like to retain if possible.

I have tried re-starting the container serveral times, always with the same result. I have tried stopping all other containers and starting Passbolt on its own with no success (and the same error).

The original install was done with this guide :

I suspect the problem could be reproduced by installing as above and doing a hard-shutdown (power cut) to the server.

I have since bought a UPS. : -)

Links in this post have been removed as new users are not allowed more than 2 links in a post.

Thanks in advance to anyone who can help.

Checklist
I have read intro post : link removed
I have read the tutorials, help and searched for similar issues
I provide relevant information about my server (component names and versions, etc.)
I provide a copy of my logs and healthcheck
I describe the steps I have taken to trouble shoot the problem
I describe the steps on how to reproduce the issue

Server : Synology NAS, DSM 7.1.1-42962 Update 4

docker compose details:

version: "3.9"
services:
  db:
    image: mariadb:jammy
    container_name: Passbolt-DB
    hostname: passbolt-db
    mem_limit: 512m
    mem_reservation: 128m
    cpu_shares: 768
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD-SHELL", "mysqladmin ping -P 3306 -prootpass | grep 'mysqld is alive' || exit 1"]
    volumes:
      - /volume1/docker/passbolt/db:/var/lib/mysql:rw
    environment:
      TZ: Australia/Melbourne
      MYSQL_ROOT_PASSWORD: rootpass
      MYSQL_DATABASE: passbolt
      MYSQL_USER: passboltuser
      MYSQL_PASSWORD: passboltpass
    restart: on-failure:5

  passbolt:
    image: passbolt/passbolt:latest-ce-non-root
    command:
      - /bin/bash
      - -c
      - /usr/bin/wait-for.sh -t 0 db:3306 -- /docker-entrypoint.sh
    container_name: Passbolt
    hostname: passbolt
    mem_limit: 1g
    cpu_shares: 512
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: curl -f *removed* || exit 1
    ports:
      - 9475:8080
    volumes:
      - /volume1/docker/passbolt/gpg:/etc/passbolt/gpg:rw
      - /volume1/docker/passbolt/jwt:/etc/passbolt/jwt:rw
    environment:
      DEBUG: false
      APP_FULL_BASE_URL: *removed*
      DATASOURCES_DEFAULT_HOST: passbolt-db
      DATASOURCES_DEFAULT_USERNAME: passboltuser
      DATASOURCES_DEFAULT_PASSWORD: passboltpass
      DATASOURCES_DEFAULT_DATABASE: passbolt
      PASSBOLT_REGISTRATION_PUBLIC: true
      EMAIL_DEFAULT_FROM: *removed*
      EMAIL_TRANSPORT_DEFAULT_HOST: *removed*
      EMAIL_TRANSPORT_DEFAULT_PORT: 25
      EMAIL_TRANSPORT_DEFAULT_USERNAME: *removed*
      EMAIL_TRANSPORT_DEFAULT_PASSWORD: *removed*
      EMAIL_TRANSPORT_DEFAULT_TLS: true
    restart: on-failure:5
    depends_on:
      db:
        condition: service_started

Healthcheck :

      ____                  __          ____  
    / __ \____  _____ ____/ /_  ____  / / /_ 
   / /_/ / __ `/ ___/ ___/ __ \/ __ \/ / __/ 
  / ____/ /_/ (__  |__  ) /_/ / /_/ / / /    
 /_/    \__,_/____/____/_.___/\____/_/\__/   

 Open source password manager for teams
-------------------------------------------------------------------------------
 Healthcheck shell         
-------------------------------------------------------------------------------

 Environment

 [PASS] PHP version 7.4.33.
 [PASS] PCRE compiled with unicode support.
 [PASS] The temporary directory and its content are writable and not executable.
 [PASS] The logs directory and its content are writable.
 [PASS] GD or Imagick extension is installed.
 [PASS] Intl extension is installed.
 [PASS] Mbstring extension is installed.

 Config files

 [PASS] The application config file is present
 [WARN] The passbolt config file is missing in /etc/passbolt/
 [HELP] Copy /etc/passbolt/passbolt.default.php to /etc/passbolt/passbolt.php
 [HELP] The passbolt config file is not required if passbolt is configured with environment variables

 Core config

 [PASS] Debug mode is off.
 [PASS] Cache is working.
 [PASS] Unique value set for security.salt
 [PASS] Full base url is set to *link removed*
 [PASS] App.fullBaseUrl validation OK.
 [FAIL] Could not reach the /healthcheck/status with the url specified in App.fullBaseUrl
 [HELP] Check that the domain name is correct in /etc/passbolt/passbolt.php
 [HELP] Check the network settings

 SSL Certificate

 [FAIL] SSL peer certificate does not validate
 [FAIL] Hostname does not match when validating certificates.
 [WARN] Using a self-signed certificate
 [HELP] Check *link removed*
 [HELP] cURL Error (6) Could not resolve host: *link removed - but it did work with this link*

 Database

 [FAIL] The application is not able to connect to the database.
 [HELP] Double check the host, database name, username and password in /etc/passbolt/passbolt.php.
 [HELP] Make sure the database exists and is accessible for the given database user.
 [FAIL] No table found
 [HELP] Run the install script to install the database tables
 [HELP] sudo su -s /bin/bash -c "/usr/share/php/passbolt/bin/cake passbolt install" www-data
 [FAIL] No default content found
 [HELP] Run the install script to set the default content such as roles and permission types
 [HELP] sudo su -s /bin/bash -c "/usr/share/php/passbolt/bin/cake passbolt install" www-data
 [FAIL] The database schema is not up to date.
 [HELP] Run the migration scripts:
 [HELP] sudo su -s /bin/bash -c "/usr/share/php/passbolt/bin/cake migrations migrate --no-lock" www-data
 [HELP] See. *link removed*

 GPG Configuration

 [PASS] PHP GPG Module is installed and loaded.
 [PASS] The environment variable GNUPGHOME is set to /var/lib/passbolt/.gnupg.
 [PASS] The directory /var/lib/passbolt/.gnupg containing the keyring is writable by the webserver user.
 [FAIL] The server OpenPGP key is not set
 [HELP] Create a key, export it and add the fingerprint to /etc/passbolt/passbolt.php
 [HELP] See. *link removed*
 [PASS] The public key file is defined in /etc/passbolt/passbolt.php and readable.
 [PASS] The private key file is defined in /etc/passbolt/passbolt.php and readable.
 [FAIL] The server key fingerprint doesn't match the one defined in /etc/passbolt/passbolt.php.
 [HELP] Double check the key fingerprint, example: 
 [HELP] sudo su -s /bin/bash -c "gpg --list-keys --fingerprint --home /var/lib/passbolt/.gnupg" www-data | grep -i -B 2 'SERVER_KEY_EMAIL'
 [HELP] SERVER_KEY_EMAIL: The email you used when you generated the server key.
 [HELP] See. *link removed*
 [FAIL] The server public key defined in the /etc/passbolt/passbolt.php (or environment variables) is not in the keyring
 [HELP] Import the private server key in the keyring of the webserver user.
 [HELP] you can try:
 [HELP] sudo su -s /bin/bash -c "gpg --home /var/lib/passbolt/.gnupg --import /etc/passbolt/gpg/serverkey_private.asc" www-data
 [FAIL] The server key does not have a valid email id.
 [HELP] Edit or generate another key with a valid email id.

 Application configuration

 [FAIL] Could not connect to passbolt repository to check versions It is not possible check if your version is up to date.
 [HELP] Check the network configuration to allow this script to check for updates.
 [PASS] Passbolt is configured to force SSL use.
 [PASS] App.fullBaseUrl is set to HTTPS.
 [PASS] Selenium API endpoints are disabled.
 [PASS] Search engine robots are told not to index content.
 [WARN] Registration is open to everyone.
 [HELP] Make sure this instance is not publicly available on the internet.
 [HELP] Or set passbolt.registration.public to false in /etc/passbolt/passbolt.php.
 [WARN] Host availability checking is disabled.
 [HELP] Make sure this instance is not publicly available on the internet.
 [HELP] Or set the PASSBOLT_EMAIL_VALIDATE_MX environment variable to true.
 [HELP] Or set passbolt.email.validate.mx to true in /etc/passbolt/passbolt.php.
 [PASS] Serving the compiled version of the javascript app.
 [PASS] All email notifications will be sent.

 JWT Authentication

 [PASS] The JWT Authentication plugin is enabled
 [PASS] The /etc/passbolt/jwt/ directory is not writable.
 [PASS] A valid JWT key pair was found

 SMTP Settings

 [PASS] The SMTP Settings plugin is enabled.
 [PASS] SMTP Settings coherent. You may send a test email to validate them.
 [WARN] The SMTP Settings source is: env variables.
 [HELP] It is recommended to set the SMTP Settings in the database through the administration section.
 [WARN] The SMTP Settings plugin endpoints are enabled.
 [HELP] It is recommended to disable the plugin endpoints.
 [HELP] Set the PASSBOLT_SECURITY_SMTP_SETTINGS_ENDPOINTS_DISABLED environment variable to true.
 [HELP] Or set passbolt.security.smtpSettings.endpointsDisabled to true in /etc/passbolt/passbolt.php.

 [FAIL] 12 error(s) found. Hang in there!

Since the healthcheck is showing that it fails to connect to the database, can you confirm that the db container is up and running and that you can connect to it?

@mariushosting have you had any hard shutdowns and issues bringing it back online?

1 Like

Thanks so much for your thoughts and suggestions. Greatly appreciated.

The Passbolt-DB container is reported as healthy in Portainer. The failure count is 0. The last output from this container is:

WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. mysqld is alive

I don’t know how to test if I can connect to the DB container.

The log from the DB container is:

2023-04-08 17:16:17+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.3+maria~ubu2204 started.
2023-04-08 17:16:23+10:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-04-08 17:16:23+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.3+maria~ubu2204 started.
2023-04-08 17:16:23+10:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-04-08 17:16:23 0 [Note] Starting MariaDB 10.10.3-MariaDB-1:10.10.3+maria~ubu2204 source revision cc8b9bcee3ce88bc52147948f96765cd5009b88a as process 1
2023-04-08 17:16:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-04-08 17:16:24 0 [Note] InnoDB: Number of transaction pools: 1
2023-04-08 17:16:24 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-04-08 17:16:24 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
2023-04-08 17:16:24 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2023-04-08 17:16:24 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-04-08 17:16:24 0 [Note] InnoDB: Completed initialization of buffer pool
2023-04-08 17:16:24 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-04-08 17:16:25 0 [Note] InnoDB: 128 rollback segments are active.
2023-04-08 17:16:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-04-08 17:16:25 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-04-08 17:16:25 0 [Note] InnoDB: log sequence number 6784085; transaction id 8658
2023-04-08 17:16:25 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-04-08 17:16:25 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-04-08 17:16:25 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2023-04-08 17:16:26 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-04-08 17:16:26 0 [Note] Server socket created on IP: '::'.
2023-04-08 17:16:26 0 [Note] mariadbd: ready for connections.
Version: '10.10.3-MariaDB-1:10.10.3+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2023-04-08 17:16:28 0 [Note] InnoDB: Buffer pool(s) load completed at 230408 17:16:28
2023-04-13  9:58:44 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
2023-04-13  9:58:44 0 [Note] InnoDB: FTS optimize thread exiting.
2023-04-13  9:58:44 0 [Note] InnoDB: Starting shutdown...
2023-04-13  9:58:44 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2023-04-13  9:58:44 0 [Note] InnoDB: Buffer pool(s) dump completed at 230413  9:58:44
2023-04-13  9:58:44 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2023-04-13  9:58:44 0 [Note] InnoDB: Shutdown completed; log sequence number 6784085; transaction id 8659
2023-04-13  9:58:45 0 [Note] mariadbd: Shutdown complete
2023-04-13 10:18:55+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.3+maria~ubu2204 started.
2023-04-13 10:19:05+10:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-04-13 10:19:05+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.3+maria~ubu2204 started.
2023-04-13 10:19:06+10:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-04-13 10:19:06 0 [Note] Starting MariaDB 10.10.3-MariaDB-1:10.10.3+maria~ubu2204 source revision cc8b9bcee3ce88bc52147948f96765cd5009b88a as process 1
2023-04-13 10:19:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-04-13 10:19:06 0 [Note] InnoDB: Number of transaction pools: 1
2023-04-13 10:19:06 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-04-13 10:19:06 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
2023-04-13 10:19:06 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2023-04-13 10:19:06 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-04-13 10:19:06 0 [Note] InnoDB: Completed initialization of buffer pool
2023-04-13 10:19:07 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-04-13 10:19:14 0 [Note] InnoDB: 128 rollback segments are active.
2023-04-13 10:19:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-04-13 10:19:14 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-04-13 10:19:14 0 [Note] InnoDB: log sequence number 6784085; transaction id 8658
2023-04-13 10:19:14 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-04-13 10:19:14 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-04-13 10:19:14 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2023-04-13 10:19:15 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-04-13 10:19:15 0 [Note] Server socket created on IP: '::'.
2023-04-13 10:19:17 0 [Note] InnoDB: Buffer pool(s) load completed at 230413 10:19:17
2023-04-13 10:19:17 0 [Note] mariadbd: ready for connections.
Version: '10.10.3-MariaDB-1:10.10.3+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

For testing the connection generally what I do is exec into the db container and then mysql -u <Passbolt DB user> -p you’ll then get the password prompt. Both the username and password are specified in your docker-compose file

It looks like I can connect to the database from within the Passbolt-DB container.

www-data@passbolt-db:/$ mysql -u *modified* -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 22543
Server version: 10.10.3-MariaDB-1:10.10.3+maria~ubu2204 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

If I try to connect to the database from within the Passbolt container then mysql is not recognised as a command.

This is correct because no db service is running in this one.

Since you are able to access the db, can you confirm tables are installed?

SHOW TABLES;

This warning is from mysql and it is saying it doesn’t want to decide between protocols (both local port and socket options are running). To resolve this, wherever there is a mysql command on the commandline, it wants something like:

mysql -u username -p -h hostname --protocol {socket, tcp}

MariaDB [passbolt]> SHOW TABLES;
+-----------------------+
| Tables_in_passbolt    |
+-----------------------+
| account_settings      |
| action_logs           |
| actions               |
| authentication_tokens |
| avatars               |
| comments              |
| email_queue           |
| entities_history      |
| favorites             |
| gpgkeys               |
| groups                |
| groups_users          |
| organization_settings |
| permissions           |
| permissions_history   |
| phinxlog              |
| profiles              |
| resource_types        |
| resources             |
| roles                 |
| secret_accesses       |
| secrets               |
| secrets_history       |
| transfers             |
| user_agents           |
| users                 |
+-----------------------+
26 rows in set (0.000 sec)

The db container seems to be working alright. Can you describe the problem again from a user point of view? What are you actually seeing when you attempt to access passbolt?

It seems like there is no path for https in your yaml but I see the healthcheck is reporting using ssl? This must be because you are handling ssl in the router in front of the app? Passbolt doesn’t need ssl enabled if this is the case.

Which container is reporting this?

The “Failed to connect to localhost port 8080 : Connection refused” is reported by the Passbolt container.

The details of the problem are:

  • When browsing to https://passbolt.mydomain.com.au I get a page not found response that is generated by the Synology NAS webserver. It is secure (the little padlock is locked).
  • There are two Docker containers: Passbolt, and Passbolt-DB. In Portainer, the Passbolt container is marked as unhealthy.
  • The final output from the Passbolt container (reported via the Portainer user interface) is Failed to connect to localhost port 8080 : Connection refused.
  • The final part of the stderr log from the Passbolt container is wait-for.sh: waiting for db:3306 without a timeout
  • The final output from the Passbolt-DB container (reported via the Portainer user interface) is WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. mysqld is alive
  • The final part of the Passbolt-DB container log is mariadbd: ready for connections followed by Version: '10.10.3-MariaDB-1:10.10.3+maria~ubu2204' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution.
  • The android app, which has been set up to access http://passbolt.mydomain.com.au crashes on launch. I appreciate that how the app responds to failed connections is another matter.

It looks to me like there is some problem with the Passbolt container accessing and using the database from the Passbolt-DB container. I don’t really know how to test that connection.

I would like to add that I greatly appreciate the time that has been put into reading and responding to my posts.

Have you tried rebuilding the passbolt container? The web server seems to not be running on 8080 in the passbolt container. I would try that next. Thanks for the comprehensive info, it’s very helpful.

No luck restarting the container:

Failed to connect to localhost port 8080: Connection refused

Re-pulled and re-deployed the stack containing both containers. The problem persists (as above).

I am wondering if there is something going on here that is not a problem with Passbolt as such, but more to do with my Docker set-up / environment / or a conflict between Docker containers. I really appreciate the help I have been provided here.

Did anything from here help? Blank Page - New Install on Synology Nas Docker

I want to provide a follow-up and conclusion to this issue. I’m pleased to say it has been resolved, and I am back up and running without any loss of data. I’m less pleased to have to admit that I don’t know how. I don’t know what caused the problem, or what fixed it. Here’s a few things I tried.

  • I had SABnzbd installed from a Community Package in Package Centre in the Synology NAS. I learned that this is hard-coded to use port 8080 and that this cannot be changed. I removed this package. This didn’t seem to help. I didn’t think it would help because the port 8080 that I was trying to use was inside the Passbolt docker container, so any changes to port 8080 outside the container, to my thinking, would be irrelevant. I restarted after uninstalling this package, and the problem persisted.
  • I updated the Synology NAS software to 7.1.1-42962 Update 5. This required a restart. After this restart the Passbolt container started and ran without generating any errors.
  • There were several re-starts in the above two steps. I may just have been that multiple re-starts were required.
  • When Passbolt was up and running, it was not working properly on Google Chrome. Other browsers worked okay. After an overnight wait, Google started working again. I don’t think this is a Passbolt issue per-se, but something to do with Google security checks. My site pb.mydomain.com.au had been (incorrectly) flagged as a phishing site by google. This seems to have sorted itself out with no intervention from my part. It may be been related to Google picking up on the multiple error pages that had been generated while Passbolt was not working.

Thanks to all those who have contributed to this issue and tried to help me.

1 Like