Avatar storage problems (Ubuntu 20, Passbolt 3.1 CE)

[jab4]

Dear all,

I’m unable to upload avatars.

This may be related to #3320 (Internal error occurs when uploading avatars), but the answers provided there are not that in-depth to help me track down the problem.

[ ] I provide a copy of my logs and healthcheck

2021-05-18 01:48:23 Error: The file "Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png" was not found.
2021-05-18 01:48:23 Error: The file "Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png" was not found.
2021-05-18 01:48:23 Error: [Burzum\FileStorage\Storage\StorageException] The file "Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png" was not found. (/usr/share/php/passbolt/vendor/burzum/cakephp-file-storage/src/Storage/Listener/AbstractListener.php:366)
Caused by: [Burzum\FileStorage\Storage\StorageException] The file "Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png" was not found. (/usr/share/php/passbolt/vendor/burzum/cakephp-file-storage/src/Storage/Listener/AbstractListener.php:395)
Caused by: [Gaufrette\Exception\FileNotFound] The file "Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png" was not found. (/usr/share/php/passbolt/vendor/knplabs/gaufrette/src/Gaufrette/Filesystem.php:292)
Request URL: /users/e582cebb-ad96-4f8e-b33d-444a740c41ab.json?api-version=v2

2021-05-18 01:49:51 Error: [App\Error\Exception\ValidationException] Could not validate user data. (/usr/share/php/passbolt/src/Controller/Users/UsersEditController.php:59)
Request URL: /users/e582cebb-ad96-4f8e-b33d-444a740c41ab.json?api-version=v2

Healthcheck is good:

# sudo -H -u www-data bash -c "/usr/share/php/passbolt/bin/cake passbolt healthcheck --verbose"

     ____                  __          ____  
    / __ \____  _____ ____/ /_  ____  / / /_ 
   / /_/ / __ `/ ___/ ___/ __ \/ __ \/ / __/ 
  / ____/ /_/ (__  |__  ) /_/ / /_/ / / /    
 /_/    \__,_/____/____/_.___/\____/_/\__/   

 Open source password manager for teams
---------------------------------------------------------------
 Healthcheck shell       
---------------------------------------------------------------

 Environment

 [PASS] PHP version 7.4.3.
 [PASS] PCRE compiled with unicode support.
 [PASS] The temporary directory and its content are writable.
 [PASS] The public image directory and its content are writable.
 [PASS] The logs directory and its content are writable.
 [PASS] GD or Imagick extension is installed.
 [PASS] Intl extension is installed.
 [PASS] Mbstring extension is installed.

 Config files

 [PASS] The application config file is present
 [PASS] The passbolt config file is present

 Core config

 [PASS] Debug mode is off.
 [PASS] Cache is working.
 [PASS] Unique value set for security.salt
 [PASS] Full base url is set to https://pass.xxxxxxxxx
 [PASS] App.fullBaseUrl validation OK.
 [PASS] /healthcheck/status is reachable.

 SSL Certificate

 [PASS] SSL peer certificate validates
 [PASS] Hostname is matching in SSL certificate.
 [PASS] Not using a self-signed certificate

 Database

 [PASS] The application is able to connect to the database
 [PASS] 33 tables found
 [PASS] Some default content is present
 [PASS] The database schema up to date.

 GPG Configuration

 [PASS] PHP GPG Module is installed and loaded.
 [PASS] The environment variable GNUPGHOME is set to /etc/passbolt/.gnupg.
 [PASS] The directory /etc/passbolt/.gnupg containing the keyring is writable by the webserver user.
 [PASS] The server gpg key is not the default one
 [PASS] The public key file is defined in config/passbolt.php and readable.
 [PASS] The private key file is defined in config/passbolt.php and readable.
 [PASS] The server key fingerprint matches the one defined in config/passbolt.php.
 [PASS] The server public key defined in the config/passbolt.php (or environment variables) is in the keyring.
 [PASS] There is a valid email id defined for the server key.
 [PASS] The public key can be used to encrypt a message.
 [PASS] The private key can be used to sign a message.
 [PASS] The public and private keys can be used to encrypt and sign a message.
 [PASS] The private key can be used to decrypt a message.
 [PASS] The private key can be used to decrypt and verify a message.
 [PASS] The public key can be used to verify a signature.

 Application configuration

 [PASS] Using latest passbolt version (3.1.0).
 [PASS] Passbolt is configured to force SSL use.
 [PASS] App.fullBaseUrl is set to HTTPS.
 [PASS] Selenium API endpoints are disabled.
 [PASS] Search engine robots are told not to index content.
 [PASS] Registration is closed, only administrators can add users.
 [PASS] Serving the compiled version of the javascript app
 [PASS] All email notifications will be sent.

 No error found. Nice one sparky!

[ ] I describe the steps I have taken to trouble shoot the problem

Everything works fine, except saving avatars.

My public image directory is: /usr/share/php/passbolt/webroot/img/public/.

My file storage table looks like this (some fields truncated) for readability:

MariaDB [passbolt]> select * from file_storage;
+-----+----------+----+--------+----------+----------+------------+-----------+------+---------------------------------------------------------------------------------------+---------+---------------------+---------------------+
| id  | user_id  | fo | model  | filename | filesize | mime_type  | extension | hash | path                                                                                  | adapter | created             | modified            |
+-----+----------+----+--------+----------+----------+------------+-----------+------+---------------------------------------------------------------------------------------+---------+---------------------+---------------------+
| 60e | 7706885f | a0 | Avatar | kisspng- |   498629 | image/png  | png       | c040 | Avatar/38/c0/aa/60eca304d7074a8f83360a9c9906da1f/60eca304d7074a8f83360a9c9906da1f.png | Local   | 2019-06-25 05:57:55 | 2019-06-25 05:57:55 |
| c04 | 5a1bd961 | 5a | Avatar | peace.jp |     9291 | image/jpeg | jpg       | c38e | Avatar/c7/fc/59/c04bbd737857496d8ab0a00adb29ae5c/c04bbd737857496d8ab0a00adb29ae5c.jpg | Local   | 2019-06-24 13:38:48 | 2019-06-24 13:38:48 |
| ea4 | e582cebb | 7b | Avatar | 580b585b |    34123 | image/png  | png       | 8e1b | Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png | Local   | 2020-07-10 07:35:13 | 2020-07-10 07:35:13 |
| fff | 5a1d3293 | 5a | Avatar | test.jpg |   183760 | image/jpeg | jpg       | 2a9d | Avatar/f3/9c/67/fff05c7575b2486681b2778d9ef2f608/fff05c7575b2486681b2778d9ef2f608.jpg | Local   | 2019-09-03 08:22:40 | 2019-09-03 08:22:40 |
+-----+----------+----+--------+----------+----------+------------+-----------+------+---------------------------------------------------------------------------------------+---------+---------------------+---------------------+
4 rows in set (0.001 sec)

It appears a directory or two has been created while uploading the avatar from my user profile page:

# l /usr/share/php/passbolt/webroot/img/public/Avatar/
total 32
drwxr-xr-x 8 www-data www-data 4096  2021-05-18  03:48:23.06  .
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:12:49.57  ..
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:20:38.41  22 
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:47:56.39  31
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:48:23.06  39
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:12:49.57  a9
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:48:07.61  e6
drwxr-xr-x 3 www-data www-data 4096  2021-05-18  03:20:51.54  f0
[root@passbolt:~] # tree /usr/share/php/passbolt/webroot/img/public/Avatar/
/usr/share/php/passbolt/webroot/img/public/Avatar/
├── 22
│   └── 76
│       └── e0
│           └── feac38c637e447fcaac9fc23109cc795
│               └── feac38c637e447fcaac9fc23109cc795.png
├── 31
│   └── 99
│       └── 57
│           └── 4c4c788599e44a298b947c00646fe56c
│               └── 4c4c788599e44a298b947c00646fe56c.png
├── 39
│   └── 72
│       └── 7e
│           └── c5423dc1da8f4ad69cdf3d62f11d7511
│               └── c5423dc1da8f4ad69cdf3d62f11d7511.png
├── a9
│   └── 8d
│       └── 44
│           └── 88cd247862be45e5bc043c5ab48843ed
│               └── 88cd247862be45e5bc043c5ab48843ed.png
├── e6
│   └── 95
│       └── 26
│           └── 053dad6c865d4a5cb73c31e4ec95a63d
│               └── 053dad6c865d4a5cb73c31e4ec95a63d.png
└── f0
    └── cc
        └── b3
            └── 788e3eb4574f4f4e9302671e7a9c9ed7
                └── 788e3eb4574f4f4e9302671e7a9c9ed7.png

24 directories, 6 files

…so my config setting can’t be that wrong: /etc/passbolt/file_storage.php

<?php
use Cake\Core\Configure;
use Burzum\FileStorage\Storage\StorageUtils;
use Burzum\FileStorage\Storage\StorageManager;

// File storage and images
Configure::write('ImageStorage.adapter', 'Local');
#Configure::write('ImageStorage.basePath', WWW_ROOT . 'img' . DS . 'public' . DS);
Configure::write('ImageStorage.basePath', '/usr/share/php/passbolt/webroot/img/public/');
Configure::write('ImageStorage.publicPath', 'img' . DS . 'public' . DS);
Configure::write('FileStorage', array(
    // Configure the `basePath` for the Local adapter, not needed when not using it
#    'basePath' => APP . 'FileStorage' . DS,
#    /passbolt/webroot/img/public/images/Avatar/65/90/13/5ab52135926c44ee8c384c6ac0a88a1b/5ab52135926c44ee8c384c6ac0a88a1b.jpg    
        'basePath' => '/usr/share/php/passbolt/webroot/img/public/',

…but still, these don’t make it into the database, and all error messages I get (apart from the frontend: An Internal Error Has Occurred) are the ones I posted from the log file above.

[ ] I describe the steps on how to reproduce the issue

No idea. I hoped to fix this issue by kinda doing a semi-fresh start. We started out in April 2019 with passbolt_pro_ubuntu_installer.sh, avatars worked then, but in the meantime we made several upgrades and some time in 2020, avatars stopped working.

I’m on Ubuntu 20.04. I’ve followed a mix of manual install instructions and the ones for the Ubuntu package.

Any pointers are appreciated.

[garrett]

Hi @jab4 Is it possible your avatar file is under the /var/www/passbolt path and it didn’t get transferred over during the upgrade?

[jab4]

Hey @garrett, thanks for your reply. Something happened indeed during the last update I just got (version 3.2.0-1 from the Passbolt Debian/Ubuntu repo).

2021-06-01 16:38:51 Error: Invalid stream reference provided
2021-06-01 16:38:51 Error: The avatar with file Avatar/38/c0/aa/60eca304d7074a8f83360a9c9906da1f/60eca304d7074a8f83360a9c9906da1f.png could not be saved.
2021-06-01 16:38:51 Error: Invalid stream reference provided
2021-06-01 16:38:51 Error: The avatar with file Avatar/c7/fc/59/c04bbd737857496d8ab0a00adb29ae5c/c04bbd737857496d8ab0a00adb29ae5c.jpg could not be saved.
2021-06-01 16:38:51 Error: Invalid stream reference provided
2021-06-01 16:38:51 Error: The avatar with file Avatar/4d/dc/19/ea44b5b1db2e49d385bb071ff94ef73e/ea44b5b1db2e49d385bb071ff94ef73e.png could not be saved.
2021-06-01 16:38:51 Error: Invalid stream reference provided
2021-06-01 16:38:51 Error: The avatar with file Avatar/f3/9c/67/fff05c7575b2486681b2778d9ef2f608/fff05c7575b2486681b2778d9ef2f608.jpg could not be saved.
 == 20210125212543 V320TransferFileStorageToAvatars: migrated 0.0263s

I just installed the package dist version of /etc/passbolt/file_storage.php.

Diff:

--- /etc/passbolt/file_storage.php	2021-05-18 03:33:43.579120837 +0200
+++ /etc/passbolt/file_storage.php.dpkg-new	2021-06-01 15:57:47.000000000 +0200
@@ -1,35 +1,35 @@
 <?php
+
+use App\Model\Table\AvatarsTable;
 use Cake\Core\Configure;
-use Burzum\FileStorage\Storage\StorageUtils;
-use Burzum\FileStorage\Storage\StorageManager;
+use Cake\ORM\TableRegistry;
+use League\Flysystem\Local\LocalFilesystemAdapter;
+
+/** @var AvatarsTable $AvatarsTable */
+$AvatarsTable = TableRegistry::getTableLocator()->get('Avatars');
+$AvatarsTable->setFilesystem(new LocalFilesystemAdapter(TMP . 'avatars'));
 
 // File storage and images
-Configure::write('ImageStorage.adapter', 'Local');
-#Configure::write('ImageStorage.basePath', WWW_ROOT . 'img' . DS . 'public' . DS);
-Configure::write('ImageStorage.basePath', '/usr/share/php/passbolt/webroot/img/public/');
+Configure::write('ImageStorage.basePath', WWW_ROOT . 'img' . DS . 'public' . DS);
 Configure::write('ImageStorage.publicPath', 'img' . DS . 'public' . DS);
-Configure::write('FileStorage', array(
-    // Configure the `basePath` for the Local adapter, not needed when not using it
-#    'basePath' => APP . 'FileStorage' . DS,
-#    /passbolt/webroot/img/public/images/Avatar/65/90/13/5ab52135926c44ee8c384c6ac0a88a1b/5ab52135926c44ee8c384c6ac0a88a1b.jpg    
-	'basePath' => '/usr/share/php/passbolt/webroot/img/public/',
+Configure::write('FileStorage', [
     'imageDefaults' => [
         'Avatar' => [
-            'medium' =>  'img' . DS . 'avatar' . DS . 'user_medium.png',
-            'small' =>  'img' . DS . 'avatar' . DS . 'user.png',
+            AvatarsTable::FORMAT_MEDIUM =>  'img' . DS . 'avatar' . DS . 'user_medium.png',
+            AvatarsTable::FORMAT_SMALL =>  'img' . DS . 'avatar' . DS . 'user.png',
         ]
     ],
     // Configure image versions on a per model base
     'imageSizes' => [
         'Avatar' => [
-            'medium' => [
+            AvatarsTable::FORMAT_MEDIUM => [
                 'thumbnail' => [
                     'mode' => 'outbound',
                     'width' => 200,
                     'height' => 200
                 ],
             ],
-            'small' => [
+            AvatarsTable::FORMAT_SMALL => [
                 'thumbnail' => [
                     'mode' => 'outbound',
                     'width' => 80,
@@ -42,11 +42,4 @@
             ],
         ]
     ]
-));
-
-StorageUtils::generateHashes();
-StorageManager::config(Configure::read('ImageStorage.adapter'), [
-    'adapterOptions' => [Configure::read('ImageStorage.basePath'), true],
-    'adapterClass' => '\Gaufrette\Adapter\Local',
-    'class' => '\Gaufrette\Filesystem'
 ]);

…and I can’t tell you exactly, what hosed my previous installation.

BUT …everything is great now! Avatars save and load just fine.

Keep up the good work, it is highly appreciated!

[remy]

@jab4 in version v3.2 we reworked how avatar are stored. A copy is now saved on database and file cache is rebuild when needed. That’s most likely why your problem is solve now. Cheers,