Fix performance issue on large password database (2k / 4k passwords)


#1

We just imported our password database and we ran into performance issues like a brickwall.

We currently are on 2100 passwords and when the whole import is done we will likely be around 4000 passwords. Currently it takes 13 seconds for chrome to load the password overview and 62 seconds for firefox (gives warnings). The server is chilling and not doing much but the clients are on 70% cpu load.

Am thinking that the plugin should perhaps get 50 passwords sorted on favorites and place the others on other pages.

info:
Passbolt version: 1.6.5
Client hardware: i5 7360U 16GB mem 1GBit connection to server
Client software: Chrome 65.0.3325.181 & Firefox 59.0.1 (64-bits)
Server hardware: 2 xeon processing cores 4GB mem (600MB in use)
Server response info:
2096 passwords
456 milliseconds
4MB response (3,948,897 bytes) from
GET /resources.json?silentLoading=false&order%5B%5D=Resource.modified+DESC

Other sorts result in the same performance issue. With the exception of groups that have fewer passwords.

EDIT:
Please note that searching is pretty fast and clearing search loads results under a second on both Firefox and Chrome. I have a feeling that the plugin is verifying the signatures which takes a long time.


Performance issue on large password database v2.2.0
#2

@eddie4 thanks for the report, we’ll do a large fixture set to try to reproduce the setup with the v2 and see where we can squeeze some more performance.


#3

The obvious would be to paginate the results, and allow API-side password search instead of always loading all passwords.


#4

Yes, @cedric also started working on getting the secrets and keys separately only when needed, this will decrease the size of the request and improve front-end performance drastically. This is something we aim at doing in april.


#5

Yes please let me know if v2 fixes the problem or when there is a fix available.


#6

We’ll be working on fix shortly and let you know when it’s available. Thank you for your patience.


#7

Is there anyway that i can help? I am willing to spend a few hours running a alpha release.


#8

This items depends on a larger framework upgrade that @cedric is working on. We’ll ping you when we have something to test. Thank you for your patience.


#9

Hey,

We’ve got the same issue like @eddie4 . We’ve imported 2400 passwords to the latest pro version of passbolt and the browser crashes. It will be nice to have pagination, so we can use the passbolt app.

Thank you!


#10

Hi guys!

Is there any update on this issue ?

Thank you =)


#11

Hi @Nikolay,

It is among the top priorities in our backlog and will be tackled as soon as we can. @cedric has already fixed the main blocker (quite a painful one), which was the upgrade the underlying canjs framework. There are a few things cooking and this performance issue is part of it. Please bear with us a little longer :slight_smile:

Cheers,


#12

I have been visiting this topic every week from when I opened the issue. As this is an opensource project I didn’t want put pressure on you guys. However the patience of the business has started to run out and they are discussing switching to an alternative. Originally the date for the fix was mid April. Now we are 3 months over that date. Can you guys give us a bit more of a hard deadline would a bounty help getting this fixed or ?


#13

Hey @eddie4,

We already wrote the spec, the issue is planned to be fixed in the next release or the one after, it will be fixed this month.


#14

Awesome, I talked to the big chief here and we will offer a bounty of 150 euro if it is indeed done this month. I understand you guys have a premium product but we already wrote our own import tool and we would prefer to support you guys this way. Energy drinks & beer will be on us.


#15

Pagination would be great…
but improving server performance quickly takes only about 12 lines of code…

  1. by default open to ‘Favorites’ ‘Recently modified’ or ‘Items I own’ instead of the whole damn list
  2. set the max return records to 50 or 100 - really, who is going to scroll through more than 50 items in a list? click a group filter or search!
  3. disallow search with NULL, or less than 3 characters in the search box… this is not so important if you have added the 50 returned record max
  4. fix the recently modified list to actually pull only the last 7 days modified date

We have about 3,200 records and implementing some quick and dirty code for the above cut both the navigation times by 80%! we found most of the resource struggle is on the client-side render. but not returning all 3,200 records EVERY time someone logged on also helped on the server side.


#16

Awesome, I talked to the big chief here and we will offer a bounty of 150 euro if it is indeed done this month.

How is the coding coming along ?


#17

Hi @eddie4. Glad to see that you didn’t forget about it. It’s going well! A new release is coming up in the next few days. Stay tuned!!


#18

Hi guys,

Is this issue fixed already?

Thanks


#19

Hello @Nikolay,
A fix will be release with v2.2.0 this week.
Thank you for your patience.


#20

Thanks @cedric :slight_smile: this is really a good news!