How I made Crypt_GPG 100 times faster

Here’s the short story of performance investigation I did to make Crypt_GPG library (used by Roundcube’s Enigma plugin) 100 times faster. My fixes improved encryption/signing performance as well as peak memory usage.

Once I was testing Enigma for messages with attachments. I noticed it is really slow when encrypting or even just signing the message that is bigger than just a few kilobytes. There must have been somewhere a bottleneck. I did some raw tests on Crypt_GPG and found that encrypting 60MB file takes 350 seconds, while doing the encryption in command line only 4 seconds (sic!). The memory peak usage of my simple PHP7 script was around 250MB.

So, I found a couple of bottlenecks in the code. It ended up that the last from the list was the main culprit, but here’s the list in order I have been investigating.

1. Chunk size in pipe operations (#21077)

The code to handle communication with GnuPG binary is quite complicated so I first tried to find if we can do better with just some configuration. It looks that using bigger chunk (and buffer) size I could make the process 6 times faster.

2. mb_substr() vs. substr() (#21080)

To workaround mbstring.func_overload “issues” Crypt_GPG uses mb_strlen() and mb_substr() functions (with ‘8bit’ encoding). I found however, that they are 5 times slower than simple strlen() and substr(). They also use much more memory, which was the problem here as we work with big data string. However, when working with small strings the difference is negligible, see below.

3. Memory re-allocations (#21081)

Crypt_GPG writes input to a pipe in chunks, that’s fine. However, on every iteration it removes the processed chunk from the input string and this causes memory re-allocation. That’s a big issue when you consider the string is several MB and you do the operation >1000 times. Instead we can just use position pointer to extract the chunk from an input string and don’t modify it at all, eliminating some substr() calls on big data.

This is it! Only this makes the whole process 100 times faster. This also causes drop in memory peak usage from 250MB to 150MB (for 60MB input), which now looks reasonable as it’s close to 60MB input + 84MB output.

It looks like a pretty common mistake. I did the same in Roundcube’s sieve script parser, but fixed it some time ago.

So, now Crypt_GPG is as fast as command line use of GnuPG binary. Nice! The memory use is still pretty high when working with string data, but I don’t see a way to make it lower. When working with files the memory use is negligible, so there’s an option.

I observed a lot of interest in Enigma, also in my blog stats. Please, note that I’m working on the plugin voluntarily. Consider sending some bounty to me (PL 76 1050 1357 1000 0091 5047 7017 ) or Roundcube to let us know we should continue our work on encryption in Roundcube.