What is the exact reason that one should not cache the vendor directly #983

The readme mentions the following:

Please do not cache vendor directory using action/cache as that will have side effects.

I would like to know more about what is exactly the issue? What are these side-effects? I currently do cache the vendor dir, since I didn't came across the line in the readme at the time when I implemented it. So now I am wondering what the reason is as it does look like it works just fine.

(I will try and benchmark the difference between caching and not caching the vendor directly.)