There is one other thing that would be too easy for a spy agency to buy off that it is hard to fathom that it hasn't happened: adding some kind of hook to popular closed-source operating systems to leak confidential information into covert channels. The effort required to modify kernel source code so that e.g., passwords or something are encoded into the output of some pseudorandom number generator used for picking e.g., TCP sequence numbers is low enough that a competent coder involved in the process between when the source code is checked out to do the production build and executing the makefile could slip it in with very low probability of detection and nobody would be any the wiser. If spy agencies can spend billions on covert spy satellite programs, surely they can spend a couple million to buy off a programmer involved in production builds to slip in carefully chosen patches. The presence of spy agencies in the world today with these kind of budgets almost guarantees that closed source products (and pre-built open source products) have backdoors. They would be stupid not to.

It follows that if you don't compile your own stuff, you're stuff is probably backdoored. On the positive side, keeping this stuff secret is almost certainly enough of a priority that the spy agencies will probably be keeping your secrets (unless you are doing something they are directly interested in) and third party crackers will probably not be any the wiser.