The idea that a nano replicator will be under the control of a larger system has other interesting facets.

To be useful, we effectively need a nano replicator to be both a "universal constructor" (capable of making any molecule with its appendages) and a "universal computer" (capable of doing the thinking necessary to build something).

Unfortunately it's very difficult to build something so small which can think as well as do tasks!

Some models call for the thinking part to be a single separate macroscopic computer which sends commands to the machines which are slaved to it. Eventually the computer would need to control a very large number of replicators indeed, but by then a new control centre could possibly be build. Humans would be in control of the control centres.

This avoids the problem of the "stray infecting pathogen" because such a machine would be helpless without the control centre. Of course, replicators such as viruses do exist, so we can't rule out an entirely self contained machine, but I suspect it would be a lot less versatile, larger, bulkier, and slower to replicate than machines which don't need to think for themselves. The smaller, faster dumb robots would probably wipe them out.