I'm not sure there's any controversy about this at all amongst the scientific community. Drivers on cell phones have been reliably associated with car accidents, and there's a fair bit of evidence that using a hands-free device doesn't help, which suggests that the challenge is attentional, not manual.

The intellectual challenge of scientific research is to come up with experimental designs that allow you to test hypotheses while minimizing expense, time, and risk to the participants. Researchers have come up with several ways of measuring the risk of cell phone distractions to drivers. A quick look at Google Scholar gets you papers on driving safety and cell phones from the Journal of Experimental Psychology: Applied (Strayer, Drews & Johnstone, 2003), another from Accident Analysis & Prevention (Matthews, Legg & Charlton, 2003), Human Factors (Strayer & Drews, 2004) and a piece that comes closest to answering your question directly, in BMJ [British Medical Journal]:

Role of mobile phones in motor vehicle crashes resulting in hospital attendance: a case-crossover study (McEvoy, Stevenson, McCartt, Woodward, Haworth, Palamara & Circarelli, 2005).

The findings are available at http://www.bmj.com/cgi/content/abstract/331/7514/428, but I can summarize here. They looked at 456 drivers who owned or used cell phones and had been injured in accidents seriously enough to require a hospital visit. Researchers checked to see who used their cell phone within ten minutes before their accident, and compared that to their cell phone use around trips at the same time of day in the previous week. "Driver's use of a mobile phone up to 10 minutes before a crash was associated with a fourfold increased likelihood of crashing (odds ratio 4.1, 95% confidence interval 2.2 to 7.7, P < 0.001)." Or, as the researchers concluded, "When drivers use a mobile phone there is an increased likelihood of a crash resulting in injury. Using a hands-free phone is not any safer."

A fourfold increase in the risk of crashing, with a P < 0.001, is weighty evidence.