Y ou can't get crystal healing on the NHS. The Department of Health doesn't fund faith healing. And most doctors believe magnets are best stuck on fridges, not patients. But ask for a treatment in which an expert examines your tongue, smells your skin and tries to unblock the flow of life force running through your body with needles and the NHS will be happy to oblige.

The government declines to say how much the health service spends on acupuncture each year, but it's estimated to be around £25m. The NHS rationing body, the National Institute for Health and Care Excellence (Nice), says that doctors can prescribe acupuncture for lower back pain and chronic tension headaches. The NHS Choices website says there is "reasonably good evidence" that acupuncture is effective at treating a range of conditions, including back pain, dental pain, headache, nausea after operations and osteoarthritis of the knee. And there are plenty of anecdotes from patients who swear it works.

Of all the branches of complementary and alternative medicine, acupuncture has without doubt the most credibility among doctors and health officials. Not everyone is convinced, however.

In the past few years, scientific rationalists – who spend their lives debunking quackery – have turned their attentions to the ancient Chinese therapy and found it wanting. There is no mechanism to explain how needles could ease pain or treat disease that has been proven, they say. There is no evidence that it works for most of the conditions that acupuncturists treat. While there is research showing it may be mildly effective, the effect is weak. In trials, some patients benefit and others don't. And when acupuncture is tested on thousands of patients, the average benefit is too small for a person to notice.

Last month, in the journal Anesthesia and Analgesia, two leading medical rationalists, pharmacologist David Colquhoun and neurologist Steven Novella, stuck in the knife: "the benefits of acupuncture are likely nonexistent, or at best are too small and too transient to be of any clinical significance," they wrote. "It seems that acupuncture is little or no more than a theatrical placebo."

So are they right? Are the NHS – and half of GPs – misguided to support it?

The earliest known accounts of acupuncture appear in The Yellow Emperor's Classic of Internal Medicine, a text from the second century BC which laid down the principles of traditional Chinese medicine. Reports of acupuncture arrived in the west in the 17th century; it grew in popularity before falling out of favour globally by the 20th century.

Modern acupuncturists owe their livelihoods to Mao Zedong, who promoted Chinese traditional medicine during the Cultural Revolution (1966-1976) as a way to boost national identity and deliver cheap healthcare. It spread back to the west after President Nixon's visit to China in 1972, and has grown steadily in popularity.

In 2000, a British Medical Association survey showed that around half of doctors had prescribed acupuncture in the UK. Over 2,000 years, different branches of acupuncture have evolved, but most rely on the principles that a "life force" called qi flows through bodies along 12 channels or "meridians". Illness and pain occur when qi cannot flow freely because of stress, poor nutrition, infection or injury. Inserting fine needles into specific acupuncture points restores the flow of qi and aids the body's natural healing response.

But needles are just part of the story. Traditional acupuncturists inspect the face and body, and check the condition of the tongue, in the belief that different parts of the tongue are linked to different organs. They listen for wheezing and unusual sounds. They sniff for peculiar odours. And they check the patient's pulse for clues about the heart, flow of qi and state of the organs. It's a far cry from the brisk – and often brusque – five minutes snatched with a GP.

However, despite more than 3,000 studies into acupuncture since the 1970s, there is no evidence that any force resembling qi exists, or that it flows along invisible energy lines. The concept comes from a 2,000-year-old misunderstanding of the human body and a culture that did not perform medical dissections.

Of course, just because qi is meaningless in the context of medical science doesn't mean acupuncture doesn't work. Some medical acupuncturists argue that the therapy triggers the release of endorphins – natural painkillers. There is some evidence that needles do trigger pain-numbing chemicals, but it has never been shown that they are released in sufficient quantities to have a noticeable effect. Others have claimed that it works through the "gate control theory of pain", proposed in the 1960s by Patrick Wall and Ronald Melzack. The theory is based on the idea that the spinal cord contains a neural "gate" that can open and close to reduce or enhance pain messages passing to the brain. Stress and tension can open these gates, as can boredom, lack of activity and focusing attention on pain. Relaxation, distraction and physical activity can close the gates.

The theory remains controversial, but could explain why distraction and rubbing arms appear to ease pain. Similarly, it could also explain why the pinpricks of acupuncturists' needles do the same for some people. But for other scientists, the most plausible explanation for why acupuncture helps some patients is the placebo effect.

Any medicinal intervention – from a drug that costs millions to develop to a harmless sugar pill – can relieve symptoms simply because the patient is receiving attention from a doctor, or some form of treatment. The placebo effect varies according to the condition and the treatment, but it appears particularly effective in pain relief. And it continues to muddy the waters of research into acupuncture.

The gold-standard test of medicine is the double blind, randomised controlled trial. This involves taking a large group of patients randomly split into two: one group receives the real treatment, the other a useless placebo. Patients are unaware which treatment they are getting, the doctors don't know which they are dispensing, and the researchers assessing the results are "blind" too. Because both groups receive apparently identical treatments, the placebo effect is neutralised and researchers can see if a real treatment works.

That's fine for pills and potions, but how do you give a meaningful placebo for a therapy that involves sticking needles into the skin? Many studies haven't bothered, and have merely compared acupuncture with no treatment. These studies suffer from bias as a result, and so tend to have more positive outcomes: patients having any procedure tend to feel better than those who have none.

Of more significance are studies comparing "real" acupuncture with "sham" acupuncture. Often, these use a placebo, where needles are inserted in randomly selected points – a reasonable test for whether qi and meridians are real, but not necessarily meaningful if acupuncture works through a more medically plausible mechanism, such as the release of endorphins.

In the past few years, trials have used an improved sham developed by emeritus professor Edzard Ernst, who researches complementary medicine at Exeter University. He devised a needle that looks identical to an acupuncture needle but which retracts into itself after an initial small prick. Patients cannot see or feel whether the needle is going into their body.

"If we control rigorously for bias, by using proper sham needles, the trials tend to produce negative results," Ernst says.

Even the retracting needles are not a perfect sham, because they involve putting pressure on the skin, and this may have some impact on pain. And the tests are not truly double blind, because the doctor is aware which treatment is real and could unwittingly be passing on information to the patient. These flaws in all acupuncture studies mean there is a real risk of bias – even in the best research. That bias can skew the results slightly.

Given that it is so difficult to test, and the studies are so prone to bias, it isn't surprising that results of clinical trials are contradictory. So, rather than look at individual trials, supporters and critics of acupuncture alike argue that the best evidence comes from pooling data from good-quality trials. In the past few years there have been a handful of such trials, mostly in pain.

In 2009, the British Medical Journal published a meta-analysis of 13 trials at the Nordic Cochrane Centre in Copenhagen, comparing sham and real acupuncture, and involving 3,025 patients. Researchers noted a small difference in efficacy between real and sham acupuncture, and a moderate difference between acupuncture and no acupuncture. They reported: "A small analgesic effect of acupuncture was found, which seems to lack clinical relevance and cannot be clearly distinguished from bias. Whether needling at acupuncture points, or at any site, reduces pain independently of the psychological impact of the treatment ritual is unclear."

Emeritus professor David Colquhoun, a pharmacologist at University College London, says the difference between acupuncture and no acupuncture was on average – over all the studies assessed – equivalent to a 10-point difference on a 100-point scale commonly used by scientists to assess pain. That may seem a lot, but Colquhoun points out that given the subjective nature of pain, and the difficulties people have describing it, a 10-point difference is pretty insignificant. Most people would not be able to distinguish that sort of difference of pain intensity, which he describes as "minimal".

Similar findings came from a 2012 meta-analysis led by Andrew Vickers of the Memorial Sloan-Kettering Cancer Centre, New York, published in Archives of Internal Medicine. It looked at 29 randomised control trials involving 17,922 patients treated for chronic pain. Again, real acupuncture was slightly better than sham treatment, while acupuncture was moderately better than no acupuncture.

"They were barely clinically significant and that was with non-blind acupuncture, too," says Colquhoun. "The average effect would be too small for most patients to notice it."

Mark Bovey, spokesman for the British Acupuncture Council, who has practised acupuncture for 30 years, disagrees. He says the Vickers paper is the "strongest evidence for acupuncture" yet – and dismisses Colquhoun's interpretation as nonsense.

"The effects from the Vickers study were 15 to 20% over a placebo," he says. "And the placebo in this study was not a real placebo, just a weaker version of acupuncture – the difference of the same order as non-steroidal anti-inflammatory drugs [NSAIDs] for back pain, which are in common use and recommended by Nice."

And, Bovey says, it is meaningless talking about average pain scores.

"Some patients get more benefit and some get less. The more useful thing is to look at the proportion of people who get a significant benefit."

The most controversial of recent meta-analyses was the Cactus trial, reported in the British Journal of General Practice in 2011 and based on patients suffering "medically unexplained physical symptoms". The authors from the University of Exeter Peninsula Medical School concluded that acupuncture "resulted in improved health status and wellbeing that was sustained for 12 months".

Yet according to David Colquhoun, the results showed very little difference between even the acupuncture and no acupuncture groups. The paper led to a flurry of complaints to the journal and a mini internet storm. The apparently real, but small, benefits from acupuncture in pain control led to Nice approving acupuncture for headaches and back pain – amid a chorus of grumbles from the sceptics.

Colquhoun believes that, according to the usual standards of medicine, acupuncture doesn't work. "Every paper on acupuncture seems to conclude that 'more research is needed'," he says. "If you can't come to a clear decision after 3,000 trials then surely that tells you something. One thing is clear: there is little or no difference between sham and real acupuncture."

He believes the reported positive effects are so weak they may not even be the work of a placebo but a regression to the mean – the phenomenon whereby patients get better on their own, but attribute their recovery to whatever treatment they have had. Does it matter? Surely if patients believe acupuncture works, that is good enough, given that the treatment is relatively cheap.

"It matters for two reasons," says Colquhoun. "It is absolutely contrary to the increased openness that doctors are supposed to have: it involves a direct lie to the patient. Second, if – as appears to be the case – most placebo effects are quite small, then patients don't get much benefit from it anyway. They might get some psychological boost, but it doesn't cure their pain effectively; and that boost is not worth the dishonesty."

If the evidence for pain management is positive but weak, the evidence that acupuncture works for other conditions is even weaker. The Cochrane Collaboration (a global network of scientists co-ordinated from Oxford) has concluded that acupuncture may be effective for treating back and pelvic pain during pregnancy, low back pain, headaches, post-operative nausea, neck disorders and bed wetting. However, it says there is no evidence that it treats depression, irritable bowel syndrome, smoking, rheumatoid arthritis, non-specific back pain, shoulder pain, carpal tunnel syndrome, chronic asthma, stroke rehabilitation, Bell's palsy, epilepsy, insomnia, morning sickness, glaucoma, period pains or vascular dementia. And to the fury of the sceptics, that doesn't stop acupuncturists treating those conditions.

Simon Singh, science writer, co-author with Edzard Ernst of Trick or Treatment?, and a campaigner against pseudo-science, believes that the NHS's approval of the treatment for some conditions gives it a veneer of respectability for other conditions where there is no or little evidence.

"It is disappointing that patients are encouraged to try acupuncture for conditions where there is painfully weak evidence of benefit, but the real danger is that this is the thin end of the wedge," he says. "Patients are steered toward therapists who may be acting irresponsibly. Some acupuncturists make a series of absurd claims, claiming to treat everything from hay fever to depression, infertility, tinnitus and 'children's health'. Such acupuncturists," he says, "offer nothing more than hope in exchange for a great deal of cash."