CHRONICLE. The more important an issue is, the cruder the measurement methods often are, thinks Akademiliv’s chronicler Henrik Sjövall. Both observational studies and qualitative literature can, in his opinion, contribute strongly to evidence-based truth.
This chronicle originated in the somewhat puzzling results of a degree project. Medical student Lovisa Perman and I studied the associations among the parameters of quality of life, health, and health knowledge in Somali women at various levels of educational attainment, and found a pattern diametrically opposed to the media image. This led me to ponder the concept of truth. I have written about this before in Akademiliv, then in connection with the Cochrane movement’s world conference (“Colloquium”) in Seoul. As I explained then, Cochrane is a global non-governmental organization dedicated to a quest for truth using a common method book. Archie Cochrane (1909–88), a British doctor, questioned much of the medical knowledge of his time, and launched the practice of the “randomized controlled trial” (RCT) to find out the truth.
Beginning with the second best instead
Reality has now caught up with RCTs. The biggest practical problem is that the costs of a well-designed RCT are so high that there must be a “benefit” from the result to make someone (in practice almost always a pharmaceutical company) willing to invest, which means that many urgently needed studies are not done. Besides, many vital questions cannot be studied through RCTs. For example, it is impossible to randomize case management with or without corruption. Instead, you must then start with the next best thing: data on “what happened”, or in other words observational studies. The Cochrane movement used to avoid those areas, but in Seoul I met some brave souls who ventured in there, and their number is said to have grown since then. The truth discussion has now been further polarized by the launch of the fake-news concept, which challenges the very notion of truth.
Frequent mismatch between reality and RCT findings
In the serious academic world, there is also relatively well-founded criticism of the evidence movement, which claims that everything we do must be based on scientifically verified knowledge. I was recently at a mini-symposium in Örebro to mark the 20th anniversary of the County Council’s Health Technology Assessment center CAMTÖ, and heard a provocative lecture by none other than Bengt Gerdin, who wrote the Macchiarini report.[1] Gerdin showed pictures that strongly challenged parts of the evidence literature, and also struck a blow for “real-life data” (from registers, for example) sometimes being necessary for reliability, since reality often fails to reproduce what one has found in the meticulously structured RCTs. The best-known RCT critic is John P.A. Ioannidis, the Stanford professor. One of his assertions is that at least half of the results in academic articles, even in highly reputed journals, cannot be reproduced. Gerdin also cited an article that, based on data from RCTs, concluded that A>B>C>A. This obviously violates the laws of logic.
[1] Translator’s note. Its full title is The Macchiarini Case: Investigation of the synthetic trachea transplantations at Karolinska University Hospital. The whole report (in Swedish) is available here, and a 32-page Summary in English here.
Is there no truth?
How can this be? Actually, it is probably not that odd; without meticulous monitoring within a study framework, after all, people often omit to take their pills, miss follow-up appointments, and, moreover, as a rule couldn’t care less about following the best-intentioned exercise and dietary advice. And all this combined sometimes entirely wipes out the benefits of drugs that even under optimal conditions have quite modest effects!
Should we still prescribe, despite knowing all this? Is the fake-news movement simply right — is there no truth? And if the studies are not reliable anyway, can’t we simply do what we want?
A poem by Hjalmar Gullberg, “The thing in itself”, comes to mind. Örtstedt, a fairly grumpy lecturer, is sitting and reading Kant: “But the philosopher’s German is heavy indeed / Soon our teacher sleeps with head on this screed.” He dreams that he is sent a parcel containing “THE THING IN ITSELF.” After much rumination, he returns it to the sender unopened.
Huge variation from one experiment to another
My interest in the notion of truth comes from the fact that during my scientific life I mainly devoted myself to measuring weak signals in complex systems. In the early 1980s, I began measuring various preparation-sensitive parameters on anesthetized experimental animals, with huge variation from one experiment to another. I then switched to human physiology, and devoted just over a decade to a curious and extremely variable motor program in the stomach and intestines, the “migrating motor complex.” Despite applying poor methods, I nonetheless gained considerable experience of managing variability.
I watched with some envy how “test-tube researchers” often seemed to need to make only one experiment — the result was always the same. I therefore worked for a period in the successful mucin group under the leadership of Gunnar Hansson. However, I learned, there too, that complex biochemistry has its variability.
My latest diversion is that I have become increasingly interested in “humanistic” research areas, such as public health or social medicine and narrative medicine. There, I have noted that the more important an issue is, the coarser the measurement methods often are; not infrequently, we have to settle for categorical outcomes of the yes/no type. These studies are inherently interesting but, viewed through strictly source-critical eyeglasses, as a rule there has not been much left of them.
Upgrading the value of “observational literature”
I had no track record in public health or social medicine, so applying for research grants did not feel realistic. It then struck me that the medical students’ degree projects might be a way of tackling interesting questions without research grants. I have now supervised nine degree projects with a public-health profile: seven in Nepal and two in Uganda. I have written about some of them here. We looked at, for example, how a small mountain hospital in Nepal functions, how village women can be instrumental in public-health work, and what factors affect doctors’ propensity to work up in the mountains or perhaps leave the country. I also had two students in Uganda who studied the effects of teaching poor women to read. Of course, all these reports had many formal shortcomings (which are acceptable in Master’s dissertations), but they nevertheless provided a great deal of interesting information. In the latest reports I have also ventured into qualitative methodology, since large quantities of important information are best packaged in a story, not in numbers.
So now I am left with the following reflections:
- Perhaps the truth requirements of the evidence-based movement are too stringent, and there is a risk of throwing the baby out with the methodological bathwater.
- We should perhaps upgrade the value of “observational literature,” and even demand confirmation with “real-world data,” before we decide that the conclusions hold.
- We should surely also stop looking down on the qualitative literature. It may well be that where the evidence runs out, the story picks up.
I shall now end the circle where I began, by describing the study on illiterate Somali women’s quality of life that I mentioned in the intro. My interest in the subject stemmed from the Ugandan study, where we found that teaching poor women in a fishing village on Lake Victoria to read and write had an obvious effect in terms of empowerment and quality of life.
Strong women looking after large families
During the work on this essay, I reflected that illiterate women are a category that exists in Sweden as well, and as in a natural follow-up project, I started looking for this demographic. At first, I hardly found any such women; they seemed to be completely under the radar. At last, however, I came across the Swedish for Immigrants (SFI) unit of the Workers’ Educational Association of Sweden in Odinsgatan, central Gothenburg.
Lovisa Perman, a medical student interested in the subject, also turned up and we started. The literacy interest groups were delighted to be asked, and as controls we used those who had reached the SFI B, C and D levels, and a group of highly educated people. It became an odd kind of cross-sectional study where we compared a large number of parameters among four groups with different levels of educational attainment, from illiterate to highly educated. We measured, for example, their quality of life, finances, health, experience of care, health literacy, children’s situation, and belief in the future.
We found something that did not match the media picture at all. These are strong women who often manage large families single-handed. They stick together and help one another, are very happy with the Swedish care services, and think health checks are extremely important. But their children’s schooling is most important of all to them.
The mere fact that this category exists should justify allocating more resources to literacy training.
Of course this can be shredded by source-critical arguments, such as that the study design includes many confounders and respondent numbers are low. But at the same time the picture fits well with the experience of those who work with this demographic. Doing an RCT on this simply does not seem feasible. The proportion of this category, Somali women of low educational attainment, to which this pattern applies is difficult to assess; we simply invited everyone we came into contact with to participate, and some said no. But conceivably the mere existence of the category should justify allocating more resources to literacy training. Today, the work is based mainly on the active involvement of individuals, which makes it highly vulnerable. In this context I must, in particular, mention the support provided by the SFI director of studies at Odinsgatan, Anders Malm, and our outstandingly skilled interpreter Radiya Mohamud. Without her phenomenal efforts, this project would not have been practicable at all.
Thank you, Radiya, for helping us to meet this group, and good luck in your continued work with these strong women who, despite having almost everything against them, never give up — never ever.
And finally: Thank you too, wonderful Lovisa, because you were bold enough to take on this risky project. If nothing else, I hope you learned something about the art of measuring weak but important signals in a complex, changing world!
