Author: Mike Pepp, Senior Commercial Consultant, Beyond Blue
Like many agencies, we are always striving to use new technologies that reduce time to insight – balancing this with the need to have confidence that the quality of our learnings and insights is maintained (or even improved). As a global agency, in recent years, we have been returning to the opportunity to use automated translation and transcription tools to support our team in extracting early insights from interviews conducted in languages other than English. This approach avoids having to wait for translated transcripts, which can take days or even weeks. Given our focus on healthcare, we are especially mindful of the need for any translation and transcription to accurately reflect both the technical elements of what physicians and patients tell us as well as the broader underlying meaning. Until recently, automated translations have simply not been good enough to be useful.
For a recent project, we needed early access to interviews to support initial analysis and hypothesis formation. As transcripts of Japanese interviews can take several days, and client budgets did not allow simultaneous translation of all interviews, we assessed the feasibility of using automated translation and transcription to support first-pass analysis. We compared the translated transcription for a single interview provided by three different sources:
1. A professional translator;
2. CoLoop, an AI-powered research and insights platform that provides automated translation;
3. An earlier auto translation tool we had access to, which we had previously agreed would not provide the clear and comprehensive translation we were seeking.
Our comparison focused on differences in meaning and the clarity of what was provided. As this was a review to support early access to insights, we did not conduct an assessment of the formal accuracy of the translation – we made an assumption that the professional translation would already provide a good indication of what was being said.
Overview of our findings
Professional translation: Has a somewhat formal tone, but includes specific elements to guide understanding (e.g., providing guidance that the hospital was a national or public hospital). Usually closely follows the original discussion, but occasionally summarises what is being said rather than giving the full detail of the moderator-respondent interaction.
CoLoop: Seems to balance accuracy with clarity about the idea being expressed and reflects the full conversational flow without becoming tired. For example, towards the end of the interview, the professional transcript noted “Basically, I’ll talk about how you should keep hitting”, which CoLoop captured as “Well, I usually tell patients that it’s better to keep receiving the injections”.
Earlier translation tool: A fairly literal translation, which can struggle with figures of speech, nuance, and technical detail, meaning it can require some interpretation, e.g., “if inflammation or pain isn’t that severe…” was transcribed as “the shape of the patient’s clothes”.
We quickly identified that CoLoop seemed to offer a better solution than our previous application, so we wanted to explore in more detail how it would compare with the professional transcripts we were planning to receive.
CoLoop vs Professional translation
To assess the usability of CoLoop-generated transcripts, we compared the output line by line against the professional transcript.
We noticed that CoLoop uses clear English, which makes it easy to follow. We also noticed four key elements:
1. As we ‘pre-trained’ CoLoop with technical terms, it consistently captured the correct names for clinical tests, conditions, and drugs.
2. There were times when CoLoop put sentences in a different order than the professional translation. These instances tended to make it easier to interpret the meaning. We did not see evidence of the reordering fundamentally changing the perceived meaning.
3. CoLoop did a good job of conveying the English meaning, where the human transcript could cause some confusion, for example, “the teacher” was translated as “the doctor”, and “the texture” was translated as “the disease”.
4. CoLoop consistently attributed the speaker and provided timestamps, increasing clarity and making it easy to query specific points if these were unclear.
Our key takeaways
CoLoop offers a good balance of accuracy, clarity, and early accessibility, making it suitable for first-pass assessment / early review to capture key insights. It seems to capture the meaning we would obtain from the human translation and keeps the context of the discussion intact. It is easy for readers unfamiliar with the original Japanese interview to follow, and we did not identify significant gaps or errors.
If these insights were useful and you’d like to discuss them further, reach out to our team, we’re always up for a conversation!