This essay was co-authored with Josh Castle.
In case you haven’t heard, I started a podcast about 15 months ago called Clearer Thinking with Spencer Greenberg. The format is that we invite brilliant guests to bring 4 or 5 “ideas that matter”, and then we aim to have a fun, intellectual discussion about those ideas. Over the course of several months, listeners repeatedly requested transcripts for the show. So, our production team started doing a little bit of research to figure out how much time and money transcripts would cost. As a trial run, Josh Castle (the show’s producer) manually transcribed a single episode. Unfortunately, he discovered that it took him between three and four hours to produce a transcript for a one-hour episode because he had to listen at slower speeds and/or repeat sections multiple times. So, we guessed that the next best thing would be to use an AI transcription service to produce a rough transcript and then (if needed) have a human follow behind and clean up mistakes it may have made. I polled my friends and listeners on Facebook and Twitter, asking them to suggest their favorite AI transcription services. The four that came up consistently were:
We’ve analyzed these four services for accuracy and cost and have published the results below. Hopefully, this analysis will be useful to you!
Cost
As of October 2021, the length of an average Clearer Thinking episode was 80.7 minutes. And, since there are about 4.3 weeks per month on average, and since (at the time of this experiment) we were releasing two episodes per week, that means we’d need to transcribe about 694 minutes per month. Importantly, Descript and Otter charge per user per month, and Happy Scribe and Rev charge per minute of audio. Based on the assumption that three people would be using the service (me, Josh, and Ryan Kessler, the show’s audio engineer) and that we’d need to transcribe about 694 minutes per month, here’s what we estimated the monthly costs to be. Note that where it says “Human transcription monthly cost” in the table below, this is referring to the fact that two of the services (Happy Scribe and Rev, but not Descript or Otter.ai) offer the option of automatically having a human on their end fix the mistakes of the A.I. We did not end up trying these A.I. + human services they offer, though as you’ll see later in the article we did end up using our own human process for fixing mistakes in the A.I. automated transcription.
(NOTE: Please bear in mind that these were the prices listed in October 2021. They may have changed since we did our research.)
Test #1: Transcribing 40 seconds of audio
Our first test was to have the artificial intelligence from each of the four services produce an automated transcript from the same short audio clip (which was about 40 seconds long). Those transcripts are included at the bottom of this post. We did not have the services identify speakers in this short clip. After getting the transcripts back, we went through and marked them up with corrections as a way of trying to visually indicate the accuracy of each transcript. Based on this first test, Happy Scribe seemed to be the 2nd to least accurate. Since it was also pretty expensive, we dropped it from the competition.
Test #2: Transcribing 5 minutes of audio
For our second test, we had the three remaining A.I. systems return an automated transcript of a 5-minute audio clip and mark speaker changes and corresponding timestamps. These longer transcripts are also included at the bottom of this document. Since the transcripts were much longer in this test, we opted for an automated approach for determining accuracy. We made our own transcript of the audio clip and then scored the other three transcripts against ours using the following algorithm.
The Levenshtein distance between two strings is the integer number of patches (i.e., insertions, deletions, or substitutions) required to transform one string into the other. A Levenshtein distance of 0 indicates that the two strings are identical because 0 patches are required. The maximum possible Levenshtein distance is the length of the longer of the two strings, which indicates that all characters must be patched. We used the Google diff-match-patch library’s Levenshtein distance function in the following way (in pseudo-code):
(Note that we standardized the speaker and timestamp notation across all of the transcripts so that differing notations wouldn’t count against a transcript’s score.)
Given that algorithm, here are the scores (where numbers closer to 1 are better):
Otter ⟶ 0.90
Rev ⟶ 0.88
Descript ⟶ 0.85
So, Otter won by a small margin on that metric.
We noticed that Otter and Rev were pretty much equally good at marking speaker changes and timestamps, but Descript was significantly worse on both (1) breaking paragraphs along speaker changes and (2) figuring out which person was speaking. Another strike against Descript is that its web interface currently has limited functionality; it seems just from our own observations that they probably use client machines instead of (or in addition to) their own servers to handle some of the heavy lifting, so full functionality is only available by downloading and installing the desktop application (which is currently only available for macOS and Windows). So, based on those factors, we decided to drop Descript from the competition.
Conclusion
Since the final comparison came down to Otter and Rev, we opted for Otter because they charge per user per month rather than per minute of audio and would therefore be significantly cheaper for us. For other podcasts that don’t need to transcribe as many minutes per month, the difference in price between the two services won’t be as significant. Also, some other variables that we didn’t consider were the time required to produce the automated transcript, the maximum number of minutes of audio that could be transcribed per month (for companies that charge per user per month), and the relative costs of AI+human transcriptions offered by some of these companies.
Even though Otter was the winner, its transcripts still weren’t accurate enough on their own, especially when technical terms or proper nouns were introduced. So we hired a human transcriptionist to clean up Otter’s automated transcripts.
I hope that this analysis will be useful to you! Feel free to leave comments and questions below! (Also, I hope you’ll check out our Clearer Thinking podcast if you haven’t already done so!)
Short Transcriptions (40-second clip)
Descript
Hannah welcome. It’s great to have you here. Very glad to be here, Spencer. So the first half I want to talk to you about is the great rationality of debate. What is the great rationality debate when it comes to the great rationality debate? I’m often confused. Many people have never actually heard about it at all, but in the whole topic of rationality, there is a couple of basically tribes that disagree with one another.
And two of these tribes are kind of the extra medic approach to rationality and to ecological approach to rational. And one of those two are counted, mandatory ski, and with the other, most people would, if they know about it, associate .
Mistakes = 17
Happy Scribe
Anna, welcome. It’s great to have you here.
Very glad to be here, Spencer. So the first top I want.
To talk to you about is the great rationality debate. What is the great rationality debate.
When it comes to the great rationality debate? I’m often confused that many people have never actually heard about it at all. But in the whole topic of rationality, there is a couple of basically tribes that disagree with one another. And two of these tribes are kind of the exotic approach to rationality and the ecological approach to rationality. And one of those two are county monitors and with the other. Most people would if they know about it. Associate, get answer.
Mistakes = 11
Otter
Anna, welcome. It’s great to have you here. Very glad to be here, Spencer. So the first tip I want to talk to you about is the great rationality debate. What is the great rationality debate? When it comes to the great rationality debate, I’m often confused. Many people have never actually heard about it at all. But in the whole topic of rationality, there is a couple of basically tribes that disagree with one another. And two of these tribes are kind of the axiomatic approach to rationality and to ecological approach to rationality. And one of those two are counting on a turski and with the other most people would if they know about it, associate get get answer.
Mistakes = 8
Rev
Hannah welcome. It’s great to have you here.
Very glad to be here, Spencer. So the
First half I want to talk to you about is the great rationality of debate. What is the great rationality debate?
When it comes to the grade rationality debate, I’m often confused. Many people have never actually heard about it at all, but in the whole topic of rationality, there is a couple of basically tribes that disagree with one another. And two of these tribes are kind of the extra medic approach to rationality and to ecological approach to rationality. And one of those two are counting mandatory ski and with the other, most people would, if they know about it, associate [inaudible].
Mistakes = 8
Long Transcriptions (5-minute clip)
Descript
Josh (0:00):
Our world events affecting your mental health. Try uplift. The Boston globe said this app could be the future of mental health and it’s used by thousands of people. And a study on uplift users felt an average of 52% better in just one month. It comes with 12 interactive and information packed sessions that help you master wellbeing skills that are typically taught in therapy.
You do the sessions independently, but the uplift narrator feels like you’ve got a compassionate mentor guiding you towards long-lasting success the entire way. You’ll also have a toolbox of mood boosting techniques in your pocket at all times that you can use to feel immediate relief when. The first session and several tools are free and you can try the full program with a seven day trial discover, uplift, and feel happier, calmer, and mentally stronger to find out more and to get started.
Visit uplift.app. That’s uplift dot a P P.
Spencer (1:01):
Okay. So, so let’s go, let’s go into the next topic, which is, uh, global warming. And maybe also we can touch on attitudes towards scientists and trust in science and things like that. So in the survey I conducted conservatives were more likely to say, quote, science is overrated. Scientists are less likely to be correct when people think liberals were more likely to say, humans are causing substantial warming our planet.
And we’re somewhat more likely to see. Uh, did they value not destroying the natural environment? Liberals were also more likely to say that they believe in the conclusions of science. So last time Hank went first. So Cassandra, do you want to go first on this one? Yeah, I mean, I like the liberal characterization.
I’m not sure that’s fair to conservatives though, but I guess that’s the result that your survey showed, like, from what I know about Hank, I, I’m not sure. Do you agree with the conservative. You know, I don’t have the same response to this as I do to several other things here, which is, I think it’s an artifact of the current situation.
I think conservatives are distrustful of. So I do think they are less likely to, to trust quote science, close quote, which is to say what’s reported as science to them because they’re skeptical in general of. What’s reported as the expert opinion. And I think there’s good reason for that actually given the, given the poor quality of reporting on things and education, frankly, on things, I don’t know that conservatives are distrustful of science and fields outside of policy.
Yeah, well, if they don’t touch on policy, but that would be interesting. That’d be interesting to see, like, are conservatives less likely to trust, let’s say chemist or something like that, some topic that has like no political bend to it, or is it just when it comes in contact with kind of political topics, it might be interesting to say, are conservatives or liberals less comfortable riding in airplanes or less trustful of computers or whatever, pick something where they’re actually testing their faith in, in the application of.
But certainly I am distrustful of the popular reporting on science, because I think that there’s, there’s too many steps between the science and the reporting, and there’s too many agendas that drive it. So I think that the reporting is biased or you think that the scientists are biased, or I think that, oh, about both the scientists are people and science, so science is going to be biased.
Like people are biased. Reporters are people. And we know that reporters are, I mean, if you just got. I don’t want to get political, but if you go by political registration, we know that they don’t one way I would expect that they’re going to express themselves. I mean, we know that school teachers are biased.
Right. I think we’ll give you a fair to say. I think it’s fair to say everyone’s bad. Right? Right. So, so if you’re in a position where you speak, where you’re, it’s your job to communicate something, it’s likely that your bias is going to affect how you communicate. And if you’re the filter between something that’s as arcane and impenetrable as science and a largely scientifically illiterate.
And you’re yourself, a scientifically illiterate reporter, probably. Right. What you’re going to do is you’re going to take the most, let’s go back to global warming for a second, and then I’ll, I’ll yield the Florida Cassandra, but you’re only going to take the most alarming statement from the IPC report.
And you’re going to turn that into a headline and you’re going to ignore the much more likely statement elsewhere in the report that where the scientists say, but that’s not likely to actually happened. It’s more likely to be within this. Right. You’re going to, because that’s what, you know, listen to the report of an earthquake.
Somewhere in the initial reports will always describe an estimate of far more people. Almost, always, not every single time, but almost always described far more people likely killed than actually are in the event. There’s a, there’s a natural tendency to get to lead with your strongest headlines. And I think that happens in the case of science too.
So. So we get, we hear that life has been discovered on Mars before we learned finally that that little fragment of rock in the Antarctic contains some structures that look a lot like what life produces in some other contexts, but no, it’s not evidence necessarily. Anyway.
Otter
Josh (0:02):
Our world events affecting your mental health, try uplift. The Boston Globe said this app could be the future of mental health. And it’s used by 1000s of people. In a study on uplift, users felt an average of 52% better in just one month. It comes with 12 interactive and information packed sessions that help you master wellbeing skills that are typically taught in therapy. You do the sessions independently, but the uplift narrator feels like you’ve got a compassionate mentor guiding you towards long lasting success the entire way. You’ll also have a toolbox of mood boosting techniques in your pocket at all times that you can use to feel immediate relief when you need it. The first session, and several tools are free. And you can try the full program with a seven day trial, discover uplift and feel happier, calmer and mentally stronger. To find out more and to get started, visit uplift.ap. That’s uplift.ap pain.
Spencer (1:01):
Okay, so let’s go. Let’s go into the next topic, which is global warming. And maybe also we can touch on attitudes towards scientists and trust and science and things like that. So in the survey I conducted, conservatives were more likely to say, quote, science is overrated, scientists are less likely to be correct when people think liberals were more likely to say humans are causing substantial warming to our planet. And were somewhat more likely to say that they value not destroying the natural environment. liberals were also more likely to say that they believe in the conclusions of science. So last time Hank went first. So Cassandra, do you want to go first on this one?
Cassandra (1:37):
Yeah. I mean, I like the liberal characterization. I’m not sure that’s fair to conservatives. So but I guess that’s the result that your survey showed, like, from what I know about, Hank, I? I’m not sure. Hank, do you agree with the conservative characterization?
Hank (1:53):
You know, I have the same response to this as I do to several other things here, which is I think it’s an artifact of the current situation. I think conservatives are distrustful of so I do think they’re less likely to to trust, quote, science, close quote, which is to say what’s reported as science to them, because they’re skeptical in general of what’s reported as the expert opinion. And I think there’s good reason for that actually, given given the poor quality of reporting on things and education, frankly, on things. I don’t know that conservatives are distrustful of science and fields outside of policy, you know, if they don’t touch on policy?
Spencer (2:33):
Well, that would be interesting. That’d be interesting to see like, are conservatives less likely to trust, let’s say, chemists, or something like that some topic that has like no political bent to it? Or is it just when it comes in contact with kind of political topics?
Hank (2:45):
It might be interesting to say, are conservatives or liberals less comfortable writing in airplanes, or less trustful of computers or whatever, pick something where they’re actually testing their faith in in the application of something. But certainly I am distrustful of the popular reporting on science. Because I think that there’s there’s too many steps between the science and the reporting. And there’s too many agendas that drive it. So.
Spencer (3:09):
So do you think that that the reporting is biased or you think that scientists are biased or both?
Hank (3:16):
I think that either both the scientists with people and science, science is going to be biased, like people are biased. reporters are people and we know that reporters are I mean, if you just go by don’t to get political, but if you go by political registration, we know that they don’t one way, I would expect that they’re going to express themselves. I mean, we know that school teachers are biased, right?
Spencer (3:36):
I think will give me a fair to say, I think it’s fair to say everyone’s
Hank (3:39):
Right. Right, right. So so if you’re in a position where you speak, where you’re, it’s your job to communicate something, it’s likely that your bias is going to affect how you communicate and what. And if you’re the filter between something that’s as arcane and impenetrable as science, and a largely scientifically illiterate public, and you’re yourself a scientifically illiterate reporter, probably right? What you’re going to do is you’re going to take the most, let’s go back to global warming for a second, then I’ll yield the Florida Cassandra, but you’re only going to take the most alarming statement from the IPCC report. And you’re going to turn that into a headline, and you’re going to ignore the much more likely statement elsewhere in the report that where the scientists say but that’s not likely to actually happen, it’s more likely to be within this range. Right? You’re going to because that’s what you know, listen to the report of an earthquake somewhere in the initial reports will always describe an estimate of far more people almost always, not every single time but almost always described, far more people likely killed than actually are in the event. there’s a there’s a natural tendency to get to lead with your strongest headline. And I think that happens in the case of science too. So we get so we get we hear that life has been discovered on Mars before we learn Finally, that that little fragment of rock in the Antarctic contains some structures that look a lot like what life produces and some other contexts, but no, it’s not evidence necessarily. Anyway, Cassandra
Rev
Josh (0:02):
Our world events affecting your mental health. Try uplift. The Boston globe said this app could be the future of mental health. And it’s used by thousands of people in a study on uplift. Users felt an average of 52% better in just one month. It comes with 12 interactive and information pack sessions that help you master wellbeing skills that are typically taught in therapy. You do the sessions independently, but the uplift narrator feels like you’ve got a compassionate mentor guiding you towards long lasting success. The entire way. You’ll also have a toolbox of mood boosting techniques in your pocket at all times that you can use to feel immediate relief when you need it. The first session and several tools are free and you can try the full program with a seven day trial discover, uplift, and feel happier, calmer, and mentally stronger to find out more and to get started, visit uplift.app. That’s uplift dot a P P.
Spencer (1:02):
Okay, so, so let’s go, let’s go into the next topic, which is, uh, global warming. And maybe also we can touch on attitudes towards scientists and trust in science and things like that. So in the survey I conducted conservatives were more likely to say, quote, science is overrated. Scientists are less likely to be correct when people think liberals were more likely to say, humans are causing substantial warming our planet. And we’re somewhat more likely to say that they value not destroying the natural environment. Liberals were also more likely to say that they believe in the conclusions of science. So last time Hank went first. So Cassandra, do you want to go first on this one?
Cassandra (1:37):
Yeah, I mean, I like the liberal characterization. I’m not sure that’s fair to conservatives though, but I guess that’s the result that your survey showed, like, from what I know about Hank, I, I’m not sure. Thank do you agree with the conservative characterization?
Hank (1:54):
I ended up the same response to this as I do to several other things here, which is, I think it’s an artifact of the current situation. I think conservatives are distrustful of. I do think they are less likely to, to trust quote science, close quote, which is to say what’s reported as science to them because they’re skeptical in general of what’s reported as the expert opinion. And I think there’s good reason for that actually given the, given the poor quality of reporting on things and education, frankly, on things, I don’t know that conservatives are distrustful of science and fields outside of policy, you know, if they don’t touch on policy.
Spencer (2:33):
Well, that would be interesting. That’d be interesting to see, like our conservatives are less likely to trust, let’s say chemists or something like that. Some topic that has no political bend to it, or is it just when it comes in contact with kind of political topics,
Hank (2:45):
It might be interesting to say, are conservatives or liberals less comfortable riding in airplanes or less trustful of computers or whatever, pick some thing where they’re actually testing their faith in, in the application of something. But certainly I am distrustful of the popular reporting on science, because I think that there’s, there’s too many steps between the science and the reporting and there’s too many agendas that drive it. So
Spencer (3:10):
You think that the reporting is biased? Or do you think that scientists are biased or both?
Hank (3:16):
I think that, oh, about both the scientists are people and science, so science is going to be biased. Like people are biased. Reporters are people. And we know that reporters are, I mean, if you just go by, don’t want to get political, but if you go by political registration, we know that they, they tilt one way. I would expect that they’re going to express themselves. I mean, we know that school teachers are biased. Right?
Spencer (3:37):
I think we’ll give you a fair to say. I think it’s fair to say everyone’s bad.
Hank (3:40):
Right? So, so if you’re in a position where you speak, where you’re, it’s your job to communicate something, it’s likely that your bias is going to affect how you communicate and what, and if you’re the filter between something that’s as arcane and impenetrable, as science and to largely scientifically illiterate public and you’re yourself, a scientifically illiterate reporter, probably right. What you’re going to do is you’re going to take the most, let’s go back to global warming for a second, and then I’ll, I’ll yield the Florida Cassandra, but you’re only going to take the most alarming statement from the IPC report. And you’re going to turn that into a headline and you’re going to ignore the much more likely statement elsewhere in the report that where the scientists say, but that’s not likely it actually happened. It’s more likely to be within this range, right? You’re going to, because that’s what, you know, listen to the report of an earthquake.
Hank (4:26):
Somewhere in the initial reports will always describe an estimate of far more people, almost double it’s, not every single time, but almost always described far more people likely killed that actually are in the event. There’s a, there’s a natural tendency to get to lead with your strongest headline. And I think that happens in the case of science too. So we get, so we get, we hear that life has been discovered on Mars before we learned finally that that little fragment of rock and the Antarctic contains some structures that look a lot like what life produces in some other contexts, but no, it’s not evidence necessarily anyway,
Human
Josh (0:02):
Are world events affecting your mental health? Try UpLift. The Boston Globe said this app could be the future of mental health, and it’s used by thousands of people. In a study on UpLift, users felt an average of fifty-two percent better in just a month. It comes with twelve interactive and information-packed sessions that help you master wellbeing skills that are typically taught in therapy. You do the sessions independently, but the UpLift narrator feels like you’ve got a compassionate mentor guiding you towards long-lasting success the entire way. You’ll also have a toolbox of mood-boosting techniques in your pocket at all times that you can use to feel immediate relief when you need it. The first session and several tools are free, and you can try the full program with a seven-day trial. Discover UpLift and feel happier, calmer, and mentally stronger. To find out more and to get started, visit UpLift dot app. That’s UpLift dot a pee pee.
Spencer (1:02):
Okay, so let’s go on to the next topic, which is global warming, and maybe we can also touch on attitudes towards scientists and trust in science and things like that. So, in the survey I conducted, conservatives were more likely to say quote, “Science is overrated; scientists are less likely to be correct than people think.” Liberals were more likely to say, “Humans are causing substantial warming to our planet,” and were somewhat more likely to say that they value not destroying the natural environment. Liberals were also more likely to say that they believe in the conclusions of science. So, last time, Hank went first; so, Cassandra, do you want to go first on this one?
Cassandra (1:38):
Yeah, I mean, I like the liberal characterization. I’m not sure that’s fair to conservatives, though; but I guess that’s the result that your survey showed. Like, from what I know about Hank, I’m not sure. Hank, do you agree with the conservative characterization?
Hank (1:54):
You know, I have the same response to this as I do to several other things here, which is I think it’s an artifact of the current situation. I think conservatives are distrustful of — I do think they’re less likely to trust [quote] “science” [close quote], which is to say what’s reported as science to them because they’re skeptical in general of what’s reported as the expert opinion. And I think there’s good reason for that, actually, given the poor quality of reporting on — and education, frankly — on things. I don’t know that conservatives are distrustful in fields outside of policy; you know, if they don’t touch on quality.
Spencer (2:34):
Well, that would be interesting to see, like, are conservatives less likely to trust let’s say chemists or something like that, some topic that has like no political bent to it; or is it just when it comes into contact with kind of political topics?
Hank (2:45):
It might be interesting to say, “Are conservatives or liberals less comfortable riding in airplanes, or less trustful of computers?” Or whatever. Pick some thing where they’re actually testing their faith in the application of something. But certainly I am distrustful of the popular reporting on science because I think that there’s too many steps between the science and the reporting, and there’s too many agendas that drive it.
Spencer (3:10):
So, do you think that the reporting is biased, or you think that the science is biased, or both?
Hank (3:17):
Well, I think that…both. Scientists are people, and so science is going to be biased like people are biased. Reporters are people, and we know that reporters are — I mean, if you just go by — don’t want to get political — but if you just go by political registration, we know that they tilt one way. I would expect that they are going to express themselves — I mean, we know that school teachers are biased, right?
Spencer (3:37):
Well, I think we can be fair, I think it’s fair to say that everyone’s biased, right?
Hank (3:40):
Right, right. So, if you’re in a position where you speak, where it’s your job to communicate something, it’s likely that your bias is going to affect how you communicate and what you communicate. And if you’re the filter between something that’s as arcane and impenetrable as science and a largely scientifically illiterate public — and you’re yourself a scientifically illiterate reporter, probably, right? — what you’re going to do is you’re going to take the most — let’s go back to global warming for a second and then I’ll yield the floor to Cassandra — but you’re going to take the most alarming statement from the IPCC report and you’re going to turn that into a headline and you’re going to ignore the much more likely statement elsewhere in the report where the scientists say, “But that’s not likely to actually happen; it’s more likely to be within this range.” Right? You’re going to — Because that’s what — you know, listen to the report of an earthquake somewhere, and initial reports will always describe an estimate of far more people — almost always, not every single time, but almost always — describe far more people likely killed than actually are in the event. There’s a natural tendency to lead with your strongest headline, and I think that happens in the case of science too. So, we get — we get — we hear that life has been discovered on Mars before we learn, finally, that that little fragment of rock in the Antarctic contains some structures that look a lot like what life produces in some other context; but no, it’s not evidence, necessarily, of life on Mars. Anyway, Cassandra.
Comments