{"slug": "sequence-modeling-with-ctc", "title": "Sequence Modeling with CTC", "summary": "Connectionist Temporal Classification (CTC) is an algorithm for training deep neural networks on sequence problems like speech and handwriting recognition where input-output alignment is unknown. CTC assigns probabilities to all possible alignments between input and output sequences, enabling efficient loss computation and inference without requiring explicit alignment.", "body_md": "A visual guide to Connectionist Temporal Classification, an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems.\n\nConsider speech recognition. We have a dataset of audio clips and corresponding transcripts. Unfortunately, we don’t know how the characters in the transcript align to the audio. This makes training a speech recognizer harder than it might at first seem.\n\nWithout this alignment, the simple approaches aren’t available to us. We could devise a rule like “one character corresponds to ten inputs”. But people’s rates of speech vary, so this type of rule can always be broken. Another alternative is to hand-align each character to its location in the audio. From a modeling standpoint this works well — we’d know the ground truth for each input time-step. However, for any reasonably sized dataset this is prohibitively time consuming.\n\nThis problem doesn’t just turn up in speech recognition. We see it in many other places. Handwriting recognition from images or sequences of pen strokes is one example. Action labelling in videos is another.\n\nConnectionist Temporal Classification (CTC) is a way to get around not knowing the alignment between the input and the output. As we’ll see, it’s especially well suited to applications like speech and handwriting recognition.\n\nTo be a bit more formal, let’s consider mapping input sequences , such as audio, to corresponding output sequences , such as transcripts. We want to find an accurate mapping from ’s to ’s.\n\nThere are challenges which get in the way of us using simpler supervised learning algorithms. In particular:\n\nThe CTC algorithm overcomes these challenges. For a given\nit gives us an output distribution over all possible ’s. We\ncan use this distribution either to *infer* a likely output or to assess\nthe *probability* of a given output.\n\nNot all ways of computing the loss function and performing inference are tractable. We’ll require that CTC do both of these efficiently.\n\n**Loss Function:** For a given input, we’d like to train our\nmodel to maximize the probability it assigns to the right answer. To do this,\nwe’ll need to efficiently compute the conditional probability\nThe function should\nalso be differentiable, so we can use gradient descent.\n\n**Inference:** Naturally, after we’ve trained the model, we\nwant to use it to infer a likely given an\nThis means solving\nIdeally can be found efficiently. With CTC we’ll settle\nfor an approximate solution that’s not too expensive to find.\n\nThe CTC algorithm can assign a probability for any given an The key to computing this probability is how CTC thinks about alignments between inputs and outputs. We’ll start by looking at these alignments and then show how to use them to compute the loss function and perform inference.\n\nThe CTC algorithm is *alignment-free* — it doesn’t require an\nalignment between the input and the output. However, to get the probability of\nan output given an input, CTC works by summing over the probability of all\npossible alignments between the two. We need to understand what these\nalignments are in order to understand how the loss function is ultimately\ncalculated.\n\nTo motivate the specific form of the CTC alignments, first consider a naive approach. Let’s use an example. Assume the input has length six and [c, a, t]. One way to align and is to assign an output character to each input step and collapse repeats.\n\nThis approach has two problems.\n\nTo get around these problems, CTC introduces a new token to the set of\nallowed outputs. This new token is sometimes called the *blank* token. We’ll\nrefer to it here as The\ntoken doesn’t correspond to anything and is simply\nremoved from the output.\n\nThe alignments allowed by CTC are the same length as the input. We allow any alignment which maps to after merging repeats and removing tokens:\n\nIf has two of the same character in a row, then a valid alignment must have an between them. With this rule in place, we can differentiate between alignments which collapse to “hello” and those which collapse to “helo”.\n\nLet’s go back to the output [c, a, t] with an input of length six. Here are a few more examples of valid and invalid alignments.\n\nThe CTC alignments have a few notable properties. First, the allowed alignments between and are monotonic. If we advance to the next input, we can keep the corresponding output the same or advance to the next one. A second property is that the alignment of to is many-to-one. One or more input elements can align to a single output element but not vice-versa. This implies a third property: the length of cannot be greater than the length of\n\nThe CTC alignments give us a natural way to go from probabilities at each time-step to the probability of an output sequence.\n\nTo be precise, the CTC objective for a single pair is:\n\nModels trained with CTC typically use a recurrent neural network (RNN) to estimate the per time-step probabilities, An RNN usually works well since it accounts for context in the input, but we’re free to use any learning algorithm which produces a distribution over output classes given a fixed-size slice of the input.\n\nIf we aren’t careful, the CTC loss can be very expensive to compute. We\ncould try the straightforward approach and compute the score for each alignment\nsumming them all up as we go. The problem is there can be a massive number of\nalignments.\n\nThankfully, we can compute the loss much faster with a dynamic programming algorithm. The key insight is that if two alignments have reached the same output at the same step, then we can merge them.\n\nSince we can have an before or after any token in , it’s easier to describe the algorithm using a sequence which includes them. We’ll work with the sequence which is with an at the beginning, end, and between every character.\n\nLet’s let be the score of the merged alignments at a given node. More precisely, is the CTC score of the subsequence after input steps. As we’ll see, we’ll compute the final CTC score, , from the ’s at the last time-step. As long as we know the values of at the previous time-step, we can compute There are two cases.\n\n**Case 1:**\n\nIn this case, we can’t jump over , the previous token in The first reason is that the previous token can be an element of , and we can’t skip elements of Since every element of in is followed by an , we can identify this when The second reason is that we must have an between repeat characters in We can identify this when\n\nTo ensure we don’t skip , we can either be there at the previous time-step or have already passed through at some earlier time-step. As a result there are two positions we can transition from.\n\n**Case 2:**\n\nIn the second case, we’re allowed to skip the previous token in We have this case whenever is an between unique characters. As a result there are three positions we could have come from at the previous step.\n\nBelow is an example of the computation performed by the dynamic programming algorithm. Every valid alignment has a path in this graph.\n\nThere are two valid starting nodes and two valid final nodes since the at the beginning and end of the sequence is optional. The complete probability is the sum of the two final nodes.\n\nNow that we can efficiently compute the loss function, the next step is to compute a gradient and train the model. The CTC loss function is differentiable with respect to the per time-step output probabilities since it’s just sums and products of them. Given this, we can analytically compute the gradient of the loss function with respect to the (unnormalized) output probabilities and from there run backpropagation as usual.\n\nFor a training set , the model’s parameters are tuned to minimize the negative log-likelihood instead of maximizing the likelihood directly.\n\nAfter we’ve trained the model, we’d like to use it to find a likely output for a given input. More precisely, we need to solve:\n\nOne heuristic is to take the most likely output at each time-step. This gives us the alignment with the highest probability:\n\nWe can then collapse repeats and remove tokens to get\n\nFor many applications this heuristic works well, especially when most of the probability mass is alloted to a single alignment. However, this approach can sometimes miss easy to find outputs with much higher probability. The problem is, it doesn’t take into account the fact that a single output can have many alignments.\n\nHere’s an example. Assume the alignments [a, a, ] and [a, a, a] individually have lower probability than [b, b, b]. But the sum of their probabilities is actually greater than that of [b, b, b]. The naive heuristic will incorrectly propose [b] as the most likely hypothesis. It should have chosen [a]. To fix this, the algorithm needs to account for the fact that [a, a, a] and [a, a, ] collapse to the same output.\n\nWe can use a modified beam search to solve this. Given limited computation, the modified beam search won’t necessarily find the most likely It does, at least, have the nice property that we can trade-off more computation (a larger beam-size) for an asymptotically better solution.\n\nA regular beam search computes a new set of hypotheses at each input step. The new set of hypotheses is generated from the previous set by extending each hypothesis with all possible output characters and keeping only the top candidates.\n\nWe can modify the vanilla beam search to handle multiple alignments mapping to the same output. In this case instead of keeping a list of alignments in the beam, we store the output prefixes after collapsing repeats and removing characters. At each step of the search we accumulate scores for a given prefix based on all the alignments which map to it.\n\nA proposed extension can map to two output prefixes if the character is a repeat. This is shown at in the figure above where ‘a’ is proposed as an extension to the prefix [a]. Both [a] and [a, a] are valid outputs for this proposed extension.\n\nWhen we extend [a] to produce [a,a], we only want include the part of the previous score for alignments which end in Remember, the is required between repeat characters. Similarly, when we don’t extend the prefix and produce [a], we should only include the part of the previous score for alignments which don’t end in\n\nGiven this, we have to keep track of two probabilities for each prefix in the beam. The probability of all alignments which end in and the probability of all alignments which don’t end in When we rank the hypotheses at each step before pruning the beam, we’ll use their combined scores.\n\nThe implementation of this algorithm doesn’t require much code, but it is\ndense and tricky to get right. Checkout this\n[gist](https://gist.github.com/awni/56369a90d03953e370f3964c826ed4b0)\nfor an example implementation in Python.\n\nIn some problems, such as speech recognition, incorporating a language model over the outputs significantly improves accuracy. We can include the language model as a factor in the inference problem.\n\nThe function computes the length of in terms of the language model tokens and acts as a word insertion bonus. With a word-based language model counts the number of words in If we use a character-based language model then counts the number of characters in The language model scores are only included when a prefix is extended by a character (or word) and not at every step of the algorithm. This causes the search to favor shorter prefixes, as measured by , since they don’t include as many language model updates. The word insertion bonus helps with this. The parameters and are usually set by cross-validation.\n\nThe language model scores and word insertion term can be included in the beam search. Whenever we propose to extend a prefix by a character, we can include the language model score for the new character given the prefix so far.\n\nWe mentioned a few important properties of CTC so far. Here we’ll go into more depth on what these properties are and what trade-offs they offer.\n\nOne of the most commonly cited shortcomings of CTC is the conditional independence assumption it makes.\n\nThe model assumes that every output is conditionally independent of the other outputs given the input. This is a bad assumption for many sequence to sequence problems.\n\nSay we had an audio clip of someone saying “triple A”.\n\nIn fact speech recognizers using CTC don’t learn a language model over the\noutput nearly as well as models which are conditionally dependent.\n\nThe conditional independence assumption made by CTC isn’t always a bad thing. Baking in strong beliefs over output interactions makes the model less adaptable to new or altered domains. For example, we might want to use a speech recognizer trained on phone conversations between friends to transcribe customer support calls. The language in the two domains can be quite different even if the acoustic model is similar. With a CTC acoustic model, we can easily swap in a new language model as we change domains.\n\nThe CTC algorithm is *alignment-free*. The objective function\nmarginalizes over all alignments. While CTC does make strong assumptions about\nthe form of alignments between and , the\nmodel is agnostic as to how probability is distributed amongst them. In some\nproblems CTC ends up allocating most of the probability to a single alignment.\nHowever, this isn’t guaranteed.\n\nAs mentioned before, CTC only allows *monotonic* alignments. In\nproblems such as speech recognition this may be a valid assumption. For other\nproblems like machine translation where a future word in a target sentence\ncan align to an earlier part of the source sentence, this assumption is a\ndeal-breaker.\n\nAnother important property of CTC alignments is that they are\n*many-to-one*. Multiple inputs can align to at most one output. In some\ncases this may not be desirable. We might want to enforce a strict one-to-one\ncorrespondence between elements of and\nAlternatively, we may want to allow multiple output\nelements to align to a single input element. For example, the characters\n“th” might align to a single input step of audio. A character based CTC model\nwould not allow that.\n\nThe many-to-one property implies that the output can’t have more time-steps\nthan the input.\n\nIn this section we’ll discuss how CTC relates to other commonly used algorithms for sequence modeling.\n\nAt a first glance, a Hidden Markov Model (HMM) seems quite different from CTC. But, the two algorithms are actually quite similar. Understanding the relationship between them will help us understand what advantages CTC has over HMM sequence models and give us insight into how CTC could be changed for various use cases.\n\nLet’s use the same notation as before, is the input sequence and is the output sequence with lengths and respectively. We’re interested in learning One way to simplify the problem is to apply Bayes’ Rule: The term can be any language model, so let’s focus on Like before we’ll let be a set of allowed alignments between and Members of have length Let’s otherwise leave unspecified for now. We’ll come back to it later. We can marginalize over alignments to get To simplify notation, let’s remove the conditioning on , it will be present in every With two assumptions we can write down the standard HMM.\n\nThe first assumption is the usual Markov property. The state is conditionally independent of all historic states given the previous state The second is that the observation is conditionally independent of everything given the current state\n\nNow we can take just a few steps to transform the HMM into CTC and see how the two models relate. First, let’s assume that the transition probabilities are uniform. This gives There are only two differences from this equation and the CTC loss function. The first is that we are learning a model of given as opposed to given The second is how the set is produced. Let’s deal with each in turn.\n\nThe HMM can be used with discriminative models which estimate To do this, we apply Bayes’ rule and rewrite the model as\n\nIf we assume a uniform prior over the states and condition on all of instead of a single element at a time, we arrive at\n\nThe above equation is essentially the CTC loss function, assuming the set\nis the same. In fact, the HMM framework does not specify what\nshould consist of. This part of the model can be designed on a\nper-problem basis. In many cases the model doesn’t condition on and the\nset consists of all possible length sequences from the\noutput alphabet. In this case, the HMM can be drawn as an *ergodic* state\ntransition diagram in which every state connects to every other state. The\nfigure below shows this model with the alphabet or set of unique hidden states\nas\n\nIn our case the transitions allowed by the model are strongly related to\nWe want the HMM to reflect this. One possible model could\nbe a simple linear state transition diagram. The figure below shows this with\nthe same alphabet as before and [a, b]. Another commonly\nused model is the *Bakis* or left-right HMM. In this model any\ntransition which proceeds from the left to the right is allowed.\n\nIn CTC we augment the alphabet with and the HMM model allows a subset of the left-right transitions. The CTC HMM has two start states and two accepting states.\n\nOne possible source of confusion is that the HMM model differs for any unique This is in fact standard in applications such as speech recognition. The state diagram changes based on the output However, the functions which estimate the observation and transition probabilities are shared.\n\nLet’s discuss how CTC improves on the original HMM model. First, we can think of the CTC state diagram as a special case HMM which works well for many problems of interest. Incorporating the blank as a hidden state in the HMM allows us to use the alphabet of as the other hidden states. This model also gives a set of allowed alignments which may be a good prior for some problems.\n\nPerhaps most importantly, CTC is discriminative. It models directly, an idea that’s been important in the past with other\ndiscriminative improvements to HMMs.\n\nThe encoder-decoder is perhaps the most commonly used framework for sequence modeling with neural networks. These models have an encoder and a decoder. The encoder maps the input sequence into a hidden representation. The decoder consumes the hidden representation and produces a distribution over the outputs. We can write this as The and functions are typically RNNs. The decoder can optionally be equipped with an attention mechanism. The hidden state sequence has the same number of time-steps as the input, Sometimes the encoder subsamples the input. If the encoder subsamples the input by a factor then will have time-steps.\n\nWe can interpret CTC in the encoder-decoder framework. This is helpful to understand the developments in encoder-decoder models that are applicable to CTC and to develop a common language for the properties of these models.\n\n**Encoder:** The encoder of a CTC model can be just about any\nencoder we find in commonly used encoder-decoder models. For example the\nencoder could be a multi-layer bidirectional RNN or a convolutional network.\nThere is a constraint on the CTC encoder that doesn’t apply to the others. The\ninput length cannot be sub-sampled so much that\nis less than the length of the output.\n\n**Decoder:** We can view the decoder of a CTC model as a simple\nlinear transformation followed by a softmax normalization. This layer should\nproject all steps of the encoder output\ninto the dimensionality of the output alphabet.\n\nWe mentioned earlier that CTC makes a conditional independence assumption over the characters in the output sequence. This is one of the big advantages that other encoder-decoder models have over CTC — they can model the dependence over the outputs. However in practice, CTC is still more commonly used in tasks like speech recognition as we can partially overcome the conditional independence assumption by including an external language model.\n\nSo far we’ve mostly developed a conceptual understanding of CTC. Here we’ll go through a few implementation tips for practitioners.\n\n**Software:** Even with a solid understanding of CTC, the\nimplementation is difficult. The algorithm has several edge cases and a fast\nimplementation should be written in a lower-level programming language.\nOpen-source software tools make it much easier to get started:\n\n**Numerical Stability:** Computing the CTC loss naively is\nnumerically unstable. One method to avoid this is to normalize the\n’s at each time-step. The original publication\nhas more detail on this including the adjustments to the gradient.\n\n**Beam Search:** There are a couple of good tips to know about\nwhen implementing and using the CTC beam search.\n\nThe correctness of the beam search can be tested as follows.\n\nA common question when using a beam search decoder is the size of the beam to use. There is a trade-off between accuracy and runtime. We can check if the beam size is in a good range. To do this first compute the CTC score for the inferred output Then compute the CTC score for the ground truth output If the two outputs are not the same, we should have If then the ground truth output actually has a higher probability under the model and the beam search failed to find it. In this case a large increase to the beam size may be warranted.\n\nThe CTC algorithm was first published by Graves et al. in 2006.\n\nOne of the first applications of CTC to large vocabulary speech recognition was\nby Graves et al. in 2014.\n\nCTC has been used successfully in many other problems. Some examples are\nlip-reading from video\n\nMany extensions and improvements to CTC have been proposed. Here are a few.\nThe *Sequence Transducer* discards the conditional independence assumption\nmade by CTC.*Gram-CTC* model\ngeneralizes CTC to marginalize over n-gram output classes.\n\nThe Hidden Markov Model was developed in the 1960’s with the first application\nto speech recognition in the 1970’s. For an introduction to the HMM and\napplications to speech recognition see Rabiner’s canonical tutorial.\n\nEncoder-decoder models were developed in 2014.\n*Distill* has an\nin-depth guide to attention in encoder-decoder\nmodels.\n\nI’m especially grateful to the Distill team for *dramatically*\nimproving the quality of this article. Thanks to Chris Olah’s suggestions and\nfeedback, both the written and visual content of the article are\nsubstantially better. Thanks to Shan Carter for substantial improvements to\nthe figures, and thanks to Ludwig Schubert for help with the Distill\ntemplate.\n\nThanks to Sanjeev Satheesh, Chris Lengerich, Dan Jurafsky and the anonymous reviewers for their feedback. I’m also very grateful to Andrew Ng for feedback on the article and his support.\n\n[Review-1 Anonymous](https://github.com/distillpub/post--ctc/issues/13)\n\n[Review-2 Anonymous](https://github.com/distillpub/post--ctc/issues/14)\n\nIf you see mistakes or want to suggest changes, please [create an issue on GitHub](https://github.com/distillpub/post--ctc/issues/new).\n\nDiagrams and text are licensed under Creative Commons Attribution [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) with the [source available on GitHub](https://github.com/distillpub/post--ctc), unless noted otherwise. The figures that have been reused from other sources don’t fall under this license and can be recognized by a note in their caption: “Figure from …”.\n\nFor attribution in academic contexts, please cite this work as\n\n```\nHannun, \"Sequence Modeling with CTC\", Distill, 2017.\n```\n\nBibTeX citation\n\n```\n@article{hannun2017sequence,\n  author = {Hannun, Awni},\n  title = {Sequence Modeling with CTC},\n  journal = {Distill},\n  year = {2017},\n  note = {https://distill.pub/2017/ctc},\n  doi = {10.23915/distill.00008}\n}\n```\n\n", "url": "https://wpnews.pro/news/sequence-modeling-with-ctc", "canonical_source": "https://distill.pub/2017/ctc/", "published_at": "2026-06-27 08:23:53+00:00", "updated_at": "2026-06-27 08:35:35.493943+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks"], "entities": ["Connectionist Temporal Classification", "CTC"], "alternates": {"html": "https://wpnews.pro/news/sequence-modeling-with-ctc", "markdown": "https://wpnews.pro/news/sequence-modeling-with-ctc.md", "text": "https://wpnews.pro/news/sequence-modeling-with-ctc.txt", "jsonld": "https://wpnews.pro/news/sequence-modeling-with-ctc.jsonld"}}