Demystifying Conditional Random Fields (CRF) for NER: From Mathematical Elegance to Practical Implementation

A new technical guide published in May 2026 explains why Conditional Random Fields (CRF) remain essential for Named Entity Recognition (NER) tasks despite the rise of Large Language Models, citing their ability to enforce structural constraints, achieve low inference latency, and eliminate hallucinations in production systems. The article provides a mathematical breakdown of CRF as a discriminative undirected graphical model that models the conditional probability of entire label sequences, and includes a practical implementation walkthrough using the CRF++ toolkit. The guide aims to bridge the gap between theory and practice for developers building structured prediction systems in domains such as medical and legal text processing.

Demystifying Conditional Random Fields CRF for NER: From Mathematical Elegance to Practical Implementation May 2026 · 16 min read A deep dive into the underlying mathematics of Conditional Random Fields CRF , why they still matter in the age of LLMs, and a practical step-by-step guide to implementing sequence labeling using CRF++. Note: this is a combination of 2 old blog posts of me: https://blog.csdn.net/Felomeng/article/details/4288492 https://felomeng.blog.csdn.net/article/details/4367250 In the era of Generative AI and Large Language Models LLMs , it is easy to default to prompting a billion-parameter model for every Natural Language Processing NLP task. However, when it comes to structured prediction tasks like Named Entity Recognition NER , modern production systems often require structural constraints, low inference latency, and zero hallucination. This is where Conditional Random Fields CRF shine. As a discriminative undirected graphical model, CRF offers an elegant mathematical framework to model sequential dependencies. In this article, we will bridge the gap between theory and practice: exploring the core mathematics behind CRFs and walking through a hands-on implementation using the classic CRF++ toolkit. 1. Why CRF Still Matters in the Age of LLMs Standard classification models assume that data instances are independent and identically distributed i.i.d. . However, language is inherently sequential. In NER, predicting a token's label depends heavily on its neighbors. While a classic Softmax layer outputs the probability of each label independently, a CRF layer models the joint probability of the entire label sequence globally. The Trade-offs: LLMs vs. CRFs in Production Sequence Constraints: LLMs can fail to adhere to structural formats e.g., generating an I-PER tag without a preceding B-PER tag in BIO tagging . CRFs enforce strict transition constraints via a learned transition matrix. Efficiency: A CRF-based model can process thousands of sentences per second on a single CPU core, costing a fraction of an LLM API call. Deterministic Boundaries: For domain-specific NER e.g., medical or legal texts , CRFs offer explicit control over feature engineering, ensuring predictable and reliable boundaries. 2. The Mathematics Behind CRF CRF is a discriminative model that directly models the conditional probability $P \mathbf{y}|\mathbf{x} $, where $\mathbf{x}$ is the input sequence words and $\mathbf{y}$ is the output sequence labels . Given a sentence $\mathbf{x}$, the conditional probability of a label sequence $\mathbf{y}$ is defined as: $$P \mathbf{y}|\mathbf{x} = \frac{1}{Z \mathbf{x} } \exp \left \sum {i=1}^{n} \sum {j} \lambda j f j \mathbf{y} {i-1}, \mathbf{y} i, \mathbf{x}, i \right $$ Where: $f j \mathbf{y} {i-1}, \mathbf{y} i, \mathbf{x}, i $ is a user-defined feature function that scores the combination of the current label, the previous label, and the input sequence at position $i$. $\lambda j$ is the weight of the $j$-th feature function, learned during training. $Z \mathbf{x} $ is the Partition Function normalization factor that guarantees the probabilities over all possible label sequences sum up to 1: $$Z \mathbf{x} = \sum {\mathbf{y}'} \exp \left \sum {i=1}^{n} \sum {j} \lambda j f j \mathbf{y}' {i-1}, \mathbf{y}' i, \mathbf{x}, i \right $$ Feature Functions: The Core Mechanism CRF allows us to inject domain knowledge using two types of feature functions: State Features Transition from Input to State : $f y i, \mathbf{x}, i $ — e.g., "If the current word $x i$ is capitalized and ends with '-stein', how likely is $y i$ to be 'B-PER'?" Transition Features State to State : $f y {i-1}, y i, \mathbf{x}, i $ — e.g., "How likely is a 'B-PER' tag to be followed by an 'I-PER' tag?" Basic Usage of CRF++ 1. Downloading the Toolkit Linux Version with source code & Windows Version: You can download them from the Official CRF++ SourceForge Page http://crfpp.sourceforge.net . The Windows version does not require installation; it can be used directly via the command line after extraction. 2. Installation Steps on Linux In a Linux environment, after extracting the package and entering the directory, you need root privileges to execute the following commands in sequence: ./configure make su make install 3. Training Corpus Format Columns and Rows: The corpus must contain at least two columns. Columns are separated by spaces or tabs . Every row except for empty lines must have the exact same number of columns. Sentence Separation: Sentences are separated by an empty line . Example with two columns of features : 太 Sd N 短 Sa N 而 Bu N 已 Eu N 。 Sw N 4. Feature Selection and Template Writing CRF++ locates features using relative positions in the format of %x row, col both row and column indices start from 0 . 1. Feature Positioning Example Suppose the current row is the row for " 京 " Jing in "北京市" Beijing City : “ Sw N 北 Bns B-LOC 京 Mns I-LOC <-- Current Row 0 市 Ens I-LOC 首 Bn N %x -1,0 represents the 1st column of the previous row, which is " 北 ". %x 0,1 represents the 2nd column of the current row, which is " Mns ". %x -1,0 /%x 0,0 represents the combination of the 1st column of the previous row and the current row, which is " 北/京 ". 2. Creating Templates Templates are mainly divided into Unigram templates starting with U and Bigram templates starting with B . Note that "Uni/Bi" here refers to the combination of output tags, not the features themselves. Template File Example: Unigram U00:%x -2,0 U01:%x -1,0 U02:%x 0,0 Note: Rows starting with are comments and will be ignored by the system. 5. Training and Decoding Commands 1. Model Training Use the crf learn command to train your model: crf learn <template file <training corpus <generated model file Meanings of Training Output Parameters: iter : The current iteration number. terr : Tag Error Rate. serr : Sentence Error Rate. obj : The current value of the objective function. The training is complete when this value converges. 2. Model Prediction / Decoding Use the crf test command for prediction, and you can use the redirect operator to save the results to a file: crf test -m <model file <test file <output path Example: crf test -m model test.txt result.txt 6. Using the CoNLL 2000 Evaluation Tool You can use the CoNLL 2000 script to evaluate the model's Precision, Recall, and F1-score. Data Requirements: The test file needs to include the gold-standard answers. After decoding with crf test , the predicted results will be appended as the last column. The evaluation tool will then compare the second-to-last column the answer with the last column the prediction . Running Command: perl conlleval.pl < <evaluation file Note: Before using this evaluation tool, you must convert all tabs in the evaluation file into spaces, otherwise the tool may throw an error. Named Entity Recognition NER Using Conditional Random Fields CRF I. Experimental Environment a Software: Windows XP Pro SP3, Visual Studio 2008 & .NET 2005 Dotnet2.0 , CRF++, Perl b Hardware: CPU: CM420, RAM: 2GB DDR533, HDD: 160GB 8M SATA Fujitsu II. Experimental Process Unless specified otherwise, the following results are obtained by splitting the provided training corpus into a 7:3 ratio for training and evaluation according to the assignment requirements. a Direct Application of CRF The format of the provided corpus perfectly matches the requirements of Conditional Random Fields, so the CRF model is applied directly for training and testing. The files for this experiment are in the test1.rar package . Convert document encoding to UTF-8 CRF++ throws an error when using UTF-16 . Define the template as follows: Unigram U00:%x -2,0 U01:%x -1,0 U02:%x 0,0 U03:%x 1,0 U04:%x 2,0 U10:%x -1,0 /%x 0,0 U11:%x 0,0 /%x 1,0 Train and learn features using CRF++ Relevant information below : Command: crf learn template file train file model - Where template file is the template file and train file is the training corpus both need to be prepared in advance ; model is the file generated by CRF++ based on the template and training corpus, which is used for decoding. i. The template file Format - The basic format of a template is %x row,col , which is used to specify a token in the input data. determines the relative row offset from the current token. row determines the absolute column index. col Refer to the layout below | col 0 | col 1 | col 2 | || |---|---|---|---|---| row -2 | 疆 Jiang | Ens | I-LOC | | row -1 | 总 Zong | Bn | N | | row 0 | 统 Tong | En | N | Current Row | row 1 | 阿 A | Bns | B-PER | | row 2 | 利 Li | Mns | I-PER | | Template | Represented Feature | |---|---| U00:%x -2,0 | 疆 | U01:%x -1,0 | 总 | U02:%x 0,0 | 统 | U03:%x 1,0 | 阿 | U04:%x 2,0 | 利 | U10:%x -1,0 /%x 0,0 | 总/统 | U11:%x 0,0 /%x 1,0 | 统/阿 | Types of Feature Templates a Unigram Template: Starts with the letter U . When a template is prefixed with U , CRF++ automatically generates a set of feature functions. The total number of feature functions generated by a model is $L \times N$, where $L$ is the number of output classes and $N$ is the number of unique strings expanded based on the given template. b Bigram Template: Starts with the letter B . It is used to describe bigram features. The system will automatically generate combinations of the current output token and the previous output token. The total number of distinct features generated is $L \times L \times N$, where $L$ is the number of output classes and $N$ is the number of unique features produced by this template. c Difference Between the Two Templates: Note that Unigram/Bigram refers to the Unigram/Bigrams of the output tokens , not the features Unigram: $\lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$ Bigram: $\lvert\text{output tag}\rvert \times \lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$ b Training Log Sample: iter=88 terr=0.01365 serr=0.23876 obj=67066.17413 diff=0.00006 Where: iter is the number of iterations; terr is the token error rate; serr is the sentence error rate; obj is the current objective value training terminates when it converges ; diff is the relative change from the previous objective value. Done 2706.41 s Execution time on Computer 1 . Testing on the Test Corpus: a Command: crf test -m model file test file result file Where model file is the generated model file, test file is the corpus to be tested, and result file is the redirection statement to output the screen stream directly into result file . b The decoding speed of CRF++ is very fast, especially when writing directly to a file. However, due to feature selection issues, the precision and recall rates are not high. c The results are evaluated using the conlleval.pl script the code is located in the root directory of the submission package . The evaluation command is: perl conlleval.pl < output.txt , where output.txt is the file to be evaluated. A Perl interpreter is required. The detailed results are as follows: | Entity | Precision | Recall | FB1 | Tokens Count | | |---|---|---|---|---|---| LOC | 63.67% | 72.93% | 67.98 | 5623 | 382251.5 | ORG | 21.26% | 35.90% | 26.71 | 4491 | 119954.6 | PER | 65.90% | 65.06% | 65.47 | 2554 | 167210.4 | Macro Average | 53.39% | Micro Average | 52.84% | ii. Expanding the Feature Set Since very few features were selected previously, it was hypothesized that incorporating more valid features would improve performance. Thus, the template was updated as follows relevant data files for this experiment are in the test2.rar package : Template 2: Unigram U00:%x -2,0 U01:%x -1,0 U02:%x 0,0 U03:%x 1,0 U04:%x 2,0 U5:%x -2,0 /%x -1,0 U6:%x -1,0 /%x 0,0 U7:%x 0,0 /%x 1,0 U8:%x 1,0 /%x 2,0 Experimental Data: a Training Process: iter=94 terr=0.00571 serr=0.12313 obj=53321.45523 diff=0.00000 Done 2915.53 s b Test Results: | Entity | Precision | Recall | FB1 | Tokens Count | | |---|---|---|---|---|---| LOC | 66.86% | 74.31% | 70.39 | 5456 | 384047.8 | ORG | 26.95% | 41.02% | 32.53 | 4048 | 131681.4 | PER | 68.29% | 65.67% | 66.96 | 2488 | 166596.5 | Macro Average | 56.63% | Micro Average | 56.90% | Analysis: While there is noticeable improvement, the scores remain significantly low. b Rule-Based Post-Processing for Optimization i. Error Analysis By analyzing the errors detailed in the files starting with error in each package , the main errors can be categorized into the following types: - When characters within the same predicted entity have conflicting types, the type with the higher character frequency wins. If the counts are equal, it defaults to LOC in most cases. - The starting character of an entity must follow the B-??? format. - The boundary tokens start and end of entities follow specific patterns e.g., delimited by stop words, verbs, etc. . - Words directly following certain fixed entities should be in the B-??? format e.g., after province names . - Entities with tiny gaps between them might be merged into a single entity. - ...etc. ii. Optimization Results Based on these characteristics, I planned to test each rule sequentially to optimize the results. Due to time constraints, only four or five rules were evaluated. The first two rules Rule 1 and Rule 2 proved to be the most effective; combining them improved the performance by about 12% . Applying these corrections to the test2 outputs yielded: | Entity | Precision | Recall | FB1 | Tokens Count | | |---|---|---|---|---|---| LOC | 79.40% | 76.43% | 77.89 | 4966 | 386801.7 | ORG | 53.86% | 52.63% | 53.24 | 3457 | 184050.7 | PER | 80.88% | 67.09% | 73.34 | 2327 | 170662.2 | Macro Average | 68.16% | Micro Average | 68.98% | Analysis: Although the F-score $FB1$ increased dramatically, the overall performance is still not ideal. c Word Segmentation and POS Tagging Prior to CRF Learning i. Intent It became clear that focusing solely on character-level features was insufficient. Therefore, I attempted to leverage word segmentation and Part-of-Speech POS tagging information. Since the original task did not provide this data, a tool was used to segment and tag the text first the segmentation tool can be found in the root directory of the attachment package . ii. Feature Representation After word segmentation and tagging, the character features are structured as follows: | Character | POS & Segmentation Tag | Entity Label | |---|---|---| | ： | Sw | N | | 印 | Bns | B-LOC | | 度 | Ens | I-LOC | | 首 | Bd | N | | 先 | Ed | N | iii. Template Customization A new template was established specifically targeting these multi-column features. iv. Training and Testing Using the new template for training, the model was decoded and evaluated via conlleval , yielding the following results: iter=226 terr=0.00935 serr=0.17661 act=2913330 obj=42785.69115 diff=0.00009 Done 4502.97 s | Entity | Precision | Recall | FB1 | Tokens Count | | |---|---|---|---|---|---| LOC | 82.05% | 89.97% | 85.83 | 20309 | 1743121 | ORG | 48.36% | 65.12% | 55.50 | 13818 | 766899 | PER | 91.52% | 93.15% | 92.33 | 9189 | 848420.4 | Macro Average | 77.89% | Micro Average | 77.53% | v. Further Rule Optimization Applying the previously built post-processing rules to these new results brought the final performance to: | Entity | Precision | Recall | FB1 | Tokens Count | | |---|---|---|---|---|---| LOC | 90.34% | 90.37% | 90.36 | 18878 | 1705816 | ORG | 70.47% | 71.54% | 71.00 | 12474 | 885654 | PER | 94.85% | 92.70% | 93.76 | 8954 | 839527 | Macro Average | 85.04% | Micro Average | 85.12% | Based on this optimal setup, the model was trained on Test utf16.ner to finally generate finalAnswer.txt . III. Experimental Results Comparison Table | ID | Strategy Used | Result F-Score | Method Improvement | Performance Gain | Notes | |---|---|---|---|---|---| 1 | Character-based CRF 1 | ~53% | - | - | | 2 | Character-based CRF 2 | ~56.7% | Used richer feature context. | ~3.7% | Features strongly impact the outcome, but due to hardware and time limits, more features couldn't be added to verify. | 3 | Character CRF + Rules | ~68.5% | Manually added rules for post-processing. | ~11.8% | Rules successfully compensate for machine learning limits. Tried various rules and altered execution order . | 4 | Segmentation + POS + CRF | ~77.7% | Paradigm shift in feature representation. | ~9.2% | Introducing the concept of "words" is clearly effective. | 5 | Segmentation + POS + CRF + Rules | ~85.1% | Introduced rules on top of strategy 4. | ~7.4% | Certain drawbacks of ML methods do not change regardless of condition changes. | IV. Future Work a Explore additional rules to minimize the inherent flaws of pure machine learning methods. b Try treating word segmentation and POS tagging as completely separate attributes to observe their distinct impacts on the results. c Improve the accuracy of the baseline word segmentation and POS tagging tools to achieve better downstream NER performance. V. Key Precautions a Encoding formats can prevent certain files from being processed correctly; stay alert to formatting errors if crashes occur. b Different programs require different delimiters mostly spaces vs. tabs . Pay close attention to whether your file delimiters meet the program specifications. c The small utility scripts developed during the experiment do not include user manuals, but their interfaces are simple and clean, making them easy to master. Developed Tools Inventory Felomeng.BackFormation : Converts between the standard corpus format and the word segmentation/tagging format. It also includes functions to merge two types of tags or delete segmentation info. Felomeng.ErrorExtractor : An error extraction tool that pulls errors from output files containing ground truth labels to facilitate experimental analysis. Felomeng.NERRules : Originally featured four functions. Since the first three proved ineffective during testing, its primary function now is to optimize output via rule-based corrections on top of machine learning predictions. Postscript: In reality, the final performance is heavily dependent on how the training and testing datasets are partitioned. I adopted a strict split of the first 70% for training and the remaining 30% for testing. By subsequently refining the data selection methodology, the accuracy can surpass 92%. Anyone interested is encouraged to experiment with different ways of extracting the training and testing corpora. Was this article helpful?