Demystifying Conditional Random Fields (CRF) for NER: From Mathematical Elegance to Practical Implementation

May 2026 · 16 min read

A deep dive into the underlying mathematics of Conditional Random Fields (CRF), why they still matter in the age of LLMs, and a practical step-by-step guide to implementing sequence labeling using CRF++. Note: this is a combination of 2 old blog posts of me: https://blog.csdn.net/Felomeng/article/details/4288492 https://felomeng.blog.csdn.net/article/details/4367250

In the era of Generative AI and Large Language Models (LLMs), it is easy to default to prompting a billion-parameter model for every Natural Language Processing (NLP) task. However, when it comes to structured prediction tasks like Named Entity Recognition (NER), modern production systems often require structural constraints, low inference latency, and zero hallucination.

This is where Conditional Random Fields (CRF) shine. As a discriminative undirected graphical model, CRF offers an elegant mathematical framework to model sequential dependencies.

In this article, we will bridge the gap between theory and practice: exploring the core mathematics behind CRFs and walking through a hands-on implementation using the classic CRF++ toolkit.

1. Why CRF Still Matters in the Age of LLMs #

Standard classification models assume that data instances are independent and identically distributed (i.i.d.). However, language is inherently sequential. In NER, predicting a token's label depends heavily on its neighbors.

While a classic Softmax layer outputs the probability of each label independently, a CRF layer models the joint probability of the entire label sequence globally.

The Trade-offs: LLMs vs. CRFs in Production

Sequence Constraints: LLMs can fail to adhere to structural formats (e.g., generating anI-PER

tag without a precedingB-PER

tag in BIO tagging). CRFs enforce strict transition constraints via a learned transition matrix.Efficiency: A CRF-based model can process thousands of sentences per second on a single CPU core, costing a fraction of an LLM API call.Deterministic Boundaries: For domain-specific NER (e.g., medical or legal texts), CRFs offer explicit control over feature engineering, ensuring predictable and reliable boundaries.

2. The Mathematics Behind CRF #

CRF is a discriminative model that directly models the conditional probability $P(\mathbf{y}|\mathbf{x})$, where $\mathbf{x}$ is the input sequence (words) and $\mathbf{y}$ is the output sequence (labels).

Given a sentence $\mathbf{x}$, the conditional probability of a label sequence $\mathbf{y}$ is defined as:

$$P(\mathbf{y}|\mathbf{x}) = \frac{1}{Z(\mathbf{x})} \exp \left( \sum_{i=1}^{n} \sum_{j} \lambda_j f_j(\mathbf{y}_{i-1}, \mathbf{y}_i, \mathbf{x}, i) \right)$$

Where:

$f_j(\mathbf{y}_{i-1}, \mathbf{y}_i, \mathbf{x}, i)$ is a user-defined feature function that scores the combination of the current label, the previous label, and the input sequence at position $i$.$\lambda_j$ is the weight of the $j$-th feature function, learned during training.$Z(\mathbf{x})$ is thePartition Function(normalization factor) that guarantees the probabilities over all possible label sequences sum up to 1:

$$Z(\mathbf{x}) = \sum_{\mathbf{y}'} \exp \left( \sum_{i=1}^{n} \sum_{j} \lambda_j f_j(\mathbf{y}'_{i-1}, \mathbf{y}'_i, \mathbf{x}, i) \right)$$

Feature Functions: The Core Mechanism

CRF allows us to inject domain knowledge using two types of feature functions:

State Features (Transition from Input to State):$f(y_i, \mathbf{x}, i)$ —*e.g., "If the current word $x_i$ is capitalized and ends with '-stein', how likely is $y_i$ to be 'B-PER'?"*Transition Features (State to State):$f(y_{i-1}, y_i, \mathbf{x}, i)$ —e.g., "How likely is a 'B-PER' tag to be followed by an 'I-PER' tag?"

1. Down the Toolkit #

Linux Version (with source code) & Windows Version: You can download them from theOfficial CRF++ SourceForge Page. The Windows version does not require installation; it can be used directly via the command line after extraction.

2. Installation Steps on Linux #

In a Linux environment, after extracting the package and entering the directory, you need root

privileges to execute the following commands in sequence:

./configure
make
su
make install

3. Training Corpus Format #

Columns and Rows: The corpus must contain at least two columns. Columns are separated byspaces ortabs. Every row (except for empty lines) must have the exact same number of columns.** Sentence Separation:Sentences are separated by an empty line**.** Example (with two columns of features):**

太 Sd N
短 Sa N
而 Bu N
已 Eu N
。 Sw N

4. Feature Selection and Template Writing #

CRF++ locates features using relative positions in the format of %x[row, col]

(both row and column indices start from 0).

1. Feature Positioning Example

Suppose the current row is the row for "京" (Jing) in "北京市" (Beijing City):

“  Sw  N
北  Bns B-LOC
京  Mns I-LOC  <-- Current Row (0)
市  Ens I-LOC
首  Bn  N

%x[-1,0]

represents the 1st column of the previous row, which is "北".%x[0,1]

represents the 2nd column of the current row, which is "Mns".%x[-1,0]/%x[0,0]

represents the combination of the 1st column of the previous row and the current row, which is "北/京".

2. Creating Templates

Templates are mainly divided into Unigram

(templates starting with U) and Bigram

(templates starting with B). Note that "Uni/Bi" here refers to the combination of output tags, not the features themselves.

Template File Example:

U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]

Note: Rows starting with # are comments and will be ignored by the system.

5. Training and Decoding Commands #

1. Model Training

Use the crf_learn

command to train your model:

crf_learn <template_file> <training_corpus> <generated_model_file>

Meanings of Training Output Parameters:iter

: The current iteration number.terr

: Tag Error Rate.serr

: Sentence Error Rate.obj

: The current value of the objective function. The training is complete when this value converges.

2. Model Prediction / Decoding

Use the crf_test

command for prediction, and you can use the >

redirect operator to save the results to a file:

crf_test -m <model_file> <test_file> > <output_path>

*Example: crf_test -m model test.txt > result.txt*

6. Using the CoNLL 2000 Evaluation Tool #

You can use the CoNLL 2000 script to evaluate the model's Precision, Recall, and F1-score.

Data Requirements: The test file needs to include the gold-standard answers. After decoding withcrf_test

, the predicted results will be appended as the last column. The evaluation tool will then compare the second-to-last column (the answer) with the last column (the prediction).Running Command:

perl conlleval.pl < <evaluation_file>

Note: Before using this evaluation tool, you must convert all tabs in the evaluation file into spaces, otherwise the tool may throw an error.

I. Experimental Environment #

a) Software: Windows XP Pro SP3, Visual Studio 2008 & .NET 2005 (Dotnet2.0), CRF++, Perlb) Hardware: CPU: CM420, RAM: 2GB DDR533, HDD: 160GB 8M SATA Fujitsu

II. Experimental Process #

Unless specified otherwise, the following results are obtained by splitting the provided training corpus into a 7:3 ratio for training and evaluation according to the assignment requirements.

a) Direct Application of CRF

The format of the provided corpus perfectly matches the requirements of Conditional Random Fields, so the CRF model is applied directly for training and testing. (The files for this experiment are in the test1.rar

package).

Convert document encoding to UTF-8(CRF++ throws an error when using UTF-16).** Define the template**as follows:

U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]
U03:%x[1,0]
U04:%x[2,0]
U10:%x[-1,0]/%x[0,0]
U11:%x[0,0]/%x[1,0]

Train and learn features using CRF++(Relevant information below):

Command:crf_learn template_file train_file model

• Where template_file

is the template file andtrain_file

is the training corpus (both need to be prepared in advance);model

is the file generated by CRF++ based on the template and training corpus, which is used for decoding.

i. The template_file

Format

• The basic format of a template is %x[row,col]

, which is used to specify a token in the input data.

determines the relative row offset from the current token.row

determines the absolute column index.col

(Refer to the layout below)

col 0	col 1	col 2
row -2
疆 (Jiang)	Ens	I-LOC
row -1
总 (Zong)	Bn	N
row 0
统 (Tong)	En	N	Current Row
row 1
阿 (A)	Bns	B-PER
row 2
利 (Li)	Mns	I-PER

Template	Represented Feature
U00:%x[-2,0]
疆
U01:%x[-1,0]
总
U02:%x[0,0]
统
U03:%x[1,0]
阿
U04:%x[2,0]
利
U10:%x[-1,0]/%x[0,0]
总/统
U11:%x[0,0]/%x[1,0]
统/阿

Types of Feature Templates

a) Unigram Template: Starts with the letterU

. When a template is prefixed withU

, CRF++ automatically generates a set of feature functions. The total number of feature functions generated by a model is $L \times N$, where $L$ is the number of output classes and $N$ is the number of unique strings expanded based on the given template.b) Bigram Template: Starts with the letterB

. It is used to describe bigram features. The system will automatically generate combinations of the current output token and the previous output token. The total number of distinct features generated is $L \times L \times N$, where $L$ is the number of output classes and $N$ is the number of unique features produced by this template.c) Difference Between the Two Templates: Note that Unigram/Bigram refers to the Unigram/Bigrams of theoutput tokens, not the features!Unigram:$\lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$** Bigram:**$\lvert\text{output tag}\rvert \times \lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$b) Training Log Sample:iter=88 terr=0.01365 serr=0.23876 obj=67066.17413 diff=0.00006

Where:iter

is the number of iterations;terr

is the token error rate;serr

is the sentence error rate;obj

is the current objective value (training terminates when it converges);diff

is the relative change from the previous objective value.

Done! 2706.41 s(Execution time on Computer 1).** Testing on the Test Corpus:**

a) Command:crf_test -m model_file test_file > result_file

Wheremodel_file

is the generated model file,test_file

is the corpus to be tested, and> result_file

is the redirection statement to output the screen stream directly intoresult_file

.b) The decoding speed of CRF++ is very fast, especially when writing directly to a file. However, due to feature selection issues, the precision and recall rates are not high.c) The results are evaluated using theconlleval.pl

script (the code is located in the root directory of the submission package). The evaluation command is:perl conlleval.pl < output.txt

, whereoutput.txt

is the file to be evaluated. A Perl interpreter is required. The detailed results are as follows:

Entity	Precision	Recall	FB1	Tokens Count
LOC
63.67%	72.93%	67.98	5623	382251.5
ORG
21.26%	35.90%	26.71	4491	119954.6
PER
65.90%	65.06%	65.47	2554	167210.4
Macro Average
53.39%
Micro Average
52.84%

ii. Expanding the Feature Set

Since very few features were selected previously, it was hypothesized that incorporating more valid features would improve performance. Thus, the template was updated as follows (relevant data files for this experiment are in the test2.rar

package):

Template 2:

U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]
U03:%x[1,0]
U04:%x[2,0]
U5:%x[-2,0]/%x[-1,0]
U6:%x[-1,0]/%x[0,0]
U7:%x[0,0]/%x[1,0]
U8:%x[1,0]/%x[2,0]

Experimental Data:

a) Training Process:iter=94 terr=0.00571 serr=0.12313 obj=53321.45523 diff=0.00000

Done! 2915.53 s

b) Test Results:

Entity	Precision	Recall	FB1	Tokens Count
LOC
66.86%	74.31%	70.39	5456	384047.8
ORG
26.95%	41.02%	32.53	4048	131681.4
PER
68.29%	65.67%	66.96	2488	166596.5
Macro Average
56.63%
Micro Average
56.90%

Analysis: While there is noticeable improvement, the scores remain significantly low.

b) Rule-Based Post-Processing for Optimization

i. Error Analysis

By analyzing the errors (detailed in the files starting with error

in each package), the main errors can be categorized into the following types:

• When characters within the same predicted entity have conflicting types, the type with the higher character frequency wins. If the counts are equal, it defaults to LOC

in most cases. - The starting character of an entity must follow the B-???

format. - The boundary tokens (start and end) of entities follow specific patterns (e.g., delimited by stop words, verbs, etc.).

• Words directly following certain fixed entities should be in the B-???

format (e.g., after province names). - Entities with tiny gaps between them might be merged into a single entity.

• ...etc.

ii. Optimization Results

Based on these characteristics, I planned to test each rule sequentially to optimize the results. Due to time constraints, only four or five rules were evaluated. The first two rules (Rule 1 and Rule 2) proved to be the most effective; combining them improved the performance by about 12%. Applying these corrections to the test2

outputs yielded:

Entity	Precision	Recall	FB1	Tokens Count
LOC
79.40%	76.43%	77.89	4966	386801.7
ORG
53.86%	52.63%	53.24	3457	184050.7
PER
80.88%	67.09%	73.34	2327	170662.2
Macro Average
68.16%
Micro Average
68.98%

Analysis: Although the F-score ($FB1$) increased dramatically, the overall performance is still not ideal.

c) Word Segmentation and POS Tagging Prior to CRF Learning

i. Intent

It became clear that focusing solely on character-level features was insufficient. Therefore, I attempted to leverage word segmentation and Part-of-Speech (POS) tagging information. Since the original task did not provide this data, a tool was used to segment and tag the text first (the segmentation tool can be found in the root directory of the attachment package).

ii. Feature Representation

After word segmentation and tagging, the character features are structured as follows:

Character	POS & Segmentation Tag	Entity Label
：	Sw	N
印	Bns	B-LOC
度	Ens	I-LOC
首	Bd	N
先	Ed	N

iii. Template Customization

A new template was established specifically targeting these multi-column features.

iv. Training and Testing

Using the new template for training, the model was decoded and evaluated via conlleval

, yielding the following results: iter=226 terr=0.00935 serr=0.17661 act=2913330 obj=42785.69115 diff=0.00009

Done! 4502.97 s

Entity	Precision	Recall	FB1	Tokens Count
LOC
82.05%	89.97%	85.83	20309	1743121
ORG
48.36%	65.12%	55.50	13818	766899
PER
91.52%	93.15%	92.33	9189	848420.4
Macro Average
77.89%
Micro Average
77.53%

v. Further Rule Optimization

Applying the previously built post-processing rules to these new results brought the final performance to:

Entity	Precision	Recall	FB1	Tokens Count
LOC
90.34%	90.37%	90.36	18878	1705816
ORG
70.47%	71.54%	71.00	12474	885654
PER
94.85%	92.70%	93.76	8954	839527
Macro Average
85.04%
Micro Average
85.12%

Based on this optimal setup, the model was trained on Test_utf16.ner

to finally generate finalAnswer.txt

.

III. Experimental Results Comparison Table #

ID	Strategy Used	Result (F-Score)	Method Improvement	Performance Gain	Notes
1
Character-based CRF (1)	~53%	-	-
2
Character-based CRF (2)	~56.7%	Used richer feature context.	~3.7%	Features strongly impact the outcome, but due to hardware and time limits, more features couldn't be added to verify.
3
Character CRF + Rules	~68.5%	Manually added rules for post-processing.	~11.8%	Rules successfully compensate for machine learning limits. Tried various rules (and altered execution order).
4
Segmentation + POS + CRF	~77.7%	Paradigm shift in feature representation.	~9.2%	Introducing the concept of "words" is clearly effective.
5
Segmentation + POS + CRF + Rules	~85.1%	Introduced rules on top of strategy 4.	~7.4%	Certain drawbacks of ML methods do not change regardless of condition changes.

IV. Future Work #

a) Explore additional rules to minimize the inherent flaws of pure machine learning methods.b) Try treating word segmentation and POS tagging as completely separate attributes to observe their distinct impacts on the results.c) Improve the accuracy of the baseline word segmentation and POS tagging tools to achieve better downstream NER performance.

V. Key Precautions #

a) Encoding formats can prevent certain files from being processed correctly; stay alert to formatting errors if crashes occur.b) Different programs require different delimiters (mostly spaces vs. tabs). Pay close attention to whether your file delimiters meet the program specifications.c) The small utility scripts developed during the experiment do not include user manuals, but their interfaces are simple and clean, making them easy to master.

Developed Tools Inventory

Felomeng.BackFormation

: Converts between the standard corpus format and the word segmentation/tagging format. It also includes functions to merge two types of tags or delete segmentation info.Felomeng.ErrorExtractor

: An error extraction tool that pulls errors from output files (containing ground truth labels) to facilitate experimental analysis.Felomeng.NERRules

: Originally featured four functions. Since the first three proved ineffective during testing, its primary function now is to optimize output via rule-based corrections on top of machine learning predictions.

Postscript: In reality, the final performance is heavily dependent on how the training and testing datasets are partitioned. I adopted a strict split of the first 70% for training and the remaining 30% for testing. By subsequently refining the data selection methodology, the accuracy can surpass 92%. Anyone interested is encouraged to experiment with different ways of extracting the training and testing corpora.

Was this article helpful?

source & further reading

noahhan.com — original article I built a fast, edge-native personal site with Astro, Cloudflare D1, and R2 Preface to the “Foundations and Frontiers of Large Language Models” Series Rapid ML Prototyping: Re-evaluating Weka for Classic Classification Tasks