Mathematical formulas for evaluating entity association, semantic embedding similarity, ordinal rank positioning, and generation performance in AI models.
To evaluate how strongly a model associates an entity $E$ (e.g., CrediBridge) with a vertical or topic concept $C$ (e.g., mortgage, insurance) in parametric memory without retrieval:
Given an evaluation prompt template $X = (x_1, x_2, \dots, x_n)$ designed to elicit the entity, where entity $E$ spans tokens $(t_1, t_2, \dots, t_m)$:
$$\log P(E \mid X) = \sum_{j=1}^{m} \log P\left(t_j \mid X, t_1, \dots, t_{j-1}\right)$$
For multi-token entities of differing lengths, compute the **Length-Normalized Association Log-Likelihood**:
$$\mathcal{S}_{\text{assoc}}(E \mid X) = \frac{1}{m} \sum_{j=1}^{m} \log P\left(t_j \mid X, t_1, \dots, t_{j-1}\right)$$
To compare entity $E$ against a candidate set of competing brands $\mathcal{E} = {E_1, E_2, \dots, E_K}$, the Softmax Choice Probability is:
$$P(E_i \mid X) = \frac{\exp\left(\mathcal{S}{\text{assoc}}(E_i \mid X) / \tau\right)}{\sum{k=1}^{K} \exp\left(\mathcal{S}_{\text{assoc}}(E_k \mid X) / \tau\right)}$$ (where $\tau$ is the temperature parameter).
Let $\mathbf{v}_E \in \mathbb{R}^d$ be the dense embedding vector of the entity name/description, and $\mathbf{v}_C \in \mathbb{R}^d$ be the centroid vector of the target concept, computed over a set of $N$ descriptive phrases ${p_1, \dots, p_N}$ defining that vertical:
$$\mathbf{v}_C = \frac{1}{N} \sum_{k=1}^{N} \mathbf{e}(p_k)$$
The Direct Cosine Association is:
$$\text{Sim}(E, C) = \frac{\mathbf{v}_E \cdot \mathbf{v}_C}{|\mathbf{v}_E|_2 , |\mathbf{v}_C|_2}$$
To eliminate generic brand popularity bias against a baseline topic $C_{\text{base}}$:
$$\Delta \text{Sim}(E; C, C_{\text{base}}) = \frac{\mathbf{v}_E \cdot \mathbf{v}_C}{|\mathbf{v}_E|_2 |\mathbf{v}_C|2} - \frac{\mathbf{v}E \cdot \mathbf{v}{C{\text{base}}}}{|\mathbf{v}E|2 |\mathbf{v}{C{\text{base}}}|_2}$$
Let a generated response contain an ordered list or sequential mention of entities:
$$\mathbf{r}_{\text{gen}} = (e_{(1)}, e_{(2)}, \dots, e_{(M)})$$
The ordinal rank of entity $E$ in the generated output is:
$$\text{Rank}_{\text{gen}}(E) = \begin{cases} k & \text{if } e_{(k)} = E \\ M + 1 & \text{if } E \notin \mathbf{r}_{\text{gen}} \end{cases}$$
Let $\text{Rank}_{\text{param}}(E) \in [1, K]$ be the ungrounded parametric baseline rank, and $\text{Rank}_{\text{ground}}(E) \in [1, L]$ be the ordinal rank of entity $E$ in retrieved sources. The **Ordinal Delta** ($\Delta R$) is:
$$\Delta R(E) = \text{Rank}_{\text{param}}(E) - \text{Rank}_{\text{gen}}(E)$$
To model the composite probability of entity $E$ being generated at rank position 1 given parametric score $\mathcal{S}_{\text{param}}(E)$ and retrieval score $\mathcal{S}_{\text{ground}}(E)$:
$$P\left(\text{Rank}(E) = 1 \mid X, D_{\text{retrieved}}\right) = \sigma\left(\alpha \cdot \mathcal{S}{\text{param}}(E) + \beta \cdot \mathcal{S}{\text{ground}}(E) + \gamma \left(\mathcal{S}{\text{param}}(E) \cdot \mathcal{S}{\text{ground}}(E)\right) + c\right)$$
Across a test benchmark of $Q$ queries within a vertical: