# How Activation Functions Shape an AI Assistant > Source: > Published: 2026-11-11 12:01:01+00:00 When we work on building an AI assistant, we think of model names, parameters, embeddings, fine-tuning and other areas. Before arrive in those technical steps, we may have important design questions also for such a system. One of such questions is — **What are the mathematical part of work - does the AI assistant need to make?** This particular question can affect the changes of the default behavior of the Transformer Model in our work. I have tried to discuss some of the details of choicing activation functions, which can eventually help us to make design of such an application. I tried to explain different business scenarios for the choices. During my writing, I have used AI Tools to generate the primary block diagrams for activation functions and their association with different Language Model specific mathematical calculation stages. I have edited those diagrams in [drawio editor](https://app.diagrams.net/) to maintain relevancy with my writing. For an assitant, we may think some sample questions / clarifications - ``` Is this expense allowed?Which department should handle this?What is our leave policy?Compare Plan A and Plan B.What should I do next? ``` The above questions / clarifications are having type of yes/no decision , ranking from high to low numbering, some of them require generating explanations. Other questions may require selecting option from many (multiple choice) and also others may require having multiple answers (checkbox type) at the same time. Here **activation functions** are having role in play. Activation functions are formulas inside a neural network and they are part of model decisions, such as - **What information should move forward, what should be suppressed, what should be treated as uncertain and other areas.** If we make a very small Transformer written in Python, It contains following steps - ``` token embeddingspositional encodingself-attentionfeed-forward layersresidual connectionsnormalizationoutput layers ``` In [Github](https://github.com/piyasde/article_companion/blob/main/mini_transformer.py), a mini transformer is commited. During work discovery phase, we should discuss with work stakeholders with their expectation of assistant behaior and identify a broad set of questions to set the boundary for the assistant. Here the starting point should not start with terminologies like R*eLU**,* *GELU**, or* *SwiGLU**.* Rather we should identify the *mathematical operations* which will cover the operations of the Assistant. For example, the categorisation may be - The above illustration can help us to make the transformer architecture or model selection with probable required activation functions. Useful distinctions for activation functions categories - Flow of control for **Internal activation functions** — **Inside Transformer → How should internal features be transformed? -> ReLU / GELU / SiLU / SwiGLU** Flow of control for **Business-output activation functions -** **Business Output -> What kind of answer is required? -> Sigmoid / Softmax** Some code snippets are for sigmoid, relu, gelu and silu, which are utilities in python codebase - ``` python import mathINPUT_NUMBERS = [-3.0, -2.0, -1.0, -0.5, 0.0, 0.5, 1.0, 2.0, 3.0]def sigmoid(number): if number >= 0.0: return 1.0 / (1.0 + math.exp(-number)) positive_exponent = math.exp(number) return positive_exponent / (1.0 + positive_exponent)def relu(number): return max(0.0, number)def gelu(number): normal_cdf = 0.5 * (1.0 + math.erf(number / math.sqrt(2.0))) return number * normal_cdfdef silu(number): return number * sigmoid(number)for x in INPUT_NUMBERS: print(f"{x:8.2f} {relu(x):10.4f} {gelu(x):10.4f} " f"{silu(x):10.4f} {sigmoid(x):10.4f}") ``` The full python codebase can be found in [Github](https://github.com/piyasde/article_companion). The output should be self explanatory - ``` ReLU(x)=max(0,x) | GELU(x)=x*NormalCDF(x) | SiLU(x)=x*sigmoid(x)Sigmoid(x)=1/(1+exp(-x)) INPUT ReLU GELU SiLU Sigmoid -3.00 0.0000 -0.0040 -0.1423 0.0474 -2.00 0.0000 -0.0455 -0.2384 0.1192 -1.00 0.0000 -0.1587 -0.2689 0.2689 -0.50 0.0000 -0.1543 -0.1888 0.3775 0.00 0.0000 0.0000 0.0000 0.5000 0.50 0.5000 0.3457 0.3112 0.6225 1.00 1.0000 0.8413 0.7311 0.7311 2.00 2.0000 1.9545 1.7616 0.8808 3.00 3.0000 2.9960 2.8577 0.9526 ``` For a question such as — ***Can an employee carry forward unused leave? —*** the transformer builds an internal representation. It forms hidden features like — ``` employee policy = strongleave = very strongcarry forward = very strongpayroll = weaktermination = weakexception rule = mediumeligibility = strong ``` We can think the hidden representations as **h.** The same representation can be used for different business purposes - ``` ┌→ Yes/No head │Transformer → h ────┼→ Intent head │ ├→ Retrieval score │ └→ Language generation ``` Here the Transformer creates reusable contextual features. The business heads decide what those features should become. A minimal Transformer core steps are like the following pipeline - **Tokens** → **Embedding -> Positional Information -> Self-Attention -> Residual + Normalization -> Feed-Forward Network -> Residual + Normalization -> Output Representation.** Here Activation-related decisions are taken in five places - ``` attention Softmaxfeed-forward activationoptional gatingtask-specific output activationfinal vocabulary Softmax ``` The attention block calculates - ``` Q = XWQK = XWKV = XWV ``` Then **AttentionScores = QK^T / sqrt(dk)** Then **AttentionWeights = Softmax(AttentionScores)** Then **Output = AttentionWeights × V** As attention is a competition, the model is effectively asking — *Among all these tokens, where should I distribute my attention?* If we have an information — t***he employee submitted the invoice because it was overdue. — f*** or the word “**it**,” attention may produce some score like - ``` invoice → 0.72employee → 0.09submitted → 0.07overdue → 0.12 ``` These values add to approximately 1 which is a probability distribution. Here the Softmax function is used to convert a vector of raw scores — logits — into a probability distribution. Here is a step by step illustrative attention function for understanding the work (can be optimized in production system with **numpy** and other useful python libraries) — ``` python def show_attention(prompt, temperature): words = tokenize(prompt) input_vectors = embedded_inputs(words) queries, keys, values = project_queries_keys_values(input_vectors) causal_weights, causal_contexts, scores = compute_attention( queries, keys, values, temperature=temperature, causal=True ) unmasked_weights, _, _ = compute_attention( queries, keys, values, temperature=temperature, causal=False ) print("Token Embedding + Position") for token_index, word in enumerate(words): print(f" {token_index}. {word:>8} : {format_vector(input_vectors[token_index])}") print("Query, Key, Value (QKV)") for token_index, word in enumerate(words): print(f" {word}: Q={format_vector(queries[token_index])}") print(f" {' ' * len(word)} K={format_vector(keys[token_index])}") print(f" {' ' * len(word)} V={format_vector(values[token_index])}") print("Dot-Product Similarity") heading = "current\\candidate".ljust(20) + "".join(w.rjust(12) for w in words) for position_index, word in enumerate(words): numbers = "".join(f"{score:12.4f}" for score in scores[position_index]) print(word.ljust(20) + numbers) if len(words) >= 2: current = len(words) - 1 candidate = 0 raw_dot = dot_product(queries[current], keys[candidate]) divisor = math.sqrt(len(queries[current])) print(f"\nWorked example: last token '{words[current]}' vs first '{words[candidate]}'") print(f" Q dot K = {raw_dot:.5f}; sqrt(width) = {divisor:.5f}") print(f" scaled score = {raw_dot / divisor:.5f}") print("Causal Mask + Softmax") print(heading) for position_index, word in enumerate(words): numbers = "".join(f"{p:12.4f}" for p in causal_weights[position_index]) print(word.ljust(20) + numbers) print("Unmasked Attention") print(heading) for position_index, word in enumerate(words): numbers = "".join(f"{p:12.4f}" for p in unmasked_weights[position_index]) print(word.ljust(20) + numbers) print("Weighted Value Combination") for position_index, word in enumerate(words): print(f" Context for '{word}' = {format_vector(causal_contexts[position_index])}") print("\nThe attention weights come from Softmax; the context is the") print("weighted average/sum of V vectors.") ``` This whole program and related utility functions are in Transformer program in [Github Repository](https://github.com/piyasde/article_companion). Following are the operations and their works in each step - Attention answers — *Where should I look?* The feed-forward network answers to — *What should I do with the information I have now?* A simple feed-forward block is — **FFN(x) = W2(ReLU(W1x + b1)) + b2** Here we can replace **ReLU** with **GELU**, **SiLU**, or redesign the block using **SwiGLU**. This is where we should do activation-level experimentation with FFN. Next, we will discuss the Liner Uinits — Activation functions with their process in diagarm with related codes. Overall Codes are in [Github](https://github.com/piyasde/article_companion/blob/main/transformer_comparison.py). In designing the HR Assistant, some of the questions are - **How many casual leaves are available? Can leave be carried forward?Who approves travel reimbursement?What is the notice period?** These questions are mostly **factual, narrow, structured, relatively low in ambiguity.** In such a case, **ReLU** may be a reasonable baseline. **ReLU** is — **ReLU(x) = max(0, x).** It will be like - ``` leave_policy = +3.2termination = -1.4salary_revision = -0.8 ``` After **ReLU** it becomes - ``` leave_policy = 3.2termination = 0salary_revision = 0 ``` The intuition is simple — Only k*eep useful positive features. Suppress the rest.* For a low-cost system, this may be enough, though it should be tested and not only assumed. In the code excercise, we have tried with ‘**cats drink**’ with a target of ‘** milk**’. Here are some hints on the execution path in the [code](https://github.com/piyasde/article_companion/blob/main/transformer_comparison.py), which we should study in [github repository](https://github.com/piyasde/article_companion) - ``` python from mini_transformer import ( ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, run_transformer,)....EXERCISES = [ ("cats drink", "milk"), ("dogs drink", "water"), ("I like", "tea"),]....def cross_entropy_for_expected_word(probability): """For a one-word target, loss = -ln(probability assigned to the target).""" return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature): .... for activation_name in activation_names: result = run_transformer(prompt, activation_name, temperature) probabilities = result["next_word_probabilities"] top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i]) target_index = NEXT_WORDS.index(target) p_target = probabilities[target_index] loss = cross_entropy_for_expected_word(p_target) parameter_count = count_feed_forward_parameters(activation_name) formatted_probabilities = " ".join(f"{number:.3f}" for number in probabilities)....def relu(number): return max(0.0, number)....def run_transformer(prompt, activation_name="relu", temperature=1.0): .... final_vectors = [] for token_vector in after_attention: details = feed_forward(token_vector, activation_name) feed_forward_details.append(details) residual_sum = vector_add(token_vector, details["output"]) final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name): .... for unit_index, first_branch_value in enumerate(input_pre_activations): if activation_name == "relu": activated = relu(first_branch_value) .... activated_units.append(activated) print("Activation units are - ",activated_units) output_vector = linear( activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS ) ``` The output should come as below in console - ``` Activation FFN params Top word P(target) loss P(milk) P(water) P(tea) P(coffee)--------------------------------------------------------------------------------------------------------token vectorcats [0.9, -0.4, 0.7, 0.2]position vectorcats [0.0, -0.0, 0.0, -0.0]token vectordrink [-0.2, 0.8, 0.3, 0.5]position vectordrink [0.08, -0.04, 0.03, -0.06]Activation units are - [2.891142495987482, 0.0, 0.0, 0.0, 0.0, 2.0732335436635374]Activation units are - [0.0, 1.981098876135741, 1.3766495329967907, 1.085066439895106, 0.0, 0.0]relu 58 coffee 0.0441 3.1221 0.044 0.183 0.323 0.450 ``` Here **loss = cross_entropy_for_expected_word(p_target)** is important factor. The loss is — 3.1221. Here the loss function measures how much probability the model assigned to that correct answer. For categorical cross-entropy with one correct target **Loss=-ln(P(correct word))** is equal to — **P(milk)=0.0441,** comes as **-ln(0.0441)** which is approx **3.122.** This is a relatively high loss because the model assigned low probability to the correct word. Here the model produces probabilities, and the loss is calculated afterward with the known target in evaluation or training phase. Here important part is at looking in the activation units, where some of them are 0.0 which is the outcome of **ReLU**. Again during designing the HR Assistant, some of the questions may be - **I have been away for three days. Can I still regularize? My manager said it is okay, but HR rejected it. What happens now?If someone resigns midway through the year, what happens to unused leave?** These questions contain — **ambiguity, context. exceptions, weak but possibly useful signals** etc. Here **GELU** can be be a fair choice as **GELU** does not make the zero or as-it-is decision as **ReLU**. Here **GELU** will keep a small transformed value instead. The intuition is — ***This signal is weak, but do not destroy it at early stages.*** That can be helpful in language because meaning emerges from many weak clues at a later training cycle. As above code explanation of RELU, here are some useful code snippet for GELU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion) ``` python from mini_transformer import ( ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, run_transformer,)....EXERCISES = [ ("cats drink", "milk"), ("dogs drink", "water"), ("I like", "tea"),]....def cross_entropy_for_expected_word(probability): """For a one-word target, loss = -ln(probability assigned to the target).""" return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature): .... for activation_name in activation_names: result = run_transformer(prompt, activation_name, temperature) probabilities = result["next_word_probabilities"] top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i]) target_index = NEXT_WORDS.index(target) p_target = probabilities[target_index] loss = cross_entropy_for_expected_word(p_target) parameter_count = count_feed_forward_parameters(activation_name) formatted_probabilities = " ".join(f"{number:.3f}" for number in probabilities)....def gelu(number): normal_cdf = 0.5 * (1.0 + math.erf(number / math.sqrt(2.0))) return number * normal_cdf....def run_transformer(prompt, activation_name="relu", temperature=1.0): .... final_vectors = [] for token_vector in after_attention: details = feed_forward(token_vector, activation_name) feed_forward_details.append(details) residual_sum = vector_add(token_vector, details["output"]) final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name): .... for unit_index, first_branch_value in enumerate(input_pre_activations): .... elif activation_name == "gelu": activated = gelu(first_branch_value) .... activated_units.append(activated) print("Activation units are - ",activated_units) output_vector = linear( activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS ) ``` Here for above excercise dataset about (“cats drink”, “milk”), activation units comes closely to - ``` Activation units are - [2.8855937569813706, -0.13352579602658668, -0.04908918220790799, -0.16690316151871892, -0.13620332184690795, 2.0336860436188022]Activation units are - [-0.014077301889820216, 1.933968349321836, 1.260583778083766, 0.9343006736195222, -0.167826258210203, -0.1499369332953556] ``` The nature of scoring is non-zero values even for non-important informations. SiLU is a smooth activation. It is — **SiLU(x) = x × sigmoid(x).** This gives an useful information — **signal × how strongly the signal allows itself through.** If the raw signal is strong and positive, it passes more weightage to next cycle. If the signal is weak or negative, it is reduced smoothly and pass less weightage in next cycle. This can make optimization smoother and can work for specific cases in neural architectures. As above code explanation , here are some useful code snippet for SILU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion) ``` python from mini_transformer import ( ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, run_transformer,)....EXERCISES = [ ("cats drink", "milk"), ("dogs drink", "water"), ("I like", "tea"),]....def cross_entropy_for_expected_word(probability): """For a one-word target, loss = -ln(probability assigned to the target).""" return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature): .... for activation_name in activation_names: result = run_transformer(prompt, activation_name, temperature) probabilities = result["next_word_probabilities"] top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i]) target_index = NEXT_WORDS.index(target) p_target = probabilities[target_index] loss = cross_entropy_for_expected_word(p_target) parameter_count = count_feed_forward_parameters(activation_name) formatted_probabilities = " ".join(f"{number:.3f}" for number in probabilities)....def silu(number): return number * sigmoid(number)....def run_transformer(prompt, activation_name="relu", temperature=1.0): .... final_vectors = [] for token_vector in after_attention: details = feed_forward(token_vector, activation_name) feed_forward_details.append(details) residual_sum = vector_add(token_vector, details["output"]) final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name): .... for unit_index, first_branch_value in enumerate(input_pre_activations): .... elif activation_name == "silu": activated = silu(first_branch_value) .... activated_units.append(activated) print("Activation units are - ",activated_units) output_vector = linear( activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS ) Activation units are - [2.7390881702111454, -0.2782831534600395, -0.2420876982940121, -0.22024136849212178, -0.15802568867005076, 1.8416000155094343]Activation units are - [-0.18567423752791756, 1.740986233750263, 1.0991890705587457, 0.811034605518993, -0.2238910250978735, -0.2744103198613716] ``` The nature of scoring is more weightage for stronger signal. **GLU** comes with an idea. Here instead of asking only — ***“What feature exists?”,*** the network also asks — *“Should this feature matter here?”* We create two paths. One path produces information — **A = xWA**. Another path produces a control gate **G = xWG**. Then **Output = A × sigmoid(G).** This can be understood as - **A = what information did we find? G = how much should we use it now?** As above code explanation , here are some useful code snippet for GLU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion) ``` python from mini_transformer import ( ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, run_transformer,)....EXERCISES = [ ("cats drink", "milk"), ("dogs drink", "water"), ("I like", "tea"),]....def cross_entropy_for_expected_word(probability): """For a one-word target, loss = -ln(probability assigned to the target).""" return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature): .... for activation_name in activation_names: result = run_transformer(prompt, activation_name, temperature) probabilities = result["next_word_probabilities"] top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i]) target_index = NEXT_WORDS.index(target) p_target = probabilities[target_index] loss = cross_entropy_for_expected_word(p_target) parameter_count = count_feed_forward_parameters(activation_name) formatted_probabilities = " ".join(f"{number:.3f}" for number in probabilities)....def silu(number): return number * sigmoid(number)def sigmoid(number): if number >= 0.0: return 1.0 / (1.0 + math.exp(-number)) positive_exponent = math.exp(number) return positive_exponent / (1.0 + positive_exponent)....def run_transformer(prompt, activation_name="relu", temperature=1.0): .... final_vectors = [] for token_vector in after_attention: details = feed_forward(token_vector, activation_name) feed_forward_details.append(details) residual_sum = vector_add(token_vector, details["output"]) final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name): .... if activation_name in ("glu", ....): gate_pre_activations = linear(input_vector, GATE_WEIGHTS, GATE_BIAS) .... for unit_index, first_branch_value in enumerate(input_pre_activations): .... elif activation_name == "glu": gate = sigmoid(gate_pre_activations[unit_index]) activated = first_branch_value * gate .... activated_units.append(activated) print("Activation units are - ",activated_units) output_vector = linear( activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS ) Activation units are - [2.1293380942953477, -0.5606983585437584, -1.4670984863133165, -0.35918495390583977, -0.16691853734558215, 0.716106547145415]Activation units are - [-0.5803804656519378, 1.4659660543453696, 0.42819269347992506, 0.30929164364610134, -0.3492711723798345, -0.8514748949632387] ``` **SwiGLU** extends the activation idea using **SiLU**. It is — **SwiGLU(x) = SiLU(xWg) × (xWu).** After above operation, a final projection brings the representation back to the model dimension. This becomes useful when the customer domain has many **overlapping concepts, multiple meanings, complex context, exception rules, long relationships etc.** For example, in a legal or enterprise assistant, the word “**approval**” may refer to - ``` legal approval orfinance approval ormanager approval orregulatory approval orexternal approval orany other approval process ``` The feature with its relevance changes depending on the current context. Here feature controlling becomes intuitively useful. There is an important engineering point. SwiGLU require more infastructure — ``` more projectionsmore parametersmore computationdifferent initialization considerationshigher memory use ``` If customer’s system have fixed intents, short questions, small dataset etc… then a simpler ReLU or GELU model may provide better business value. So the better thinking is — *The customer has context-sensitive feature interactions, so we should assume that a controlled FFN may help and we will test that hypothesis.* As above code explanation , here are some useful code snippet for SwiGLU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion) ``` python from mini_transformer import ( ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, run_transformer,)....EXERCISES = [ ("cats drink", "milk"), ("dogs drink", "water"), ("I like", "tea"),]....def cross_entropy_for_expected_word(probability): """For a one-word target, loss = -ln(probability assigned to the target).""" return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature): .... for activation_name in activation_names: result = run_transformer(prompt, activation_name, temperature) probabilities = result["next_word_probabilities"] top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i]) target_index = NEXT_WORDS.index(target) p_target = probabilities[target_index] loss = cross_entropy_for_expected_word(p_target) parameter_count = count_feed_forward_parameters(activation_name) formatted_probabilities = " ".join(f"{number:.3f}" for number in probabilities)....def silu(number): return number * sigmoid(number)....def run_transformer(prompt, activation_name="relu", temperature=1.0): .... final_vectors = [] for token_vector in after_attention: details = feed_forward(token_vector, activation_name) feed_forward_details.append(details) residual_sum = vector_add(token_vector, details["output"]) final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name): .... if activation_name in ("glu", ....): gate_pre_activations = linear(input_vector, GATE_WEIGHTS, GATE_BIAS) .... for unit_index, first_branch_value in enumerate(input_pre_activations): .... gate = gate_pre_activations[unit_index] activated = silu(first_branch_value) * gate .... activated_units.append(activated) print("Activation units are - ",activated_units) output_vector = linear( activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS ) Activation units are - [2.815444702643749, 0.052585312093312096, -0.2645649263075179, -0.05701958653148752, 0.047145516636503355, -1.1773280335221021]Activation units are - [0.2259350766078342, 1.8208016659637194, -0.8741443667445493, -0.7458092598759276, -0.02992290648368431, -0.34554301678727567] ``` After customer interviews, collect representative questions. If we gather 1,000 examples, We may group them like this - Now we translate these into computational needs. **Factual Questions** A factual question needs - ``` php semantic understanding -> retrieval -> evidence ranking -> answer generation ``` This tells us that the quality of embeddings, retrieval, context window, internal representation, generation etc. may matter more than simply changing one activation function. **Yes/No Questions** For ***“Is this expense allowed?”*** we may produce a hidden representation **h**. Then **z = Wbh + b,** Then **P = sigmoid(z).** If **P = 0.91** the system can interpret this as ***91% probability of “allowed”.*** This is a classic use of Sigmoid. **Intent Routing** If the system must route to one of ``` HRFinanceITLegal ``` The Transformer produces **h(hidden weight)**. Then **z = Wch + b.** Suppose **z = [3.1, 1.2, 0.4, 2.1].** Softmax converts this into something like - ``` HR 0.61Finance 0.09IT 0.04Legal 0.26 ``` Now **HR** is the strongest choice. **Softmax** makes sense because the classes compete. **Multi-Label Requirements** If the customer says ***“My laptop is broken, I cannot log in, and I need a replacement.”,*** the relevant labels might be - ``` hardwareauthenticationreplacementbilling ``` These are not mutually exclusive. We want - ``` hardware 0.96authentication 0.89replacement 0.91billing 0.06 ``` Here we use independent Sigmoid outputs. Softmax would be wrong because it would force the labels to compete. **Generation** For a generative assistant, the model produces scores for vocabulary tokens. If the next-token logits are - ``` because 8.1due 7.2from 4.1elephant -3.2 ``` Softmax converts these into probabilities. Then the model chooses or samples the next token. This process repeats token by token. Vocabulary Softmax is a different use of Softmax from intent classification. It is having same mathematical family with different role. Let us think, we have a RAG system. We compute a relevance score between a question **q** and document **d**. If the question is — ***“Is this document relevant?” —*** that is an independent yes/no decision. Here Sigmoid makes sense. We might get - ``` Doc 1 = 0.93Doc 2 = 0.71Doc 3 = 0.09 ``` But if the question becomes — ***“Which document is best among these five?”*** then the scores of the documents compete. Softmax becomes more appropriate. This distinction is useful even beyond language generation. For enterprise assistants a stronger pipeline is - **Question** → **Transformer Understanding → Retrieve Evidence -> Generate Answer -> Verify Answer -> Accept / Retry / Abstain** The verifier may output **P(supported)** using **Sigmoid**. For example - ``` 0.96 → answer0.68 → retrieve more0.31 → abstain ``` In a real system, improving verification layer may produce more business value than changing GELU to SiLU. That is an important design lesson. A common mistake in AI projects is optimizing before we have a stable baseline. We should start activation function implementation with simple one - **ReLU** Then measure - ``` training lossvalidation lossaccuracylatencymemory useparameter countbusiness-task performance ``` Then we should test with **GELU, t** hen **SiLU, t** hen **SwiGLU**, which indicates towards change one important thing at a time. If 80% of customer questions are factual or yes/no and other 20% contain exceptions, a model may score 90% and fail to make right answer on the questions that are complex with reasoning. So the benchmark should be split into categories. For example - ``` Factual accuracyYes/No accuracyException accuracyComparison accuracyLong-context accuracyAmbiguity resolution ``` We consider a normal question — “**What is the payment period?**” This may have a factual answer — **30 days**. Now we have an exception question — “** Does the 30-day payment period still apply if the invoice is disputed and the supplier has failed to deliver?**” This question requires understanding conditions and relationships between clauses. A model with high overall accuracy but low exception accuracy may appear reliable in test period but may producing incorrect answers in complex situations. If the HR assistant has 1,000 benchmark questions - ``` 300 factual200 yes/no200 explanation150 comparison150 policy exception ``` We may train four variants of FFN Activation Functions. Now if we find something like - Here the trade-off becomes a business decision. If accuracy on exceptions is critical, SwiGLU may be worth the additional compute. If the model must respond under 45 ms on CPU, GELU may be a better choice. That is how **activation-function** selection becomes engineering decision than choosing terminology. Collect real examples and classify them. Examples are - ``` FactualYes/NoMulti-labelRoutingComparisonExplanationRecommendationGeneration ``` Ask - ``` Is this classification?Ranking?Generation?Binary decision?Multi-label?Retrieval? ``` For example - **Embedding** -> **Positional Encoding -> Multi-Head Attention -> FFN + ReLU -> Output** ``` Binary → SigmoidMulti-class → SoftmaxMulti-label → SigmoidGeneration → Vocabulary Softmax ``` Compare - ``` ReLUGELUSiLUSwiGLU ``` Measure - ``` accuracyF1precisionrecallretrieval qualitygroundednesshallucinationlatencymemorycost ``` The better equation for solve the Question-Answer modelling is - **Business Value = Accuracy + Reliability + Latency + Cost + Maintainability** In the Transformer, **Attention is the research component.** It asks — *“Where should I look?”* **The feed-forward network is the analyst.** It asks — *“What does this information mean?”* **ReLU, GELU, SiLU and SwiGLU affect how the analyst (FFN) transforms uncertain information.** **Sigmoid is the yes/no risk determination factor.** It asks — *“How likely is this?”* **Softmax is the ranking factor.** It asks — *“Which option deserves the highest probability?”* **The output head is the business department.** It asks — *“What form should the final decision take?”* This mental map may give useful architecture map. One chain is business-oriented — **Customer Question → Mathematical Task → Model Head → Output Activation.** The second is representation-oriented — **Language Complexity → Transformer Representation → FFN Design → Internal Activation or Controlling.** Keeping the above two chains makes activation-function optimization easier to understand. It helps us to take decision onuse activation functions which are not straight forward implementation of business rules. Activation functions are mathematical mechanisms that help the model transform and express information. The business requirement tells us **which mathematical mechanism we need and where we need it**. [How Activation Functions Shape an AI Assistant](https://pub.towardsai.net/how-activation-functions-shape-an-ai-assistant-1cac32b119be) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.