{"slug": "how-activation-functions-shape-an-ai-assistant", "title": "How Activation Functions Shape an AI Assistant", "summary": "A practitioner write-up on building an AI assistant argues that activation-function choice should follow the mathematical operation a business question requires, not start from terminology such as ReLU, GELU or SwiGLU. The author maps internal Transformer activations (ReLU, GELU, SiLU, SwiGLU) to feature transformation and business-output activations (Sigmoid, Softmax) to yes/no and multi-choice answers, and publishes a mini Transformer in Python plus sigmoid, ReLU, GELU and SiLU implementations in a GitHub repository (article_companion). The piece frames the design step as a discovery-phase exercise in which stakeholders' expected assistant behavior defines the question types before model selection.", "body_md": "When we work on building an AI assistant, we think of model names, parameters, embeddings, fine-tuning and other areas. Before arrive in those technical steps, we may have important design questions also for such a system. One of such questions is —\n\n**What are the mathematical part of work - does the AI assistant need to make?**\n\nThis particular question can affect the changes of the default behavior of the Transformer Model in our work. I have tried to discuss some of the details of choicing activation functions, which can eventually help us to make design of such an application. I tried to explain different business scenarios for the choices.\n\nDuring my writing, I have used AI Tools to generate the primary block diagrams for activation functions and their association with different Language Model specific mathematical calculation stages. I have edited those diagrams in [drawio editor](https://app.diagrams.net/) to maintain relevancy with my writing.\n\nFor an assitant, we may think some sample questions / clarifications -\n\n```\nIs this expense allowed?Which department should handle this?What is our leave policy?Compare Plan A and Plan B.What should I do next?\n```\n\nThe above questions / clarifications are having type of yes/no decision , ranking from high to low numbering, some of them require generating explanations. Other questions may require selecting option from many (multiple choice) and also others may require having multiple answers (checkbox type) at the same time. Here **activation functions** are having role in play. Activation functions are formulas inside a neural network and they are part of model decisions, such as -\n\n**What information should move forward, what should be suppressed, what should be treated as uncertain and other areas.**\n\nIf we make a very small Transformer written in Python, It contains following steps -\n\n```\ntoken embeddingspositional encodingself-attentionfeed-forward layersresidual connectionsnormalizationoutput layers\n```\n\nIn [Github](https://github.com/piyasde/article_companion/blob/main/mini_transformer.py), a mini transformer is commited.\n\nDuring work discovery phase, we should discuss with work stakeholders with their expectation of assistant behaior and identify a broad set of questions to set the boundary for the assistant. Here the starting point should not start with terminologies like R*eLU**,* *GELU**, or* *SwiGLU**.* Rather we should identify the *mathematical operations* which will cover the operations of the Assistant.\n\nFor example, the categorisation may be -\n\nThe above illustration can help us to make the transformer architecture or model selection with probable required activation functions.\n\nUseful distinctions for activation functions categories -\n\nFlow of control for **Internal activation functions** —\n\n**Inside Transformer → How should internal features be transformed? -> ReLU / GELU / SiLU / SwiGLU**\n\nFlow of control for **Business-output activation functions -**\n\n**Business Output -> What kind of answer is required? -> Sigmoid / Softmax**\n\nSome code snippets are for sigmoid, relu, gelu and silu, which are utilities in python codebase -\n\n``` python\nimport mathINPUT_NUMBERS = [-3.0, -2.0, -1.0, -0.5, 0.0, 0.5, 1.0, 2.0, 3.0]def sigmoid(number):    if number >= 0.0:        return 1.0 / (1.0 + math.exp(-number))    positive_exponent = math.exp(number)    return positive_exponent / (1.0 + positive_exponent)def relu(number):    return max(0.0, number)def gelu(number):    normal_cdf = 0.5 * (1.0 + math.erf(number / math.sqrt(2.0)))    return number * normal_cdfdef silu(number):    return number * sigmoid(number)for x in INPUT_NUMBERS:        print(f\"{x:8.2f} {relu(x):10.4f} {gelu(x):10.4f} \"              f\"{silu(x):10.4f} {sigmoid(x):10.4f}\")\n```\n\nThe full python codebase can be found in [Github](https://github.com/piyasde/article_companion). The output should be self explanatory -\n\n```\nReLU(x)=max(0,x) | GELU(x)=x*NormalCDF(x) | SiLU(x)=x*sigmoid(x)Sigmoid(x)=1/(1+exp(-x))   INPUT       ReLU       GELU       SiLU    Sigmoid   -3.00     0.0000    -0.0040    -0.1423     0.0474   -2.00     0.0000    -0.0455    -0.2384     0.1192   -1.00     0.0000    -0.1587    -0.2689     0.2689   -0.50     0.0000    -0.1543    -0.1888     0.3775    0.00     0.0000     0.0000     0.0000     0.5000    0.50     0.5000     0.3457     0.3112     0.6225    1.00     1.0000     0.8413     0.7311     0.7311    2.00     2.0000     1.9545     1.7616     0.8808    3.00     3.0000     2.9960     2.8577     0.9526\n```\n\nFor a question such as — ***Can an employee carry forward unused leave? —*** the transformer builds an internal representation. It forms hidden features like —\n\n```\nemployee policy       = strongleave                 = very strongcarry forward         = very strongpayroll               = weaktermination           = weakexception rule        = mediumeligibility           = strong\n```\n\nWe can think the hidden representations as **h.** The same representation can be used for different business purposes -\n\n```\n┌→ Yes/No head                    │Transformer → h ────┼→ Intent head                    │                    ├→ Retrieval score                    │                    └→ Language generation\n```\n\nHere the Transformer creates reusable contextual features. The business heads decide what those features should become.\n\nA minimal Transformer core steps are like the following pipeline -\n\n**Tokens** → **Embedding -> Positional Information -> Self-Attention -> Residual + Normalization -> Feed-Forward Network -> Residual + Normalization -> Output Representation.**\n\nHere Activation-related decisions are taken in five places -\n\n```\nattention Softmaxfeed-forward activationoptional gatingtask-specific output activationfinal vocabulary Softmax\n```\n\nThe attention block calculates -\n\n```\nQ = XWQK = XWKV = XWV\n```\n\nThen **AttentionScores = QK^T / sqrt(dk)**\n\nThen **AttentionWeights = Softmax(AttentionScores)**\n\nThen **Output = AttentionWeights × V**\n\nAs attention is a competition, the model is effectively asking — *Among all these tokens, where should I distribute my attention?*\n\nIf we have an information — t***he employee submitted the invoice because it was overdue. — f*** or the word “**it**,” attention may produce some score like -\n\n```\ninvoice   → 0.72employee  → 0.09submitted → 0.07overdue   → 0.12\n```\n\nThese values add to approximately 1 which is a probability distribution. Here the Softmax function is used to convert a vector of raw scores — logits — into a probability distribution. Here is a step by step illustrative attention function for understanding the work (can be optimized in production system with **numpy** and other useful python libraries) —\n\n``` python\ndef show_attention(prompt, temperature):    words = tokenize(prompt)    input_vectors = embedded_inputs(words)        queries, keys, values = project_queries_keys_values(input_vectors)    causal_weights, causal_contexts, scores = compute_attention(        queries, keys, values, temperature=temperature, causal=True    )    unmasked_weights, _, _ = compute_attention(        queries, keys, values, temperature=temperature, causal=False    )    print(\"Token Embedding + Position\")    for token_index, word in enumerate(words):        print(f\"  {token_index}. {word:>8} : {format_vector(input_vectors[token_index])}\")        print(\"Query, Key, Value (QKV)\")    for token_index, word in enumerate(words):        print(f\"  {word}: Q={format_vector(queries[token_index])}\")        print(f\"  {' ' * len(word)}  K={format_vector(keys[token_index])}\")        print(f\"  {' ' * len(word)}  V={format_vector(values[token_index])}\")    print(\"Dot-Product Similarity\")    heading = \"current\\\\candidate\".ljust(20) + \"\".join(w.rjust(12) for w in words)    for position_index, word in enumerate(words):        numbers = \"\".join(f\"{score:12.4f}\" for score in scores[position_index])        print(word.ljust(20) + numbers)    if len(words) >= 2:        current = len(words) - 1        candidate = 0        raw_dot = dot_product(queries[current], keys[candidate])        divisor = math.sqrt(len(queries[current]))        print(f\"\\nWorked example: last token '{words[current]}' vs first '{words[candidate]}'\")        print(f\"  Q dot K = {raw_dot:.5f}; sqrt(width) = {divisor:.5f}\")        print(f\"  scaled score = {raw_dot / divisor:.5f}\")    print(\"Causal Mask + Softmax\")    print(heading)    for position_index, word in enumerate(words):        numbers = \"\".join(f\"{p:12.4f}\" for p in causal_weights[position_index])        print(word.ljust(20) + numbers)    print(\"Unmasked Attention\")    print(heading)    for position_index, word in enumerate(words):        numbers = \"\".join(f\"{p:12.4f}\" for p in unmasked_weights[position_index])        print(word.ljust(20) + numbers)    print(\"Weighted Value Combination\")    for position_index, word in enumerate(words):        print(f\"  Context for '{word}' = {format_vector(causal_contexts[position_index])}\")    print(\"\\nThe attention weights come from Softmax; the context is the\")    print(\"weighted average/sum of V vectors.\")\n```\n\nThis whole program and related utility functions are in Transformer program in [Github Repository](https://github.com/piyasde/article_companion). Following are the operations and their works in each step -\n\nAttention answers — *Where should I look?*\n\nThe feed-forward network answers to — *What should I do with the information I have now?*\n\nA simple feed-forward block is —\n\n**FFN(x) = W2(ReLU(W1x + b1)) + b2**\n\nHere we can replace **ReLU** with **GELU**, **SiLU**, or redesign the block using **SwiGLU**. This is where we should do activation-level experimentation with FFN.\n\nNext, we will discuss the Liner Uinits — Activation functions with their process in diagarm with related codes. Overall Codes are in [Github](https://github.com/piyasde/article_companion/blob/main/transformer_comparison.py).\n\nIn designing the HR Assistant, some of the questions are -\n\n**How many casual leaves are available? Can leave be carried forward?Who approves travel reimbursement?What is the notice period?**\n\nThese questions are mostly **factual, narrow, structured, relatively low in ambiguity.** In such a case, **ReLU** may be a reasonable baseline. **ReLU** is — **ReLU(x) = max(0, x).** It will be like -\n\n```\nleave_policy     = +3.2termination      = -1.4salary_revision  = -0.8\n```\n\nAfter **ReLU** it becomes -\n\n```\nleave_policy     = 3.2termination      = 0salary_revision  = 0\n```\n\nThe intuition is simple — Only k*eep useful positive features. Suppress the rest.* For a low-cost system, this may be enough, though it should be tested and not only assumed.\n\nIn the code excercise, we have tried with ‘**cats drink**’ with a target of ‘** milk**’. Here are some hints on the execution path in the [code](https://github.com/piyasde/article_companion/blob/main/transformer_comparison.py), which we should study in [github repository](https://github.com/piyasde/article_companion) -\n\n``` python\nfrom mini_transformer import (    ACTIVATION_NAMES,    NEXT_WORDS,    count_feed_forward_parameters,    format_vector,    run_transformer,)....EXERCISES = [    (\"cats drink\", \"milk\"),    (\"dogs drink\", \"water\"),    (\"I like\", \"tea\"),]....def cross_entropy_for_expected_word(probability):    \"\"\"For a one-word target, loss = -ln(probability assigned to the target).\"\"\"    return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature):    ....    for activation_name in activation_names:        result = run_transformer(prompt, activation_name, temperature)        probabilities = result[\"next_word_probabilities\"]        top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i])        target_index = NEXT_WORDS.index(target)        p_target = probabilities[target_index]        loss = cross_entropy_for_expected_word(p_target)        parameter_count = count_feed_forward_parameters(activation_name)        formatted_probabilities = \"  \".join(f\"{number:.3f}\" for number in probabilities)....def relu(number):    return max(0.0, number)....def run_transformer(prompt, activation_name=\"relu\", temperature=1.0):    ....    final_vectors = []    for token_vector in after_attention:        details = feed_forward(token_vector, activation_name)        feed_forward_details.append(details)        residual_sum = vector_add(token_vector, details[\"output\"])        final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name):    ....    for unit_index, first_branch_value in enumerate(input_pre_activations):        if activation_name == \"relu\":            activated = relu(first_branch_value)            ....                 activated_units.append(activated)    print(\"Activation units are - \",activated_units)    output_vector = linear(        activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS    )\n```\n\nThe output should come as below in console -\n\n```\nActivation  FFN params Top word    P(target)       loss  P(milk)  P(water)  P(tea)  P(coffee)--------------------------------------------------------------------------------------------------------token vectorcats [0.9, -0.4, 0.7, 0.2]position vectorcats [0.0, -0.0, 0.0, -0.0]token vectordrink [-0.2, 0.8, 0.3, 0.5]position vectordrink [0.08, -0.04, 0.03, -0.06]Activation units are -  [2.891142495987482, 0.0, 0.0, 0.0, 0.0, 2.0732335436635374]Activation units are -  [0.0, 1.981098876135741, 1.3766495329967907, 1.085066439895106, 0.0, 0.0]relu                58 coffee         0.0441     3.1221  0.044  0.183  0.323  0.450\n```\n\nHere **loss = cross_entropy_for_expected_word(p_target)** is important factor. The loss is — 3.1221. Here the loss function measures how much probability the model assigned to that correct answer. For categorical cross-entropy with one correct target **Loss=-ln(P(correct word))** is equal to — **P(milk)=0.0441,** comes as **-ln(0.0441)** which is approx **3.122.** This is a relatively high loss because the model assigned low probability to the correct word. Here the model produces probabilities, and the loss is calculated afterward with the known target in evaluation or training phase. Here important part is at looking in the activation units, where some of them are 0.0 which is the outcome of **ReLU**.\n\nAgain during designing the HR Assistant, some of the questions may be -\n\n**I have been away for three days. Can I still regularize? My manager said it is okay, but HR rejected it. What happens now?If someone resigns midway through the year, what happens to unused leave?**\n\nThese questions contain — **ambiguity, context. exceptions, weak but possibly useful signals** etc.\n\nHere **GELU** can be be a fair choice as **GELU** does not make the zero or as-it-is decision as **ReLU**. Here **GELU** will keep a small transformed value instead. The intuition is — ***This signal is weak, but do not destroy it at early stages.*** That can be helpful in language because meaning emerges from many weak clues at a later training cycle. As above code explanation of RELU, here are some useful code snippet for GELU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion)\n\n``` python\nfrom mini_transformer import (    ACTIVATION_NAMES,    NEXT_WORDS,    count_feed_forward_parameters,    format_vector,    run_transformer,)....EXERCISES = [    (\"cats drink\", \"milk\"),    (\"dogs drink\", \"water\"),    (\"I like\", \"tea\"),]....def cross_entropy_for_expected_word(probability):    \"\"\"For a one-word target, loss = -ln(probability assigned to the target).\"\"\"    return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature):    ....    for activation_name in activation_names:        result = run_transformer(prompt, activation_name, temperature)        probabilities = result[\"next_word_probabilities\"]        top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i])        target_index = NEXT_WORDS.index(target)        p_target = probabilities[target_index]        loss = cross_entropy_for_expected_word(p_target)        parameter_count = count_feed_forward_parameters(activation_name)        formatted_probabilities = \"  \".join(f\"{number:.3f}\" for number in probabilities)....def gelu(number):    normal_cdf = 0.5 * (1.0 + math.erf(number / math.sqrt(2.0)))    return number * normal_cdf....def run_transformer(prompt, activation_name=\"relu\", temperature=1.0):    ....    final_vectors = []    for token_vector in after_attention:        details = feed_forward(token_vector, activation_name)        feed_forward_details.append(details)        residual_sum = vector_add(token_vector, details[\"output\"])        final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name):    ....    for unit_index, first_branch_value in enumerate(input_pre_activations):         ....         elif activation_name == \"gelu\":            activated = gelu(first_branch_value)            ....                 activated_units.append(activated)    print(\"Activation units are - \",activated_units)    output_vector = linear(        activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS    )\n```\n\nHere for above excercise dataset about (“cats drink”, “milk”), activation units comes closely to -\n\n```\nActivation units are -  [2.8855937569813706, -0.13352579602658668, -0.04908918220790799, -0.16690316151871892, -0.13620332184690795, 2.0336860436188022]Activation units are -  [-0.014077301889820216, 1.933968349321836, 1.260583778083766, 0.9343006736195222, -0.167826258210203, -0.1499369332953556]\n```\n\nThe nature of scoring is non-zero values even for non-important informations.\n\nSiLU is a smooth activation. It is — **SiLU(x) = x × sigmoid(x).** This gives an useful information — **signal × how strongly the signal allows itself through.** If the raw signal is strong and positive, it passes more weightage to next cycle. If the signal is weak or negative, it is reduced smoothly and pass less weightage in next cycle. This can make optimization smoother and can work for specific cases in neural architectures. As above code explanation , here are some useful code snippet for SILU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion)\n\n``` python\nfrom mini_transformer import (    ACTIVATION_NAMES,    NEXT_WORDS,    count_feed_forward_parameters,    format_vector,    run_transformer,)....EXERCISES = [    (\"cats drink\", \"milk\"),    (\"dogs drink\", \"water\"),    (\"I like\", \"tea\"),]....def cross_entropy_for_expected_word(probability):    \"\"\"For a one-word target, loss = -ln(probability assigned to the target).\"\"\"    return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature):    ....    for activation_name in activation_names:        result = run_transformer(prompt, activation_name, temperature)        probabilities = result[\"next_word_probabilities\"]        top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i])        target_index = NEXT_WORDS.index(target)        p_target = probabilities[target_index]        loss = cross_entropy_for_expected_word(p_target)        parameter_count = count_feed_forward_parameters(activation_name)        formatted_probabilities = \"  \".join(f\"{number:.3f}\" for number in probabilities)....def silu(number):    return number * sigmoid(number)....def run_transformer(prompt, activation_name=\"relu\", temperature=1.0):    ....    final_vectors = []    for token_vector in after_attention:        details = feed_forward(token_vector, activation_name)        feed_forward_details.append(details)        residual_sum = vector_add(token_vector, details[\"output\"])        final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name):    ....    for unit_index, first_branch_value in enumerate(input_pre_activations):         ....         elif activation_name == \"silu\":            activated = silu(first_branch_value)                     ....                 activated_units.append(activated)    print(\"Activation units are - \",activated_units)    output_vector = linear(        activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS    )\nActivation units are -  [2.7390881702111454, -0.2782831534600395, -0.2420876982940121, -0.22024136849212178, -0.15802568867005076, 1.8416000155094343]Activation units are -  [-0.18567423752791756, 1.740986233750263, 1.0991890705587457, 0.811034605518993, -0.2238910250978735, -0.2744103198613716]\n```\n\nThe nature of scoring is more weightage for stronger signal.\n\n**GLU** comes with an idea. Here instead of asking only — ***“What feature exists?”,*** the network also asks — *“Should this feature matter here?”*\n\nWe create two paths. One path produces information — **A = xWA**. Another path produces a control gate **G = xWG**. Then **Output = A × sigmoid(G).** This can be understood as -\n\n**A = what information did we find? G = how much should we use it now?**\n\nAs above code explanation , here are some useful code snippet for GLU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion)\n\n``` python\nfrom mini_transformer import (    ACTIVATION_NAMES,    NEXT_WORDS,    count_feed_forward_parameters,    format_vector,    run_transformer,)....EXERCISES = [    (\"cats drink\", \"milk\"),    (\"dogs drink\", \"water\"),    (\"I like\", \"tea\"),]....def cross_entropy_for_expected_word(probability):    \"\"\"For a one-word target, loss = -ln(probability assigned to the target).\"\"\"    return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature):    ....    for activation_name in activation_names:        result = run_transformer(prompt, activation_name, temperature)        probabilities = result[\"next_word_probabilities\"]        top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i])        target_index = NEXT_WORDS.index(target)        p_target = probabilities[target_index]        loss = cross_entropy_for_expected_word(p_target)        parameter_count = count_feed_forward_parameters(activation_name)        formatted_probabilities = \"  \".join(f\"{number:.3f}\" for number in probabilities)....def silu(number):    return number * sigmoid(number)def sigmoid(number):    if number >= 0.0:        return 1.0 / (1.0 + math.exp(-number))    positive_exponent = math.exp(number)    return positive_exponent / (1.0 + positive_exponent)....def run_transformer(prompt, activation_name=\"relu\", temperature=1.0):    ....    final_vectors = []    for token_vector in after_attention:        details = feed_forward(token_vector, activation_name)        feed_forward_details.append(details)        residual_sum = vector_add(token_vector, details[\"output\"])        final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name):    ....    if activation_name in (\"glu\", ....):        gate_pre_activations = linear(input_vector, GATE_WEIGHTS, GATE_BIAS)    ....    for unit_index, first_branch_value in enumerate(input_pre_activations):         ....         elif activation_name == \"glu\":            gate = sigmoid(gate_pre_activations[unit_index])            activated = first_branch_value * gate         ....                 activated_units.append(activated)    print(\"Activation units are - \",activated_units)    output_vector = linear(        activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS    )\nActivation units are -  [2.1293380942953477, -0.5606983585437584, -1.4670984863133165, -0.35918495390583977, -0.16691853734558215, 0.716106547145415]Activation units are -  [-0.5803804656519378, 1.4659660543453696, 0.42819269347992506, 0.30929164364610134, -0.3492711723798345, -0.8514748949632387]\n```\n\n**SwiGLU** extends the activation idea using **SiLU**.\n\nIt is — **SwiGLU(x) = SiLU(xWg) × (xWu).**\n\nAfter above operation, a final projection brings the representation back to the model dimension. This becomes useful when the customer domain has many **overlapping concepts, multiple meanings, complex context, exception rules, long relationships etc.** For example, in a legal or enterprise assistant, the word “**approval**” may refer to -\n\n```\nlegal approval orfinance approval ormanager approval orregulatory approval orexternal approval orany other approval process\n```\n\nThe feature with its relevance changes depending on the current context. Here feature controlling becomes intuitively useful. There is an important engineering point. SwiGLU require more infastructure —\n\n```\nmore projectionsmore parametersmore computationdifferent initialization considerationshigher memory use\n```\n\nIf customer’s system have fixed intents, short questions, small dataset etc… then a simpler ReLU or GELU model may provide better business value. So the better thinking is — *The customer has context-sensitive feature interactions, so we should assume that a controlled FFN may help and we will test that hypothesis.*\n\nAs above code explanation , here are some useful code snippet for SwiGLU, which can be found i[n Github Repository.](https://github.com/piyasde/article_companion)\n\n``` python\nfrom mini_transformer import (    ACTIVATION_NAMES,    NEXT_WORDS,    count_feed_forward_parameters,    format_vector,    run_transformer,)....EXERCISES = [    (\"cats drink\", \"milk\"),    (\"dogs drink\", \"water\"),    (\"I like\", \"tea\"),]....def cross_entropy_for_expected_word(probability):    \"\"\"For a one-word target, loss = -ln(probability assigned to the target).\"\"\"    return -math.log(max(probability, 1e-15))....def show_comparison(prompt, target, activation_names, temperature):    ....    for activation_name in activation_names:        result = run_transformer(prompt, activation_name, temperature)        probabilities = result[\"next_word_probabilities\"]        top_word_index = max(range(len(probabilities)), key=lambda i: probabilities[i])        target_index = NEXT_WORDS.index(target)        p_target = probabilities[target_index]        loss = cross_entropy_for_expected_word(p_target)        parameter_count = count_feed_forward_parameters(activation_name)        formatted_probabilities = \"  \".join(f\"{number:.3f}\" for number in probabilities)....def silu(number):    return number * sigmoid(number)....def run_transformer(prompt, activation_name=\"relu\", temperature=1.0):    ....    final_vectors = []    for token_vector in after_attention:        details = feed_forward(token_vector, activation_name)        feed_forward_details.append(details)        residual_sum = vector_add(token_vector, details[\"output\"])        final_vectors.append(layer_normalize(residual_sum))....def feed_forward(input_vector, activation_name):    ....    if activation_name in (\"glu\", ....):        gate_pre_activations = linear(input_vector, GATE_WEIGHTS, GATE_BIAS)    ....    for unit_index, first_branch_value in enumerate(input_pre_activations):         ....            gate = gate_pre_activations[unit_index]            activated = silu(first_branch_value) * gate         ....                 activated_units.append(activated)    print(\"Activation units are - \",activated_units)    output_vector = linear(        activated_units, FEED_FORWARD_OUTPUT_WEIGHTS, FEED_FORWARD_OUTPUT_BIAS    )\nActivation units are -  [2.815444702643749, 0.052585312093312096, -0.2645649263075179, -0.05701958653148752, 0.047145516636503355, -1.1773280335221021]Activation units are -  [0.2259350766078342, 1.8208016659637194, -0.8741443667445493, -0.7458092598759276, -0.02992290648368431, -0.34554301678727567]\n```\n\nAfter customer interviews, collect representative questions. If we gather 1,000 examples, We may group them like this -\n\nNow we translate these into computational needs.\n\n**Factual Questions**\n\nA factual question needs -\n\n``` php\nsemantic understanding -> retrieval -> evidence ranking -> answer generation\n```\n\nThis tells us that the quality of embeddings, retrieval, context window, internal representation, generation etc. may matter more than simply changing one activation function.\n\n**Yes/No Questions**\n\nFor ***“Is this expense allowed?”*** we may produce a hidden representation **h**. Then **z = Wbh + b,** Then **P = sigmoid(z).** If **P = 0.91** the system can interpret this as ***91% probability of “allowed”.*** This is a classic use of Sigmoid.\n\n**Intent Routing**\n\nIf the system must route to one of\n\n```\nHRFinanceITLegal\n```\n\nThe Transformer produces **h(hidden weight)**. Then **z = Wch + b.**\n\nSuppose **z = [3.1, 1.2, 0.4, 2.1].** Softmax converts this into something like -\n\n```\nHR       0.61Finance  0.09IT       0.04Legal    0.26\n```\n\nNow **HR** is the strongest choice. **Softmax** makes sense because the classes compete.\n\n**Multi-Label Requirements**\n\nIf the customer says ***“My laptop is broken, I cannot log in, and I need a replacement.”,*** the relevant labels might be -\n\n```\nhardwareauthenticationreplacementbilling\n```\n\nThese are not mutually exclusive. We want -\n\n```\nhardware        0.96authentication  0.89replacement     0.91billing         0.06\n```\n\nHere we use independent Sigmoid outputs. Softmax would be wrong because it would force the labels to compete.\n\n**Generation**\n\nFor a generative assistant, the model produces scores for vocabulary tokens. If the next-token logits are -\n\n```\nbecause   8.1due       7.2from      4.1elephant -3.2\n```\n\nSoftmax converts these into probabilities. Then the model chooses or samples the next token. This process repeats token by token. Vocabulary Softmax is a different use of Softmax from intent classification. It is having same mathematical family with different role.\n\nLet us think, we have a RAG system. We compute a relevance score between a question **q** and document **d**. If the question is — ***“Is this document relevant?” —*** that is an independent yes/no decision. Here Sigmoid makes sense. We might get -\n\n```\nDoc 1 = 0.93Doc 2 = 0.71Doc 3 = 0.09\n```\n\nBut if the question becomes — ***“Which document is best among these five?”*** then the scores of the documents compete. Softmax becomes more appropriate. This distinction is useful even beyond language generation.\n\nFor enterprise assistants a stronger pipeline is -\n\n**Question** → **Transformer Understanding → Retrieve Evidence -> Generate Answer -> Verify Answer -> Accept / Retry / Abstain**\n\nThe verifier may output **P(supported)** using **Sigmoid**. For example -\n\n```\n0.96 → answer0.68 → retrieve more0.31 → abstain\n```\n\nIn a real system, improving verification layer may produce more business value than changing GELU to SiLU. That is an important design lesson.\n\nA common mistake in AI projects is optimizing before we have a stable baseline. We should start activation function implementation with simple one -\n\n**ReLU**\n\nThen measure -\n\n```\ntraining lossvalidation lossaccuracylatencymemory useparameter countbusiness-task performance\n```\n\nThen we should test with **GELU, t** hen **SiLU, t** hen **SwiGLU**, which indicates towards change one important thing at a time.\n\nIf 80% of customer questions are factual or yes/no and other 20% contain exceptions, a model may score 90% and fail to make right answer on the questions that are complex with reasoning. So the benchmark should be split into categories. For example -\n\n```\nFactual accuracyYes/No accuracyException accuracyComparison accuracyLong-context accuracyAmbiguity resolution\n```\n\nWe consider a normal question — “**What is the payment period?**” This may have a factual answer — **30 days**. Now we have an exception question — “** Does the 30-day payment period still apply if the invoice is disputed and the supplier has failed to deliver?**” This question requires understanding conditions and relationships between clauses. A model with high overall accuracy but low exception accuracy may appear reliable in test period but may producing incorrect answers in complex situations.\n\nIf the HR assistant has 1,000 benchmark questions -\n\n```\n300 factual200 yes/no200 explanation150 comparison150 policy exception\n```\n\nWe may train four variants of FFN Activation Functions. Now if we find something like -\n\nHere the trade-off becomes a business decision.\n\nIf accuracy on exceptions is critical, SwiGLU may be worth the additional compute.\n\nIf the model must respond under 45 ms on CPU, GELU may be a better choice.\n\nThat is how **activation-function** selection becomes engineering decision than choosing terminology.\n\nCollect real examples and classify them. Examples are -\n\n```\nFactualYes/NoMulti-labelRoutingComparisonExplanationRecommendationGeneration\n```\n\nAsk -\n\n```\nIs this classification?Ranking?Generation?Binary decision?Multi-label?Retrieval?\n```\n\nFor example -\n\n**Embedding** -> **Positional Encoding -> Multi-Head Attention -> FFN + ReLU -> Output**\n\n```\nBinary      → SigmoidMulti-class → SoftmaxMulti-label → SigmoidGeneration  → Vocabulary Softmax\n```\n\nCompare -\n\n```\nReLUGELUSiLUSwiGLU\n```\n\nMeasure -\n\n```\naccuracyF1precisionrecallretrieval qualitygroundednesshallucinationlatencymemorycost\n```\n\nThe better equation for solve the Question-Answer modelling is -\n\n**Business Value = Accuracy + Reliability + Latency + Cost + Maintainability**\n\nIn the Transformer, **Attention is the research component.** It asks — *“Where should I look?”* **The feed-forward network is the analyst.** It asks — *“What does this information mean?”*\n\n**ReLU, GELU, SiLU and SwiGLU affect how the analyst (FFN) transforms uncertain information.**\n\n**Sigmoid is the yes/no risk determination factor.** It asks — *“How likely is this?”* **Softmax is the ranking factor.** It asks — *“Which option deserves the highest probability?”* **The output head is the business department.** It asks — *“What form should the final decision take?”*\n\nThis mental map may give useful architecture map.\n\nOne chain is business-oriented — **Customer Question → Mathematical Task → Model Head → Output Activation.**\n\nThe second is representation-oriented — **Language Complexity → Transformer Representation → FFN Design → Internal Activation or Controlling.**\n\nKeeping the above two chains makes activation-function optimization easier to understand. It helps us to take decision onuse activation functions which are not straight forward implementation of business rules. Activation functions are mathematical mechanisms that help the model transform and express information. The business requirement tells us **which mathematical mechanism we need and where we need it**.\n\n[How Activation Functions Shape an AI Assistant](https://pub.towardsai.net/how-activation-functions-shape-an-ai-assistant-1cac32b119be) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-activation-functions-shape-an-ai-assistant", "canonical_source": "https://pub.towardsai.net/how-activation-functions-shape-an-ai-assistant-1cac32b119be?source=rss----98111c9905da---4", "published_at": "2026-11-11 12:01:01+00:00", "updated_at": "2026-11-11 12:23:01.800867+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks"], "entities": ["GitHub", "drawio", "ReLU", "GELU", "SwiGLU", "Sigmoid", "Softmax", "Transformer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-activation-functions-shape-an-ai-assistant", "markdown": "https://wpnews.pro/news/how-activation-functions-shape-an-ai-assistant.md", "text": "https://wpnews.pro/news/how-activation-functions-shape-an-ai-assistant.txt", "jsonld": "https://wpnews.pro/news/how-activation-functions-shape-an-ai-assistant.jsonld"}}