How Activation Functions Shape an AI Assistant

A practitioner write-up on building an AI assistant argues that activation-function choice should follow the mathematical operation a business question requires, not start from terminology such as ReLU, GELU or SwiGLU. The author maps internal Transformer activations (ReLU, GELU, SiLU, SwiGLU) to feature transformation and business-output activations (Sigmoid, Softmax) to yes/no and multi-choice answers, and publishes a mini Transformer in Python plus sigmoid, ReLU, GELU and SiLU implementations in a GitHub repository (article_companion). The piece frames the design step as a discovery-phase exercise in which stakeholders' expected assistant behavior defines the question types before model selection.

When we work on building an AI assistant, we think of model names, parameters, embeddings, fine-tuning and other areas. Before arrive in those technical steps, we may have important design questions also for such a system. One of such questions is — What are the mathematical part of work - does the AI assistant need to make? This particular question can affect the changes of the default behavior of the Transformer Model in our work. I have tried to discuss some of the details of choicing activation functions, which can eventually help us to make design of such an application. I tried to explain different business scenarios for the choices. During my writing, I have used AI Tools to generate the primary block diagrams for activation functions and their association with different Language Model specific mathematical calculation stages. I have edited those diagrams in drawio editor https://app.diagrams.net/ to maintain relevancy with my writing. For an assitant, we may think some sample questions / clarifications - Is this expense allowed?Which department should handle this?What is our leave policy?Compare Plan A and Plan B.What should I do next? The above questions / clarifications are having type of yes/no decision , ranking from high to low numbering, some of them require generating explanations. Other questions may require selecting option from many multiple choice and also others may require having multiple answers checkbox type at the same time. Here activation functions are having role in play. Activation functions are formulas inside a neural network and they are part of model decisions, such as - What information should move forward, what should be suppressed, what should be treated as uncertain and other areas. If we make a very small Transformer written in Python, It contains following steps - token embeddingspositional encodingself-attentionfeed-forward layersresidual connectionsnormalizationoutput layers In Github https://github.com/piyasde/article companion/blob/main/mini transformer.py , a mini transformer is commited. During work discovery phase, we should discuss with work stakeholders with their expectation of assistant behaior and identify a broad set of questions to set the boundary for the assistant. Here the starting point should not start with terminologies like R eLU , GELU , or SwiGLU . Rather we should identify the mathematical operations which will cover the operations of the Assistant. For example, the categorisation may be - The above illustration can help us to make the transformer architecture or model selection with probable required activation functions. Useful distinctions for activation functions categories - Flow of control for Internal activation functions — Inside Transformer → How should internal features be transformed? - ReLU / GELU / SiLU / SwiGLU Flow of control for Business-output activation functions - Business Output - What kind of answer is required? - Sigmoid / Softmax Some code snippets are for sigmoid, relu, gelu and silu, which are utilities in python codebase - python import mathINPUT NUMBERS = -3.0, -2.0, -1.0, -0.5, 0.0, 0.5, 1.0, 2.0, 3.0 def sigmoid number : if number = 0.0: return 1.0 / 1.0 + math.exp -number positive exponent = math.exp number return positive exponent / 1.0 + positive exponent def relu number : return max 0.0, number def gelu number : normal cdf = 0.5 1.0 + math.erf number / math.sqrt 2.0 return number normal cdfdef silu number : return number sigmoid number for x in INPUT NUMBERS: print f"{x:8.2f} {relu x :10.4f} {gelu x :10.4f} " f"{silu x :10.4f} {sigmoid x :10.4f}" The full python codebase can be found in Github https://github.com/piyasde/article companion . The output should be self explanatory - ReLU x =max 0,x | GELU x =x NormalCDF x | SiLU x =x sigmoid x Sigmoid x =1/ 1+exp -x INPUT ReLU GELU SiLU Sigmoid -3.00 0.0000 -0.0040 -0.1423 0.0474 -2.00 0.0000 -0.0455 -0.2384 0.1192 -1.00 0.0000 -0.1587 -0.2689 0.2689 -0.50 0.0000 -0.1543 -0.1888 0.3775 0.00 0.0000 0.0000 0.0000 0.5000 0.50 0.5000 0.3457 0.3112 0.6225 1.00 1.0000 0.8413 0.7311 0.7311 2.00 2.0000 1.9545 1.7616 0.8808 3.00 3.0000 2.9960 2.8577 0.9526 For a question such as — Can an employee carry forward unused leave? — the transformer builds an internal representation. It forms hidden features like — employee policy = strongleave = very strongcarry forward = very strongpayroll = weaktermination = weakexception rule = mediumeligibility = strong We can think the hidden representations as h. The same representation can be used for different business purposes - ┌→ Yes/No head │Transformer → h ────┼→ Intent head │ ├→ Retrieval score │ └→ Language generation Here the Transformer creates reusable contextual features. The business heads decide what those features should become. A minimal Transformer core steps are like the following pipeline - Tokens → Embedding - Positional Information - Self-Attention - Residual + Normalization - Feed-Forward Network - Residual + Normalization - Output Representation. Here Activation-related decisions are taken in five places - attention Softmaxfeed-forward activationoptional gatingtask-specific output activationfinal vocabulary Softmax The attention block calculates - Q = XWQK = XWKV = XWV Then AttentionScores = QK^T / sqrt dk Then AttentionWeights = Softmax AttentionScores Then Output = AttentionWeights × V As attention is a competition, the model is effectively asking — Among all these tokens, where should I distribute my attention? If we have an information — t he employee submitted the invoice because it was overdue. — f or the word “ it ,” attention may produce some score like - invoice → 0.72employee → 0.09submitted → 0.07overdue → 0.12 These values add to approximately 1 which is a probability distribution. Here the Softmax function is used to convert a vector of raw scores — logits — into a probability distribution. Here is a step by step illustrative attention function for understanding the work can be optimized in production system with numpy and other useful python libraries — python def show attention prompt, temperature : words = tokenize prompt input vectors = embedded inputs words queries, keys, values = project queries keys values input vectors causal weights, causal contexts, scores = compute attention queries, keys, values, temperature=temperature, causal=True unmasked weights, , = compute attention queries, keys, values, temperature=temperature, causal=False print "Token Embedding + Position" for token index, word in enumerate words : print f" {token index}. {word: 8} : {format vector input vectors token index }" print "Query, Key, Value QKV " for token index, word in enumerate words : print f" {word}: Q={format vector queries token index }" print f" {' ' len word } K={format vector keys token index }" print f" {' ' len word } V={format vector values token index }" print "Dot-Product Similarity" heading = "current\\candidate".ljust 20 + "".join w.rjust 12 for w in words for position index, word in enumerate words : numbers = "".join f"{score:12.4f}" for score in scores position index print word.ljust 20 + numbers if len words = 2: current = len words - 1 candidate = 0 raw dot = dot product queries current , keys candidate divisor = math.sqrt len queries current print f"\nWorked example: last token '{words current }' vs first '{words candidate }'" print f" Q dot K = {raw dot:.5f}; sqrt width = {divisor:.5f}" print f" scaled score = {raw dot / divisor:.5f}" print "Causal Mask + Softmax" print heading for position index, word in enumerate words : numbers = "".join f"{p:12.4f}" for p in causal weights position index print word.ljust 20 + numbers print "Unmasked Attention" print heading for position index, word in enumerate words : numbers = "".join f"{p:12.4f}" for p in unmasked weights position index print word.ljust 20 + numbers print "Weighted Value Combination" for position index, word in enumerate words : print f" Context for '{word}' = {format vector causal contexts position index }" print "\nThe attention weights come from Softmax; the context is the" print "weighted average/sum of V vectors." This whole program and related utility functions are in Transformer program in Github Repository https://github.com/piyasde/article companion . Following are the operations and their works in each step - Attention answers — Where should I look? The feed-forward network answers to — What should I do with the information I have now? A simple feed-forward block is — FFN x = W2 ReLU W1x + b1 + b2 Here we can replace ReLU with GELU , SiLU , or redesign the block using SwiGLU . This is where we should do activation-level experimentation with FFN. Next, we will discuss the Liner Uinits — Activation functions with their process in diagarm with related codes. Overall Codes are in Github https://github.com/piyasde/article companion/blob/main/transformer comparison.py . In designing the HR Assistant, some of the questions are - How many casual leaves are available? Can leave be carried forward?Who approves travel reimbursement?What is the notice period? These questions are mostly factual, narrow, structured, relatively low in ambiguity. In such a case, ReLU may be a reasonable baseline. ReLU is — ReLU x = max 0, x . It will be like - leave policy = +3.2termination = -1.4salary revision = -0.8 After ReLU it becomes - leave policy = 3.2termination = 0salary revision = 0 The intuition is simple — Only k eep useful positive features. Suppress the rest. For a low-cost system, this may be enough, though it should be tested and not only assumed. In the code excercise, we have tried with ‘ cats drink ’ with a target of ‘ milk ’. Here are some hints on the execution path in the code https://github.com/piyasde/article companion/blob/main/transformer comparison.py , which we should study in github repository https://github.com/piyasde/article companion - python from mini transformer import ACTIVATION NAMES, NEXT WORDS, count feed forward parameters, format vector, run transformer, ....EXERCISES = "cats drink", "milk" , "dogs drink", "water" , "I like", "tea" , ....def cross entropy for expected word probability : """For a one-word target, loss = -ln probability assigned to the target .""" return -math.log max probability, 1e-15 ....def show comparison prompt, target, activation names, temperature : .... for activation name in activation names: result = run transformer prompt, activation name, temperature probabilities = result "next word probabilities" top word index = max range len probabilities , key=lambda i: probabilities i target index = NEXT WORDS.index target p target = probabilities target index loss = cross entropy for expected word p target parameter count = count feed forward parameters activation name formatted probabilities = " ".join f"{number:.3f}" for number in probabilities ....def relu number : return max 0.0, number ....def run transformer prompt, activation name="relu", temperature=1.0 : .... final vectors = for token vector in after attention: details = feed forward token vector, activation name feed forward details.append details residual sum = vector add token vector, details "output" final vectors.append layer normalize residual sum ....def feed forward input vector, activation name : .... for unit index, first branch value in enumerate input pre activations : if activation name == "relu": activated = relu first branch value .... activated units.append activated print "Activation units are - ",activated units output vector = linear activated units, FEED FORWARD OUTPUT WEIGHTS, FEED FORWARD OUTPUT BIAS The output should come as below in console - Activation FFN params Top word P target loss P milk P water P tea P coffee --------------------------------------------------------------------------------------------------------token vectorcats 0.9, -0.4, 0.7, 0.2 position vectorcats 0.0, -0.0, 0.0, -0.0 token vectordrink -0.2, 0.8, 0.3, 0.5 position vectordrink 0.08, -0.04, 0.03, -0.06 Activation units are - 2.891142495987482, 0.0, 0.0, 0.0, 0.0, 2.0732335436635374 Activation units are - 0.0, 1.981098876135741, 1.3766495329967907, 1.085066439895106, 0.0, 0.0 relu 58 coffee 0.0441 3.1221 0.044 0.183 0.323 0.450 Here loss = cross entropy for expected word p target is important factor. The loss is — 3.1221. Here the loss function measures how much probability the model assigned to that correct answer. For categorical cross-entropy with one correct target Loss=-ln P correct word is equal to — P milk =0.0441, comes as -ln 0.0441 which is approx 3.122. This is a relatively high loss because the model assigned low probability to the correct word. Here the model produces probabilities, and the loss is calculated afterward with the known target in evaluation or training phase. Here important part is at looking in the activation units, where some of them are 0.0 which is the outcome of ReLU . Again during designing the HR Assistant, some of the questions may be - I have been away for three days. Can I still regularize? My manager said it is okay, but HR rejected it. What happens now?If someone resigns midway through the year, what happens to unused leave? These questions contain — ambiguity, context. exceptions, weak but possibly useful signals etc. Here GELU can be be a fair choice as GELU does not make the zero or as-it-is decision as ReLU . Here GELU will keep a small transformed value instead. The intuition is — This signal is weak, but do not destroy it at early stages. That can be helpful in language because meaning emerges from many weak clues at a later training cycle. As above code explanation of RELU, here are some useful code snippet for GELU, which can be found i n Github Repository. https://github.com/piyasde/article companion python from mini transformer import ACTIVATION NAMES, NEXT WORDS, count feed forward parameters, format vector, run transformer, ....EXERCISES = "cats drink", "milk" , "dogs drink", "water" , "I like", "tea" , ....def cross entropy for expected word probability : """For a one-word target, loss = -ln probability assigned to the target .""" return -math.log max probability, 1e-15 ....def show comparison prompt, target, activation names, temperature : .... for activation name in activation names: result = run transformer prompt, activation name, temperature probabilities = result "next word probabilities" top word index = max range len probabilities , key=lambda i: probabilities i target index = NEXT WORDS.index target p target = probabilities target index loss = cross entropy for expected word p target parameter count = count feed forward parameters activation name formatted probabilities = " ".join f"{number:.3f}" for number in probabilities ....def gelu number : normal cdf = 0.5 1.0 + math.erf number / math.sqrt 2.0 return number normal cdf....def run transformer prompt, activation name="relu", temperature=1.0 : .... final vectors = for token vector in after attention: details = feed forward token vector, activation name feed forward details.append details residual sum = vector add token vector, details "output" final vectors.append layer normalize residual sum ....def feed forward input vector, activation name : .... for unit index, first branch value in enumerate input pre activations : .... elif activation name == "gelu": activated = gelu first branch value .... activated units.append activated print "Activation units are - ",activated units output vector = linear activated units, FEED FORWARD OUTPUT WEIGHTS, FEED FORWARD OUTPUT BIAS Here for above excercise dataset about “cats drink”, “milk” , activation units comes closely to - Activation units are - 2.8855937569813706, -0.13352579602658668, -0.04908918220790799, -0.16690316151871892, -0.13620332184690795, 2.0336860436188022 Activation units are - -0.014077301889820216, 1.933968349321836, 1.260583778083766, 0.9343006736195222, -0.167826258210203, -0.1499369332953556 The nature of scoring is non-zero values even for non-important informations. SiLU is a smooth activation. It is — SiLU x = x × sigmoid x . This gives an useful information — signal × how strongly the signal allows itself through. If the raw signal is strong and positive, it passes more weightage to next cycle. If the signal is weak or negative, it is reduced smoothly and pass less weightage in next cycle. This can make optimization smoother and can work for specific cases in neural architectures. As above code explanation , here are some useful code snippet for SILU, which can be found i n Github Repository. https://github.com/piyasde/article companion python from mini transformer import ACTIVATION NAMES, NEXT WORDS, count feed forward parameters, format vector, run transformer, ....EXERCISES = "cats drink", "milk" , "dogs drink", "water" , "I like", "tea" , ....def cross entropy for expected word probability : """For a one-word target, loss = -ln probability assigned to the target .""" return -math.log max probability, 1e-15 ....def show comparison prompt, target, activation names, temperature : .... for activation name in activation names: result = run transformer prompt, activation name, temperature probabilities = result "next word probabilities" top word index = max range len probabilities , key=lambda i: probabilities i target index = NEXT WORDS.index target p target = probabilities target index loss = cross entropy for expected word p target parameter count = count feed forward parameters activation name formatted probabilities = " ".join f"{number:.3f}" for number in probabilities ....def silu number : return number sigmoid number ....def run transformer prompt, activation name="relu", temperature=1.0 : .... final vectors = for token vector in after attention: details = feed forward token vector, activation name feed forward details.append details residual sum = vector add token vector, details "output" final vectors.append layer normalize residual sum ....def feed forward input vector, activation name : .... for unit index, first branch value in enumerate input pre activations : .... elif activation name == "silu": activated = silu first branch value .... activated units.append activated print "Activation units are - ",activated units output vector = linear activated units, FEED FORWARD OUTPUT WEIGHTS, FEED FORWARD OUTPUT BIAS Activation units are - 2.7390881702111454, -0.2782831534600395, -0.2420876982940121, -0.22024136849212178, -0.15802568867005076, 1.8416000155094343 Activation units are - -0.18567423752791756, 1.740986233750263, 1.0991890705587457, 0.811034605518993, -0.2238910250978735, -0.2744103198613716 The nature of scoring is more weightage for stronger signal. GLU comes with an idea. Here instead of asking only — “What feature exists?”, the network also asks — “Should this feature matter here?” We create two paths. One path produces information — A = xWA . Another path produces a control gate G = xWG . Then Output = A × sigmoid G . This can be understood as - A = what information did we find? G = how much should we use it now? As above code explanation , here are some useful code snippet for GLU, which can be found i n Github Repository. https://github.com/piyasde/article companion python from mini transformer import ACTIVATION NAMES, NEXT WORDS, count feed forward parameters, format vector, run transformer, ....EXERCISES = "cats drink", "milk" , "dogs drink", "water" , "I like", "tea" , ....def cross entropy for expected word probability : """For a one-word target, loss = -ln probability assigned to the target .""" return -math.log max probability, 1e-15 ....def show comparison prompt, target, activation names, temperature : .... for activation name in activation names: result = run transformer prompt, activation name, temperature probabilities = result "next word probabilities" top word index = max range len probabilities , key=lambda i: probabilities i target index = NEXT WORDS.index target p target = probabilities target index loss = cross entropy for expected word p target parameter count = count feed forward parameters activation name formatted probabilities = " ".join f"{number:.3f}" for number in probabilities ....def silu number : return number sigmoid number def sigmoid number : if number = 0.0: return 1.0 / 1.0 + math.exp -number positive exponent = math.exp number return positive exponent / 1.0 + positive exponent ....def run transformer prompt, activation name="relu", temperature=1.0 : .... final vectors = for token vector in after attention: details = feed forward token vector, activation name feed forward details.append details residual sum = vector add token vector, details "output" final vectors.append layer normalize residual sum ....def feed forward input vector, activation name : .... if activation name in "glu", .... : gate pre activations = linear input vector, GATE WEIGHTS, GATE BIAS .... for unit index, first branch value in enumerate input pre activations : .... elif activation name == "glu": gate = sigmoid gate pre activations unit index activated = first branch value gate .... activated units.append activated print "Activation units are - ",activated units output vector = linear activated units, FEED FORWARD OUTPUT WEIGHTS, FEED FORWARD OUTPUT BIAS Activation units are - 2.1293380942953477, -0.5606983585437584, -1.4670984863133165, -0.35918495390583977, -0.16691853734558215, 0.716106547145415 Activation units are - -0.5803804656519378, 1.4659660543453696, 0.42819269347992506, 0.30929164364610134, -0.3492711723798345, -0.8514748949632387 SwiGLU extends the activation idea using SiLU . It is — SwiGLU x = SiLU xWg × xWu . After above operation, a final projection brings the representation back to the model dimension. This becomes useful when the customer domain has many overlapping concepts, multiple meanings, complex context, exception rules, long relationships etc. For example, in a legal or enterprise assistant, the word “ approval ” may refer to - legal approval orfinance approval ormanager approval orregulatory approval orexternal approval orany other approval process The feature with its relevance changes depending on the current context. Here feature controlling becomes intuitively useful. There is an important engineering point. SwiGLU require more infastructure — more projectionsmore parametersmore computationdifferent initialization considerationshigher memory use If customer’s system have fixed intents, short questions, small dataset etc… then a simpler ReLU or GELU model may provide better business value. So the better thinking is — The customer has context-sensitive feature interactions, so we should assume that a controlled FFN may help and we will test that hypothesis. As above code explanation , here are some useful code snippet for SwiGLU, which can be found i n Github Repository. https://github.com/piyasde/article companion python from mini transformer import ACTIVATION NAMES, NEXT WORDS, count feed forward parameters, format vector, run transformer, ....EXERCISES = "cats drink", "milk" , "dogs drink", "water" , "I like", "tea" , ....def cross entropy for expected word probability : """For a one-word target, loss = -ln probability assigned to the target .""" return -math.log max probability, 1e-15 ....def show comparison prompt, target, activation names, temperature : .... for activation name in activation names: result = run transformer prompt, activation name, temperature probabilities = result "next word probabilities" top word index = max range len probabilities , key=lambda i: probabilities i target index = NEXT WORDS.index target p target = probabilities target index loss = cross entropy for expected word p target parameter count = count feed forward parameters activation name formatted probabilities = " ".join f"{number:.3f}" for number in probabilities ....def silu number : return number sigmoid number ....def run transformer prompt, activation name="relu", temperature=1.0 : .... final vectors = for token vector in after attention: details = feed forward token vector, activation name feed forward details.append details residual sum = vector add token vector, details "output" final vectors.append layer normalize residual sum ....def feed forward input vector, activation name : .... if activation name in "glu", .... : gate pre activations = linear input vector, GATE WEIGHTS, GATE BIAS .... for unit index, first branch value in enumerate input pre activations : .... gate = gate pre activations unit index activated = silu first branch value gate .... activated units.append activated print "Activation units are - ",activated units output vector = linear activated units, FEED FORWARD OUTPUT WEIGHTS, FEED FORWARD OUTPUT BIAS Activation units are - 2.815444702643749, 0.052585312093312096, -0.2645649263075179, -0.05701958653148752, 0.047145516636503355, -1.1773280335221021 Activation units are - 0.2259350766078342, 1.8208016659637194, -0.8741443667445493, -0.7458092598759276, -0.02992290648368431, -0.34554301678727567 After customer interviews, collect representative questions. If we gather 1,000 examples, We may group them like this - Now we translate these into computational needs. Factual Questions A factual question needs - php semantic understanding - retrieval - evidence ranking - answer generation This tells us that the quality of embeddings, retrieval, context window, internal representation, generation etc. may matter more than simply changing one activation function. Yes/No Questions For “Is this expense allowed?” we may produce a hidden representation h . Then z = Wbh + b, Then P = sigmoid z . If P = 0.91 the system can interpret this as 91% probability of “allowed”. This is a classic use of Sigmoid. Intent Routing If the system must route to one of HRFinanceITLegal The Transformer produces h hidden weight . Then z = Wch + b. Suppose z = 3.1, 1.2, 0.4, 2.1 . Softmax converts this into something like - HR 0.61Finance 0.09IT 0.04Legal 0.26 Now HR is the strongest choice. Softmax makes sense because the classes compete. Multi-Label Requirements If the customer says “My laptop is broken, I cannot log in, and I need a replacement.”, the relevant labels might be - hardwareauthenticationreplacementbilling These are not mutually exclusive. We want - hardware 0.96authentication 0.89replacement 0.91billing 0.06 Here we use independent Sigmoid outputs. Softmax would be wrong because it would force the labels to compete. Generation For a generative assistant, the model produces scores for vocabulary tokens. If the next-token logits are - because 8.1due 7.2from 4.1elephant -3.2 Softmax converts these into probabilities. Then the model chooses or samples the next token. This process repeats token by token. Vocabulary Softmax is a different use of Softmax from intent classification. It is having same mathematical family with different role. Let us think, we have a RAG system. We compute a relevance score between a question q and document d . If the question is — “Is this document relevant?” — that is an independent yes/no decision. Here Sigmoid makes sense. We might get - Doc 1 = 0.93Doc 2 = 0.71Doc 3 = 0.09 But if the question becomes — “Which document is best among these five?” then the scores of the documents compete. Softmax becomes more appropriate. This distinction is useful even beyond language generation. For enterprise assistants a stronger pipeline is - Question → Transformer Understanding → Retrieve Evidence - Generate Answer - Verify Answer - Accept / Retry / Abstain The verifier may output P supported using Sigmoid . For example - 0.96 → answer0.68 → retrieve more0.31 → abstain In a real system, improving verification layer may produce more business value than changing GELU to SiLU. That is an important design lesson. A common mistake in AI projects is optimizing before we have a stable baseline. We should start activation function implementation with simple one - ReLU Then measure - training lossvalidation lossaccuracylatencymemory useparameter countbusiness-task performance Then we should test with GELU, t hen SiLU, t hen SwiGLU , which indicates towards change one important thing at a time. If 80% of customer questions are factual or yes/no and other 20% contain exceptions, a model may score 90% and fail to make right answer on the questions that are complex with reasoning. So the benchmark should be split into categories. For example - Factual accuracyYes/No accuracyException accuracyComparison accuracyLong-context accuracyAmbiguity resolution We consider a normal question — “ What is the payment period? ” This may have a factual answer — 30 days . Now we have an exception question — “ Does the 30-day payment period still apply if the invoice is disputed and the supplier has failed to deliver? ” This question requires understanding conditions and relationships between clauses. A model with high overall accuracy but low exception accuracy may appear reliable in test period but may producing incorrect answers in complex situations. If the HR assistant has 1,000 benchmark questions - 300 factual200 yes/no200 explanation150 comparison150 policy exception We may train four variants of FFN Activation Functions. Now if we find something like - Here the trade-off becomes a business decision. If accuracy on exceptions is critical, SwiGLU may be worth the additional compute. If the model must respond under 45 ms on CPU, GELU may be a better choice. That is how activation-function selection becomes engineering decision than choosing terminology. Collect real examples and classify them. Examples are - FactualYes/NoMulti-labelRoutingComparisonExplanationRecommendationGeneration Ask - Is this classification?Ranking?Generation?Binary decision?Multi-label?Retrieval? For example - Embedding - Positional Encoding - Multi-Head Attention - FFN + ReLU - Output Binary → SigmoidMulti-class → SoftmaxMulti-label → SigmoidGeneration → Vocabulary Softmax Compare - ReLUGELUSiLUSwiGLU Measure - accuracyF1precisionrecallretrieval qualitygroundednesshallucinationlatencymemorycost The better equation for solve the Question-Answer modelling is - Business Value = Accuracy + Reliability + Latency + Cost + Maintainability In the Transformer, Attention is the research component. It asks — “Where should I look?” The feed-forward network is the analyst. It asks — “What does this information mean?” ReLU, GELU, SiLU and SwiGLU affect how the analyst FFN transforms uncertain information. Sigmoid is the yes/no risk determination factor. It asks — “How likely is this?” Softmax is the ranking factor. It asks — “Which option deserves the highest probability?” The output head is the business department. It asks — “What form should the final decision take?” This mental map may give useful architecture map. One chain is business-oriented — Customer Question → Mathematical Task → Model Head → Output Activation. The second is representation-oriented — Language Complexity → Transformer Representation → FFN Design → Internal Activation or Controlling. Keeping the above two chains makes activation-function optimization easier to understand. It helps us to take decision onuse activation functions which are not straight forward implementation of business rules. Activation functions are mathematical mechanisms that help the model transform and express information. The business requirement tells us which mathematical mechanism we need and where we need it . How Activation Functions Shape an AI Assistant https://pub.towardsai.net/how-activation-functions-shape-an-ai-assistant-1cac32b119be was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.