What to do in such situations: Developing an app integrated with generative AI

Hello, I’m pj-corridor, an engineer. In this article, I will introduce the challenges I faced while developing an application integrated with generative AI, such as dealing with abuse and navigating trial-and-error and trade-offs in prompt tuning. I previously had the opportunity to work with Synergy Marketing, and I am grateful that they kindly agreed to publish this article on the TECHSCORE BLOG 😊 Thank you very much.

First, please try the personality assessment app I initially developed. The annoying ads 😆 can be hidden by adjusting the window width.

Like the popular MBTI, CAPS and DiSC are so-called pseudoscientific personality assessments. They should not be used as a basis for decision-making, but for example, having workshop participants share their results can help energize self-introduction sessions and warm up the atmosphere. Please consider using this assessment app on such occasions.

For reference, here is my CAPS assessment result.

The advice and user manual are generated by AI, but the sudden appearance of “Respondent” in Japanese sentences feels a bit unnatural. In this way, while generative AI provides rich expressive capabilities, it also introduces quality risks. That said, I wanted to keep the cost of ensuring quality within the scope of a hobby project. Therefore, I kept the rule-based scoring logic deterministic and limited the generative AI output to supplementary text, thereby controlling the impact of hallucinations.

From here, I will introduce concrete examples of trial and error and how I approached trade-offs, divided into three chapters.

The assessment app can be accessed anonymously, but the backend uses a paid generative AI service (Amazon Bedrock). Therefore, being overwhelmed by bot abuse is a plausible scenario. However, rather than aiming to eliminate all threats, I adopted a basic policy of considering the attacker’s incentives: “accept threats with low manifestation risk, and mitigate the rest.”

For WAF, referring to AWS’s standard protection pack, I adopted:

GeoRule (blocking IPs from regions with frequent attacks)AWS-AWSManagedRulesAmazonIpReputationList (blocking known malicious IPs)AWS-AWSManagedRulesAnonymousIpList (blocking anonymized/tunneled IPs)GlobalRateBasedRule for the assessment app (request rate limiting)RateBasedRulePOST (limiting POST/PUT/DELETE requests) and removed unnecessary rules such as AWS-AWSManagedRulesSQLiRuleSet. However, when using a Lambda Function URL as a public endpoint, there is concern that requests not routed through CloudFront could bypass the WAF. While it is possible to enforce CloudFront (and thus WAF) routing, doing so would require AWS-specific implementations, such as:

AuthType to AWS_IAM and using CloudFront OAC 👉 This requires support for x-amz-content-sha256 in POST requests In any case, AWS-dependent implementation would be necessary. Considering the risk that a malicious actor could discover the Lambda Function URL and bypass the WAF, I decided not to implement this measure for now.

As a side note, settings configured in the AWS console are easy to forget. Forgetting to revert temporary settings can cause issues. To avoid this, I recommend managing managed service configurations in a repository (something close to IaC).

For example, for a WAF protection pack:

Also, as a follow-up: with the traffic scale of this app, the fixed cost of WAF ended up exceeding the usage-based cost of Bedrock. A good lesson in estimating costs 😅. Going forward, I am considering suspending WAF until traffic increases and instead implementing POST request limits using DynamoDB.

For access control, I implemented:

Set-Cookie credentials: 'include' Additionally, I applied:

SameSite=Strict to block cookies in cross-domain POST requests (see Access-Control-Allow-Origin to a whitelist to limit browser-based access These measures can be bypassed by more sophisticated bots, but I decided to start within the scope of the basic policy and consider additional measures while monitoring access patterns.

Here I describe trial and error related to Lambda and API Gateway.

Lambda was a reasonable choice for the runtime environment, but I initially preferred:

I leaned toward (1) because I already had a local PHP testing environment, which seemed suitable for agile development and testing. However, considering compatibility with Lambda and concerns about CI complexity, I ultimately chose (2). In hindsight, I spent more time dealing with AWS environment nuances and black-box behavior, so it turned out to be the right choice.

Speaking of trial and error, repeatedly creating Lambda functions via the AWS console automatically generates new IAM roles each time. Not just these remnants, but leaving unused resources can become technical debt, so they should be cleaned up promptly.

I resisted the urge to jump into development and prepared a mechanism to handle Lambda and test environments transparently.

This later became the foundation for prompt tuning.

As for whether to use API Gateway:

Given this, I decided not to use API Gateway at this stage, as the cost outweighed the benefits.

With abuse prevention and technology selection in place, it was time for development. Once completed, I wanted to share it globally via Reddit. That led to the motivation to support multiple languages, so I implemented a simple i18n class to support 9 languages.

const GREETING = i18n.text({
    en : 'Hello',
    ja : 'こんにちは',
    fr : 'Bonjour',
    de : 'Hallo',
    es : 'Hola',
    pt : 'Olá',
    hi : 'नमस्ते',
    ko : '안녕하세요',
    zh : '你好'
})

To stabilize AI output quality, I used the user’s native language for UI but fixed the AI input language to English.

I also unified state management, UI components, and interfaces with Lambda/test environments between DiSC and CAPS, enabling reuse for future assessments like MBTI.

Finally, I focused on improving and stabilizing AI output quality through prompt tuning. Instead of rushing, I first built a foundation:

During this process, I discovered missing outputs in hi, ko, and zh, likely due to token differences across languages.

Also, AI suggestions tend to bloat prompts with redundant instructions, so consolidating and refactoring prompt structure is recommended.

Finally, about “Respondent”: the requirement was to fix the subject as colleagues/friends while referring to the user consistently. Attempts like:

...use the {{lang}} term for "Respondent".

resulted in “you,” while banning pronouns led to “Respondent.” To stabilize output, I used:

always use the fixed keyword "_RESPONDENT_"

and replaced it client-side 😅

const RESPONDENT = i18n.text({
    en : 'Respondent',
    ja : '回答者',
    fr : 'Répondant',
    de : 'Befragter',
    es : 'Encuestado',
    pt : 'Respondente',
    hi : 'उत्तरदाता',
    ko : '응답자',
    zh : '受访者'
});

Thank you for reading. In this project, I focused on structuring prompt tuning while controlling hallucination impact. Future improvements may include automated output quality checks:

I hope my trial-and-error and trade-off decisions are helpful to you.