I Built an AI Studio That Turns 1 Line of Text Into a 5-Minute Short Drama

From idea to 1080p video — AI scriptwriting, storyboarding, multi-voice dubbing, and video compositing in one pipeline.

I spent the past month building Shortify AI, an open-source platform that takes a creative prompt like "a time-traveling maid from ancient China lands in a modern office" and outputs a full short drama episode — script, illustrated storyboard, multi-character voiceover, and a complete 1080p video.

Here's how it works under the hood, the architecture decisions I made, and the full pipeline that ties it together.

China's short drama market hit $70B in 2025. These are vertical, fast-paced 1–5 minute episodes — basically TikTok meets TV series. The production pipeline is:

Total: 5–10 people, 1–2 weeks per episode.

I wanted to compress this to: 1 person, 5 minutes.

Here's the end-to-end pipeline:

User Input ("穿越到现代的女将军")
       │
       ▼
┌─────────────────────┐
│   AI Scriptwriter   │  ← GLM-4-Flash / DeepSeek / Qwen
│  (characters +      │
│   scenes + shots)   │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Storyboard Images  │  ← Wan2.7-image / CogView-3-Plus
│  (1 image per shot) │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Multi-voice TTS    │  ← iFlytek WebSocket / Edge-TTS
│  (male / female /   │
│   narrator per role)│
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Video Compositing  │  ← FFmpeg (Ken Burns + AI video)
│  (1080p, subtitles, │
│   background music) │
└─────────┬───────────┘
          │
          ▼
    COS Storage + Share URL

Each stage runs independently and can be swapped — more on that below.

The LLM acts as a screenwriting assistant. Given a creative prompt, it generates a structured script with:

The prompt engineering was the hardest part. Early versions produced flat narration-style scripts. The breakthrough was switching to dialogue-centric formatting:

// Simplified prompt structure
const systemPrompt = `
Generate a short drama script in this format:
{
  "characters": [{ "name": "...", "gender": "male|female", "voiceId": "..." }],
  "shots": [
    {
      "shotNumber": 1,
      "character": "...",
      "dialogue": "...",
      "sceneDescription": "...",
      "cameraDirection": "close-up|wide|over-shoulder"
    }
  ]
}
Rules:
- Each shot = one line of dialogue
- Include scene descriptions for every shot
- Mark characters explicitly so we can assign voice models
- Total length: 7-12 shots per episode
`;

API: GLM-4-Flash (free tier), but any OpenAI-compatible LLM works via our model resolver layer.

For each shot, we generate an illustration. The challenge wasn't the image generation itself — it was managing cost and consistency.

Wan2.7-image (DashScope, ~$0.03/image)  → 2K resolution, synchronous
  └── fallback → Wanx-v1 (older, cheaper) → 720p, async polling
    └── fallback → CogView-3-Plus (Zhipu) → fallback with different API

For character consistency across shots, we inject appearance descriptors into every prompt:

function buildAppearancePrompt(shot, characters): string {
  const shotChars = characters.filter(c => shot.character === c.name);
  const appearanceDesc = shotChars
    .map(c => `${c.name}: ${c.appearance}`)
    .join(". ");

  return `${shot.sceneDescription}. ${appearanceDesc}. 
          Cinematic lighting, 16:9 widescreen, photorealistic.`;
}

Images are uploaded to Tencent Cloud COS (private bucket) with signed URLs for secure access.

This was the most fun to build. The LLM script tells us which character speaks in each shot, and we assign voice IDs accordingly:

const voiceMap: Record<string, VoiceConfig> = {
  "male-lead":    { edgeTTS: "zh-CN-YunxiNeural",  iFlytek: "x4_yehaoyun_oral" },
  "female-lead":  { edgeTTS: "zh-CN-XiaoxiaoNeural", iFlytek: "x4_shisan_oral" },
  "narrator":     { edgeTTS: "zh-CN-YunjianNeural", iFlytek: "x4_yunbai_oral" },
  "child":        { edgeTTS: "zh-CN-XiaoxuanNeural", iFlytek: "x4_yunxiaoyan_oral" },
};

Failure handling: The primary TTS (iFlytek WebSocket) sometimes rate-limits. In that case, the pipeline auto-falls back to Edge-TTS (free, runs locally). The entire voiceover stage runs at ~3 seconds per shot, so a 12-shot episode takes ~36 seconds for all dubbing.

This is where most of the engineering effort went. The compositing pipeline:

For each shot:
  1. Generate AI video (Wan2.7-t2v / CogVideoX) → or fallback to Ken Burns
  2. Mix voiceover audio → sync with video
  3. Speed-ramp video to match audio duration
  4. Apply fade-in/fade-out
  5. Generate SRT subtitles

Then:
  6. Concat all shot videos → episode
  7. Burn subtitles into final video
  8. Upload to COS
  9. Generate share URL

When AI video generation is disabled (or fails), we fall back to static images with camera motion. FFmpeg's zoompan

filter creates 10 different effects:

ffmpeg -y -loop 1 -i "image.jpg" -i "audio.mp3" \
  -filter_complex "
    [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,
          pad=1920:1080:(ow-iw)/2:(oh-ih)/2:color=black,
          zoompan=z='min(zoom+0.002,1.8)':d=240:s=1920x1080:fps=24,
          fade=t=in:st=0:d=0.4,
          fade=t=out:st=${(dur-0.4).toFixed(1)}:d=0.4[v];
    [1:a]adelay=1|1[a]
  " -map "[v]" -map "[a]" -c:v libx264 -crf 20 -preset fast -y "output.mp4"

10 camera effects: zoom-in, zoom-out, pan-left, pan-right, pan-up, pan-down, zoom-in-left, zoom-in-right, zoom-out-left, zoom-out-right. Each shot picks one in round-robin, making static images feel dynamic.

AI video generation has a ~10-15% failure rate (API timeouts, rate limits, content filters). The original pipeline simply showed a black screen for failed shots. The fix was a three-tier defense:

Tier 1: AI video generation → 80% success rate
Tier 2: Ken Burns zoompan → ~15% (catches most failures)
Tier 3: Static image + text overlay → 100% reliability

Tier 2 was the most fragile — the zoompan filter chain with complex expressions often failed on edge cases (very long/short audio, special characters in subtitles). Tier 3 uses ffmpeg's drawtext

to render the subtitle over a dark gradient background — it literally never fails.

We abstracted the entire end-to-end flow into a single CLI command:

npx tsx scripts/full-pipeline.ts

This reads the drama metadata from PostgreSQL, iterates over every episode and shot, runs the 4-stage pipeline, and uploads everything to cloud storage. The same script powers the web app's backend API.

Before (4 AI video generations): ~20 minutes per episode (5 min/shot × 4 shots + 1080p encoding)

After (Ken Burns only): ~2 minutes per episode

Early on, I chased the best AI video models (CogVideoX, Kling, Jimeng). But they all have ~10% failure rates, and a single failed shot ruins the entire episode. Investing in a rock-solid fallback chain that guaranteed no black screens was worth more than a 10% quality bump.

No single AI provider covers everything. We use:

The model config system lets users bring their own API keys for any stage.

AI content generation is getting fast and cheap. The bottleneck is now ffmpeg. A 12-shot episode with Ken Burns processing takes ~3 minutes just for the encoding. If you're building an AI video platform, invest in your compositing pipeline first.

Here's a 5-episode drama generated entirely by the pipeline:

The project is open-source under MIT:

git clone https://github.com/ycbing/Shortify-AI.git
cd Shortify-AI
npm install
npm run db:push
npm run dev

Or run the full pipeline directly:

npx tsx scripts/full-pipeline.ts

You only need a GLM API key to get started — everything else has free tiers or falls back gracefully.

Built with Next.js 16, FFmpeg, PostgreSQL, and a lot of API calls. Star on GitHub if you found this interesting.