# Building a Private AI Diary with On-Device Speech Transcription

> Source: <https://dev.to/congcong/building-a-private-ai-diary-with-on-device-speech-transcription-5bim>
> Published: 2026-07-11 13:48:13+00:00

Writing a diary sounds simple until the blank page appears at the end of a long day. The friction is usually not a lack of things to say. It is the work of opening a keyboard, choosing a structure, and turning a messy thought into a sentence before the thought disappears.

That is the problem behind [You Speak, I Journal](https://nishuowoji.top), a voice journal app for iPhone. The product starts with speech instead of a blank text field: speak naturally, let the phone transcribe the recording, and then ask AI to organize the transcript into a journal entry, todos, reminders, and moments worth remembering.

This article explains the design choices behind that workflow, especially the boundary between on-device speech transcription and server-side AI organization.

Speech recognition and journal organization are related, but they do not need to happen in the same place.

The first job is transcription. The app needs to turn a person's voice into text while they are recording. Keeping this step on the device has practical benefits:

The second job is organization. A transcript may contain several topics at once: a meeting decision, a grocery item, a worry about tomorrow, and a small moment from the day. A language model is useful for separating those threads into a readable entry and a short list of concrete actions.

The important design decision is to send only the current transcript needed for that organization step. The app does not need to upload a complete diary archive every time a new note is processed.

"Private" is too vague to be a useful technical claim unless the boundary is specific. For this app, the boundary is:

This is not the same as claiming that no text ever leaves the device. The AI organization step needs the current transcript, so the product should say that plainly. Precise privacy language is less exciting than a sweeping promise, but it is much easier for users to understand and verify.

It is tempting to make an AI diary sound expressive by adding a strong assistant personality. In practice, that can make a personal record feel artificial. A diary transformation should stay close to what the person actually said.

The organization step therefore has a narrow job:

For example, a spoken note such as "I need to send the revised screenshots to Sam before Friday, and I was relieved that the review finally moved forward" contains two different kinds of information. The first is an actionable todo. The second belongs in the journal body. Keeping those outputs separate makes the result useful without turning the app into a full task manager.

A useful AI diary response is not just a polished block of prose. The app needs predictable fields so that each part can be shown in the right place:

```
{
  "diary": {
    "title": "The review finally moved forward",
    "summary": "A short reflection on today's progress"
  },
  "sections": [
    {
      "type": "daily",
      "title": "What happened",
      "content": "The review moved forward, and I felt relieved after the delay."
    }
  ],
  "todos": [
    {
      "title": "Send the revised screenshots to Sam",
      "scope": "today"
    }
  ],
  "reminders": []
}
```

The exact schema can evolve, but the principle is stable: the model produces structured data, while the app owns presentation, local persistence, editing, and deletion. This separation also makes it easier to test whether a revision changed the user's meaning or accidentally created duplicate todos.

Voice capture happens in real life, not in a clean demo. A user may lose connectivity, stop speaking halfway through a thought, or receive an imperfect transcription. The safe fallback is to keep the transcript or a local draft whenever possible instead of replacing it with an error screen.

The same principle applies to AI output. If organization fails, the original text should remain available. The user should be able to retry, edit the draft manually, or save the note without AI. An AI diary should make the model optional at the point of failure, not make the user's memory depend on a successful API call.

The most important part of a voice journal is not the model's writing style. It is the reduction of friction between an experience and a saved memory.

On-device speech transcription helps someone start talking. A narrowly scoped AI organization step helps turn that talk into something readable and actionable. Local-first storage gives the finished entry a clear home. Together, those choices create a calmer workflow than asking the user to write, organize, and categorize everything at once.

That is the direction we are taking with [You Speak, I Journal](https://nishuowoji.top): speak first, organize second, and keep the finished journal close to the person who wrote it.