Slashy's Cross-App Agent Architecture: Memory, Semantic Search, and Custom Tools

Slashy (YC S25) is a cross-app agent that reads data and executes actions across disconnected SaaS APIs. The architecture exposes three core primitives: custom tools, semantic search, and personalized memory. The demo shows an agent pulling financial data from one app, cross-referencing it with another, and triggering an action in a third without manual API stitching.

The interesting question is not what it does. The interesting question is how it maintains context, routes queries, and handles failure when one API rate-limit cascades into a multi-step workflow spanning three other services.

Slashy's agent layer sits between the LLM and a collection of app connectors. Each connector wraps a SaaS API (Slack, Google Calendar, Notion, financial platforms) and exposes a set of tools. The agent decides which tools to invoke based on user intent, then coordinates state across tool calls.

Custom tools are the agent's interface to external systems. Each tool is a function signature with input schema, output schema, and execution logic. When a user asks "What did I spend on AWS last month?", the agent:

The tool registry is versioned. If AWS changes its API, the tool definition updates without retraining the agent. This decouples the LLM from API churn.

Semantic search indexes data from connected apps. When the agent needs to answer "Find the email where Sarah mentioned the Q3 budget", it:

The search layer is not a unified index. Each app connector maintains its own embedding store. The agent queries multiple stores in parallel, then merges results by relevance score. This avoids the complexity of syncing all app data into a single vector database.

Personalized memory stores user preferences, past actions, and context from previous sessions. When the agent sees "Book a meeting with the team", it recalls:

Memory entries are scoped by user ID and app context. A memory entry from Slack does not leak into a Google Calendar tool call unless explicitly referenced. This prevents context pollution.

The agent maintains a session object that tracks:

When a tool call fails, the agent checks the error type. If it's a rate limit (HTTP 429), the agent s that tool and tries an alternative path. If it's an auth failure (HTTP 401), the agent prompts the user to reconnect the app.

The session object is ephemeral. It lives in memory for the duration of a task, then discards. Long-term state (user preferences, app credentials) lives in a persistent store.

The most dangerous failure is silent: the agent completes a task using stale data from one app and fresh data from another, then returns a confident but incorrect answer. Slashy mitigates this by timestamping every data fetch and surfacing data age in the UI.

class AgentSession:
    def __init__(self, user_id, connected_apps):
        self.user_id = user_id
        self.tools = self._load_tools(connected_apps)
        self.memory = self._load_memory(user_id)
        self.active_calls = []

    def execute_task(self, user_query):
        intent = self._parse_intent(user_query)
        plan = self._generate_plan(intent, self.tools)

        for step in plan:
            tool = self.tools[step.tool_name]
            try:
                result = tool.invoke(step.params, timeout=10)
                self.memory.store(step.tool_name, result)
                self.active_calls.append((step, result))
            except RateLimitError:
                self._handle_rate_limit(step)
            except AuthError:
                return self._prompt_reauth(step.tool_name)

        return self._synthesize_response(self.active_calls)

    def _handle_rate_limit(self, step):
        cached = self.memory.get_cached(step.tool_name)
        if cached and cached.age < 3600:
            return cached.data
        else:
            time.sleep(2 ** step.retry_count)
            step.retry_count += 1

This is pseudocode, not production code. The real implementation likely uses async I/O for parallel tool calls and a more sophisticated retry policy. The key point is that each tool call is isolated, errors are caught per-tool, and the agent decides whether to fail fast or degrade gracefully.

Each app connector runs with scoped credentials. The agent cannot access data from App A using credentials from App B. When a user connects Slack, the agent receives an OAuth token scoped to Slack's API. That token never touches the Google Calendar connector.

Memory entries are encrypted at rest and scoped by user ID. The agent cannot read another user's memory, even if both users connect the same apps.

Tool invocations are logged for audit. If a user asks "Did the agent access my bank account?", the log shows which tools were called, when, and with what parameters.

The agent emits structured logs for:

These logs feed a monitoring dashboard. If AWS's API starts returning 500 errors, the ops team sees a spike in tool call failures before users complain.

The agent also tracks token usage per LLM call. If a query triggers 20 tool calls and burns through 100k tokens, the cost is attributed to that user and surfaced in billing.

Slashy runs as a hosted service. Users connect apps via OAuth, then interact with the agent through a web UI or Slack bot. The agent backend is a Python service (likely FastAPI or Flask) that orchestrates LLM calls, tool invocations, and memory lookups.

The tool registry is a JSON schema stored in a database. When a new app connector is added, the schema updates and the agent picks it up without redeployment.

Memory and session state live in Redis for fast access. Long-term data (user profiles, app credentials) lives in Postgres.

At scale, Redis memory limits become a constraint. If session state grows beyond a few megabytes per user, the architecture needs a tiered cache (hot data in Redis, warm data in Postgres). LLM token costs also scale linearly with tool calls. A workflow that fans out to 10 apps can burn 50k tokens per query, which adds up quickly at high user volumes.

The hardest problem is maintaining context across long workflows. If a user asks "Find the invoice Sarah sent last week, then create a calendar event to discuss it", the agent must:

If step 2 fails (the invoice is a PDF attachment, not inline text), the agent cannot complete step 4. The user sees "I couldn't find the invoice" and has to start over.

Slashy handles this by surfacing intermediate results. If the agent finds the email but can't parse the invoice, it shows the email and asks the user to confirm the date manually. This turns a hard failure into a soft degradation.

Use Slashy's architecture when:

Avoid this architecture when:

The core insight is that cross-app agents are coordination engines, not data pipelines. They trade strong consistency for flexibility. If your use case demands both, you need a different architecture (event-driven workflows with durable queues, transactional guarantees, and explicit rollback logic). If your use case tolerates stale data and occasional retries in exchange for zero integration code, this pattern fits.

The architecture shines for financial research workflows: pulling transaction data from one app, cross-referencing it with invoices in another, and generating reports in a third. It struggles with high-frequency trading or real-time portfolio rebalancing where milliseconds matter and partial execution is unacceptable.