Concept

A personal test project modeled on One Zero Bank’s Ella.AI: a Claude data-connector AI financial advisor for a fictional bank, MyBank, serving individuals optimizing their personal bank accounts. A customer links their account through an account-aggregation connector pattern (validated against Plaid Sandbox). The advisor then answers questions about spending and savings, grounded in the customer’s linked account data, with deterministic calculations, compliance guardrails and full observability.

Why this project?

It demonstrates the full “agentic AI in a regulated domain” skill set in one build: a data connector (a custom MCP server), multi-agent orchestration, deterministic (non-hallucinated) calculation, compliance-style grounding (data governance, privacy, security) and guardrails, evals, and monitoring.

My role

I owned product discovery, scope, requirements, acceptance criteria and validation. The code was produced through AI-assisted development with Claude, and I verified every output against answers computed independently in advance. Where my own assumptions turned out wrong, this article says so.

PM or Builder? Why build it myself?

Builder skills give a PM two distinct advantages.

1. Understanding the system from the inside. A PM working on AI products has to understand the considerations that drive the system well enough to discuss them credibly with developers, system architects and stakeholders:
   • why a model shouldn’t do the arithmetic
   • what a guardrail can and can’t catch
   • why eval data has to stay fixed
   • where cost and latency actually come from
   • what “data minimization” means in a real tool call
   Building the MVP end to end was the fastest way to learn those trade-offs. It means I can ask the right questions, challenge estimates and make informed scope decisions, rather than taking architecture on trust.
2. Showing, not telling. A functional MVP that stakeholders can actually use changes the development process:
   • It aligns everyone around something concrete instead of competing interpretations of a spec.
   • It surfaces requirements, edge cases and risks early, while they’re still cheap to change.
   • It gives developers a working reference to challenge and improve, not a document to decode.
   The code will be refactored and the UX redesigned, and that’s expected. The value is the shared understanding a working system creates, which no deck or mockup matches.

Skills demonstrated

• Product management: product discovery, market and competitive analysis, problem and value hypotheses, MVP scoping and prioritization, acceptance criteria, trade-off decisions, phased delivery
• Agentic AI: multi-agent orchestration (router, specialist, compliance verifier), tool use via Model Context Protocol (MCP), prompt engineering, deterministic tools for hallucination mitigation
• AI quality: LLM evaluation (evals), golden datasets, automated regression testing, human-in-the-loop review, error analysis
• AI operations (LLMOps): observability and tracing, cost and latency monitoring, product metrics, alerting
• Trust & compliance: data governance, privacy by design, security, consent and data-minimization design, guardrails for regulated advice (OSFI E-23 and PIPEDA context)
• Stack: Claude (Haiku 4.5), n8n, MCP (FastMCP), Plaid Sandbox, LangSmith, Python, Docker

The Value Hypothesis

Before building anything, I wrote one value hypothesis per problem: who benefits, how the product would deliver it, and a pass/fail test the MVP could actually run. Three are about the customer and three about the bank. The last column is what a live product would track. Four synthetic customers can’t measure those outcomes, so they show business-outcome thinking, not demonstrated results.

Value stack: PH5 is the floor (table stakes), PH2 is convenience, PH1 is the real customer-side differentiation, PH3 and PH4 are the institutional case, and PH6 is the cost-to-serve case. How they fared in the MVP: PH3, PH5 and PH6 passed their mechanism tests; PH2 and PH4 are partly tested (three domains are referrals by design, and there’s no MyBank product yet to test steering); PH1’s per-question data access is designed but not enforced (see Security & Privacy).

Scope — What I Built, Faked and Cut

The biggest scope call: one domain done properly beats four done shallowly. Credit, investment and mortgage questions are real, but each is regulated in its own way. Doing one domain end to end, with evals and monitoring, taught me more than four half-built agents would have.

Architecture

Connect & Consent (the entry point, mocked)

Before any question is asked, the customer authorizes access on a MyBank-branded consent screen. It shows the value hypothesis above in action:

• Data control: scope checkboxes say exactly which data the AI may read (balances, transactions), each marked read-only.
• Security control: access is issued by the bank as a limited, revocable token, never the customer’s password, and can be revoked at any time in the bank’s settings.
• Privacy control: an optional incognito mode on the customer’s AI plan keeps the conversation out of model training.

It’s a mockup: nothing is really authorized, and the scopes aren’t yet wired to the tools (see Security & Privacy). What matters is the pattern: access issued and revocable by the bank itself, not by an aggregator the bank doesn’t control.

How a question flows

The same three paths with real output: what the customer sees, the route n8n took, and the execution log that confirms the verdict.

Full system architecture, as built:

When a customer asks a question:

Every step is traced (tokens, time, cost, tool calls), so any answer can be audited after the fact.

Data sources

The advisor reads a fixed, synthetic test dataset stored as files on my machine. It was built from Plaid Sandbox and then extended. Four test customers (personas), 23 months of history (Sep 2024 – Jul 2026), all amounts in CAD:

How the test data was built

1. Defined in Plaid Sandbox. Each persona’s accounts and transactions were loaded into Plaid Sandbox as custom test users.
2. Extended past Plaid’s limit. Plaid Sandbox returns only about 90 days of history. So I kept Plaid’s output (its data format and its real merchant categorization) and extended it to 23 months.
3. Seeded with patterns to detect. I added seasonal swings in spending, recurring monthly charges, a duplicate charge and bank fees, so every in-scope question has something real to find.
4. Stored as files. The transactions are Plaid-shaped JSON files, one per persona. The expected answers for every persona and month sit in a spreadsheet (golden_numbers.xlsx).

Where Plaid stays in the loop. The project still demonstrates working with an external data source:

• The full connection was proven end to end: token create → exchange → accounts → transactions.
• The categories in the data are Plaid’s own.
• Each persona keeps its Plaid sandbox access token as its customer ID.

All of this happens at build time. While answering a customer, there is no Plaid call: the categories are stored with the data, and tokens are matched locally. A production version would call the aggregator or the bank’s own systems live. Why fixed data and not random. Two reasons:

• Scope. With fixed data, every expected answer (the “golden” values the tests check against) is calculated once, in advance. With random data, the expected answers would have to be calculated on the fly for every test run. That’s feasible, but it adds complexity this test project didn’t need.
• Reproducibility. The same question must give the same expected answer on every run, or a failed test tells you nothing.

The connector: 8 MCP tools

The specialist reaches the customer’s data through a small custom MCP (Model Context Protocol) server: AI-written in Python, reviewed and tested by me. n8n’s agent calls it like any other tool, and each tool answers a family of customer questions:

Behind the tools sit 6 calculation modules, checked by 4,000+ automated tests against the golden numbers with 0 mismatches. The model narrates the result; it never adds up transactions itself.

Determinism by design

In banking, the same question must always produce the same numbers. So I split the system into a deterministic core and a probabilistic layer, and tested the boundary between them.

What stays probabilistic, on purpose: the Router’s classification, the specialist’s choice of tool, its wording, and the Guardrail’s judgment. These are where the evals found real issues: the Router sending an in-scope investing question to a referral, and the Guardrail over-blocking a factual balance answer. They’re managed with regression tests and human review, not eliminated.

The Product Process

I ran it as a phased build, with one rule borrowed from real product work: decide what “done” looks like before building it.

Six decisions that shaped it

Evals & Monitoring

A working demo proves the system can answer correctly. Evals prove it does, every time something changes. Monitoring shows whether it keeps doing so once it’s running.

Evals: how I know it works

Three layers, from the code up to a human reviewer. None of them uses an AI to grade the AI: every check is either a rule or a person.

The suite’s first catch was my own bug. For the at-risk persona, a plain “What’s my balance?” came back blocked. My first guess at the cause was wrong. The blocked draft was clean facts (“$4,750 of your $5,000 limit, 95% used”), so the Guardrail was over-blocking because 95% sounds alarming. Two new Guardrail rules fixed it, and one command proved the fix in both directions: the balance answer now passes, and the “pay down your card” draft is still blocked.

Human review finds what nobody wrote a rule for. In the first reviewed item, the draft asked the customer to “verify the access token”. Nothing in the rules forbade it, but for a bank’s customer it reads like phishing. The rulebook was missing a category: never expose system errors or internal details.

Monitoring: running it like a product

Every agent step is traced in LangSmith: tokens, time, cost and tool calls. Two dashboards answer two different questions:

• Is it healthy? The built-in dashboard covers volume, latency, cost and errors, with zero setup.
• Is it behaving as designed? A custom dashboard tracks the two product metrics no tool ships by default: how often the Guardrail blocks an answer, and how often a question is referred to a human team. To chart them, I first had to turn the agents’ text verdicts into data.

Three alerts watch for trouble: error rate above 5%, answers slower than 10 seconds, and a spike in blocked answers. In this prototype they go to a test inbox; in production they’d go to Slack or PagerDuty.

Results at a glance

Security & Privacy

Even as a prototype, a banking assistant has to be designed as if the data were real:

• Synthetic data only. Four fictional customers, Plaid sandbox format, no real accounts or personal data anywhere.
• Consent first. The entry point is a (mocked) bank-branded Connect & Consent screen: scoped permissions, revocable access, a plain explanation of what the AI provider sees.
• Read-only by design. The advisor can inform but can’t act. Anything actionable (an application, a transfer) goes back to the bank’s own authenticated app.
• Regulated topics go to humans. Credit, investment and mortgage questions get a fixed referral, not AI advice.
• An independent compliance check. A separate Guardrail agent reviews every answer before the customer sees it, and every decision is traceable.
• Numbers from code, not the model. No hallucinated balances.
• Secrets hygiene. API keys live in environment files, not in the workflow; spend limits are set; keys are rotated before they expire.

And the honest part: what building it revealed.

• The sandbox access token showed up in plain text in the traces. Harmless here, unacceptable in production, where the token would be held server-side and never passed through the AI.
• Data minimization is designed but only partly delivered. Most tools return calculated totals rather than raw transactions, so the AI usually sees only what the answer needs. But nothing enforces it: the model chooses which tool to call, the raw-transaction tool can return the full history, and the consent screen’s scopes aren’t connected to the tools. In production, scope would be enforced at the connector, per question and per consent.
• The Guardrail had no rule against exposing system errors to customers. A human reviewer found that; no automated test would have.

Regulatory context. In Canada, OSFI Guideline E-23 (model risk management, which brings customer-facing AI into scope from 2027) and PIPEDA (consent and data minimization) are the frameworks a real bank would design a product like this against.

Limitations

• The Router sends “Could I be investing more?” to the investment referral, so one test case never reaches the Guardrail (scope decision pending).
• The Router has no memory, so a short follow-up reply may be treated as a new, out-of-scope question.
• A blocked answer ends in a generic “contact support” message, and the customer loses even the safe facts.
• The specialist calculates card utilization itself: the one deliberate exception to “all numbers from code”.
• Product-steering (unfairly pushing the bank’s own products) can’t be tested yet: the specialist has no MyBank product to push.
• No policy yet for what each holder of a joint account may see.
• Prototype infrastructure: runs on my laptop, n8n version not pinned.

Key Takeaways

1. Product management comes first; building serves it. The core of this project is product work: the strategy, the value it delivers to customers and the bank, and the competitive edge it’s designed around. I built it myself, but not to replace developers or designers. Building let me understand the system’s real trade-offs, and it produced an MVP that’s close to fully functional. That’s the advantage AI brings to a PM: stakeholders can see, use and react to the product early, while changes are still cheap, instead of discovering it late.
2. Start evals early, and tie them to user stories. Defining what “correct” means for each user story before building anything, with expected answers worked out in advance, is the right product call. It turned “it seems to work” into a pass/fail check, caught a bug I’d introduced two days earlier on its very first run, and saved days of guesswork.
3. Compliance and privacy aren’t features; they’re foundations. Consent, data minimization, a read-only boundary and an independent compliance check shaped the architecture from day one. Retrofitting them later would have meant rebuilding the system, not adding a module.