This post is the second installment of a series where we discuss quality at Powens. In this one, we’ll explain how we’ve started experimenting with AI agents to accelerate connector maintenance. Disclaimer: this is still at an early stage and subject to rapid changes.
As we explained in our first post, connectivity services like Powens (think Zapier but for connecting bank accounts) have very specific maintenance constraints. Imagine your software is, by definition, a collection of integrations with external systems you don’t control: it means that you literally import hundreds of issues, randomly. Banks update their websites. APIs change without notice. Every time that happens, a connector breaks – and someone needs to fix it.
In our previous post, we described how we prioritize which anomalies to fix first. It’s a traditional human process helped with smart monitoring tools. The system works well: it ensures that the right issues get attention. But it doesn’t solve a more fundamental problem: bandwidth.
Our connector maintenance team, X-Connectors, can only resolve a fraction of the average number of unique anomalies we detect each month. By focusing on the issues that impact the most users, we can ensure a stable service for 99% of them. But that means that we still leave many anomalies unfixed – even if they only impact a handful of users on smaller banks. And that is, of course, very frustrating for us and our customers.
Moreover, as some of you may know, when scraping financial data is involved, fixes can be particularly time-consuming. Reproducing the error, typically, is *slightly* more challenging than a typical software bug. Our security protocols forbid developers from using end-user credentials to facilitate debugging. We’re relying instead on browsing session logs and a series of internal tooling which help us pinpoint the issue. But it’s not perfect, and often, debugging is a trial-and-error process.
So here’s the million dollar question: can an LLM do that work? Can it help us fix the lower-priority issues that our developers have no capacity to handle – but also, eventually, speed up the resolution of higher-priority issues?
When you think about applying LLMs to web scraping, two fundamentally different approaches come to mind. And the choice between them reveals a lot about your priorities.
The first approach is to put the LLM in the loop at runtime. Every time you scrape a bank, the model interprets the raw HTML and extracts the data on the fly. Basically, the LLM becomes the scraper. No rigid code to maintain – the model adapts dynamically, for each bank website, and each browsing session. If the bank redesigns its website overnight, the model just figures out the new layout. Zero downtime. It’s elegant, and could be the long-term future of data extraction.
But today, it has two showstoppers:
The second approach is to keep hard-coded connectors, but have the LLM write fixes like a developer. The model analyzes the problem, produces code, and opens a pull request. A human developer reviews and merges. With that method, the cost drops to €5-15 per fix – a one-time expense. The code is deterministic once deployed. And every change goes through human review before reaching production – so we get better control.
It’s less cool, for sure. But we’re a company that handles millions of bank connections and operates under financial regulation: control isn’t a nice-to-have. So we made the obvious choice and went for “LLM fixes connectors”.
Now, here’s the twist: these two approaches aren’t mutually exclusive. We can already imagine a hybrid model where runtime extraction serves as a fallback for edge cases – catching the 5% of users who hit a rare page layout – while the code-based approach handles the steady state. But that’s a story for another post.
Powens’ connector infrastructure is built on Woob, an open-source framework we’ve been developing for years, but that is often mocked by new developers joining Powens. They find it arcane, rigid, and sometimes way too heavyweight for simple tasks. So when we first considered having an AI write Woob code, there was internal skepticism. Woob is not exactly mainstream. Could a language model really understand a niche framework well enough to produce reliable connectors?
An early proof-of-concept, run live on a real bank account, settled the debate rather quickly.
It turns out that a structured framework is an advantage, not a handicap. Woob’s core handles all the boilerplate: HTTP sessions, cookie management, pagination, error handling. A connector only contains the logic specific to the bank’s website: which page to visit, which fields to extract, how to navigate between sections. This means the LLM works on small, well-scoped files rather than monolithic scripts. And because all connectors follow the same patterns, the model can learn conventions from one and transfer them to another.
Without a framework, the model tends to produce ad hoc code – chained HTTP calls, inline error handling, spaghetti navigation. With Woob, it produces structured, maintainable code that a developer used to the framework can actually review in a reasonable amount of time. If you’re considering a similar approach for your own codebase, that’s probably the single most important takeaway: give the model guardrails, and the output quality goes up dramatically.
In the end, it took us a couple of months to come up with the first prototype of our AI connector fixer. We called it woob-autofix (crazy, right?). Its design principle is simple: the agent follows the exact same workflow as a human developer.
And then it stops. The agent never merges its own code. The X-Connectors team reviews every pull request exactly as they would for any human-authored contribution. Again, it felt obvious to us to take this strict design choice right now. Switching to a more flexible approach eventually is not excluded. But it will require a fair amount of learning and evaluation – and, obviously, hard evidence that woob-autofix can sustain a steady level of human-level performance over time.
Under the hood, the agent is powered by Claude via the Claude Code SDK. We initially ran both Claude Code and Kiro (AWS’s coding agent) in parallel. But Kiro lacked a native SDK, which meant spawning the executable, parsing its output, and losing context between steps. Claude Code’s SDK allowed seamless integration into our orchestration pipeline, with full context continuity. We made the pragmatic choice to consolidate – and the results improved.
When we wrote the first draft of this article, we were running on Opus 4.6, but the upgrade from 4.5 wasn’t as good as we had anticipated. Our 4.5 prompts were slightly too direct for 4.6, which requires a bit more leeway to produce good results. After some tedious prompt re-engineering, it got better. We’ve since gotten better at handling these upgrades. At the time of publication, we moved to Opus 5.5.
Ask anyone building AI-assisted developer tools what the biggest challenge is, and you may hear “prompt engineering” a lot. True, it’s a lot of work – probably most of it, quantitatively. But it’s not the hardest.
The truly hard problem is knowing whether you’re actually getting better. In most software projects, you measure AI code quality with unit tests or fixed scenarios. But we can’t do that, even if we solved all possible security challenges. Bank websites evolve constantly. A test case from three months ago may target a page layout that no longer exists, using a session for a user who has since closed their account. Our ground truth has an expiration date.
So we borrowed an idea from quantitative finance: backtesting. We go back in time, take a JIRA ticket from two months ago, check out the codebase as it was, and let the agent attempt the fix. Then we compare its pull request against the one the human developer actually shipped. An independent LLM judge evaluates both solutions against the original problem statement.
This approach caught something that test suites alone never would. Early on, the agent had a favorite trick: it would “solve” errors by catching exceptions rather than addressing the root cause. No more crashes, technically. But also no more data. By comparing against a human reference, the judge can tell the difference between a real fix and a band-aid.
Where do we stand today? The agent produces a fix at least as good as the developer’s in 25-30% of cases, on the first attempt, with no review feedback. Up from about 10% when we started. And we have reasons to believe we’re underestimating: early human reviews suggest the LLM judge is too strict in some cases. We’re running calibration rounds with the X-Connectors team right now.
Even where the fix isn’t perfect, the time savings are real. The agent takes 10-15 minutes. A developer takes one to two days. Even a fix that needs adjustments gives the reviewer a massive head start. And remember: those fixes would probably have never been done anyway. So, better done than perfect.
But when the fix is good, it can get really good.
In one case, a ticket reported that several accounts at a major French bank were incorrectly typed (meaning: we were mixing up between checking accounts, savings, or something else). The agent confirmed the issue, and then – without being asked – ran a web search for each account product name to determine the correct type. It identified a PEA (equity savings plan), a market account, and others, then updated the connector accordingly. A pure business-reasoning approach that none of us had explicitly designed for.
In another case, a connector couldn’t retrieve transactions for a specific account. The agent noticed that the URL pattern didn’t match the other pages. So it started experimenting: it reconstructed the URL by combining the account name and product type, tried several variations, found one that returned valid data, and implemented the parsing logic. A trial-and-error engineering approach – the kind of thing a developer would do with browser devtools open.
Neither behavior was hardcoded. The agent figured them out from context. And that’s precisely what makes this approach promising: it’s not a script with a thousand rules. It’s a model that reasons about problems.
All aggregation customers benefit from the development. In the previous sprint we spent 40% less time on maintenance and more than doubled the quality of bottom-tier connectors (improving success rate from 27% to 60%+). Looking six months ahead, customers can expect significantly shorter resolution times for connector issues and improved day-to-day service quality. — Vincent Paredes, Powens VP AI & Architecture
Today, woob-autofix handles scraping connectors in the Wealth vertical (savings & investment accounts, not provided by PSD2 APIs). We’ve started to extend it to PSD2 connectors, starting with account typing issues. Document retrieval should follow quickly.
The big unlock ahead is browser navigation. Currently, the agent can only work with pages that the connector already knows about – at least in most cases. So currently we’re building controlled browser access so the agent can discover new pages autonomously – with strict restrictions.
Further out, we want to close the loop entirely. Today, the agent picks up JIRA tickets – issues that have already been detected, triaged, and reported by humans. But our monitoring already captures crashes in real time. Imagine: a bank pushes a breaking change at midnight. The agent detects the error, proposes a fix, and opens a PR. A developer reviews it at 9 AM. By 10, the connector is live again. That’s where we’re heading.
Finally, we’re already experimenting with the counterpart of fixing connectors: building them. Imagine if any customer (or end-user) request to connect a very specific bank or platform, not yet supported, could be met in an hour. This might not be that far-fetched – but we’ll keep this for another post.
As you see, we’re not replacing developers with AI. We’re giving them leverage. The X-Connectors team still reviews every fix, still owns the codebase, still makes the judgment calls. But instead of spending two days on a parsing change, they can spend fifteen minutes reviewing a well-structured pull request – and redirect their expertise toward harder, higher-value work. And they’re happy to know that, finally, some lower-priority issues will be taken care of.
In our industry, trust is everything. Our clients depend on us for mission-critical financial data. That’s why we chose the approach that keeps humans in control while using AI where it shines: speed, pattern recognition, and tireless iteration. It’s the same philosophy that drives our tiering system, our monitoring, and everything else we build. Do the hard work. Be transparent about it. And never compromise on the controls.