The QA report came back clean. 

The developer had migrated an approval workflow module from a legacy stack to a modern stack. Parity check passed. Functionality verified. The page looked right, behaved right, and matched the spec. 

The bug report came in the next day. Data entered in that module wasn't propagating correctly to a connected downstream page — one nobody had been asked to test, one that wasn't in scope for the migration. This scenario isn't unique to migration projects. Any change in a complex, long-lived system carries the same propagation risk. The page they were given to test was fine. The problem was that nobody knew the change would affect a page three screens away. 

This is the gap that AI-assisted testing, by itself, cannot close. 

AI Tests What You Tell It to Test 

There's a version of AI-assisted QA that sounds transformative but is actually just faster manual testing. You describe the screen. You describe the interactions. The AI runs through them, flags issues, generates a report. It's more efficient than a human clicking through the same steps. It catches more than a rushed manual check. 

But it's still bounded by what you asked it to check. 

The fundamental constraint of page-level testing — whether it's manual, scripted, or AI-driven — is that it tests what's in front of it. It has no way of knowing that approving a status on this page changes a flag that controls button visibility on a page you haven't opened. It has no visibility into the data flows that connect modules together. It tests the surface. The connections are invisible. 

For straightforward applications, this is manageable. For systems that have been in production for years — where modules interact in ways that weren't fully documented, where business logic is distributed across services, where a single status field might drive conditional behaviour in five different places — page-level testing has a structural blind spot. 

The incident described above wasn't a one-off. It's the predictable output of a testing approach that's scoped to pages rather than change propagation. 

What Impact Analysis Actually Does 

The team built a different kind of QA capability. Instead of starting from the test cases for a given page, it starts from the change itself — and asks: what else in this codebase is connected to what just changed? 

The system reads the codebase at the code level. Not the spec, not the documentation, not the test suite — the actual code. It traces which other pages reference the same data, services, or conditional logic that the changed page touches. It produces a report: here is the change you made, here are the pages it will affect, here is how they're connected. 

For the migration scenario above, that report would have surfaced the connected downstream page before the developer pushed — not because someone remembered to include it in the test plan, but because the code said it was connected. 

That's the structural shift. From "test what you're told to test" to "understand what a change touches." 

The Report Is Only the Start 

Knowing which pages are affected is useful. Verifying that they're actually behaving correctly after the change is harder. 

Once the report identifies impacted pages, the team decides how to verify them — either navigating manually using the report as a guide, or instructing Claude to attempt verification directly. The latter is early-stage and works reliably only for pages with straightforward navigation paths and visible data fields. In practice, it catches a meaningful portion of impact issues automatically. 

What it doesn't always catch: pages where the navigation path requires specific session state that's hard to reconstruct, conditional UI elements that only appear when certain system parameters are enabled, and permissions-gated features where the relevant security function isn't active in the test environment. 

For those, the system does something more useful than pretending it covered them. It produces a list: here are the impacted areas I couldn't verify automatically, and here's why. Here are the security functions and system parameters that control the relevant behaviour. Go check these manually. 

That list is the difference between "we don't know what we don't know" and "we know exactly what still needs human eyes." 

What This Approach Can't Do 

This approach is more capable than page-level testing. It isn't complete. 

Session context limits mean that in longer automated runs, Claude can lose track of earlier state. Navigation paths through complex applications sometimes can't be resolved programmatically — the system knows a page is affected but can't find the route to it. Some business logic is only exercised under conditions that are hard to reliably reproduce in a test environment. 

The team doesn't paper over these limits. The manual verification list is a first-class output of the process, not a fallback for when things go wrong. The goal isn't to replace human judgment — it's to make human judgment precise. To narrow "you should check the whole application" down to "you should check these three specific things, for these specific reasons." 

That's a different kind of confidence than a passing QA report on a single page. 

The Test Pipeline That Makes This Systematic 

Impact analysis sits inside a broader QA pipeline built around AI assistance. It's worth understanding how the pieces connect. 

Before a developer ships a feature, the QA process starts from four inputs: the OpenSpec file from the developer's branch, the Jira story with acceptance criteria, the Figma designs, and the legacy application code. These get fed into a Claude skill that reads all four, surfaces any discrepancies between them, and generates a Functional Specification Document (FSD). If there are gaps or contradictions between the inputs, the skill raises them as clarifying questions before generating anything — catching requirements ambiguity before it becomes a test gap. 

That FSD becomes the input to the test case preparation stage. Test cases are generated from what the feature is supposed to do, not just what the UI looks like. After the developer completes implementation, a locator retrieval skill captures the actual UI elements from the live application. Those locators feed into the script generation stage, which produces Playwright test scripts against the real, deployed interface. 

The skill layer can improve with every sprint — but only when the team is disciplined about feeding missed cases back in. Every time a test run misses something that manual review subsequently catches, that missed case gets fed back into the skill with an explanation of why it was missed. The skill is updated. The next sprint, that case is covered. 

The pipeline is built as a Claude skill using the codebase as the core analysis input. It's a working approach being actively refined — not a finished product. 

What this means in practice: the QA capability compounds alongside the product, not ahead of it. 

The Question Your Test Suite Isn't Answering 

If your team is testing AI-assisted development output, the question worth asking is not "are our test cases thorough?" Thoroughness is a property of what you know to test. 

The question is: "When a change ships, do we know what it's connected to?" 

Most teams don't. Not because they're careless — because the tooling they use doesn't tell them. Page-level testing, even excellent page-level testing, doesn't surface connection-level risk. 

The bug three screens away was always there. The change propagation was always happening. The question was whether anyone would find it before or after production. 

AI doesn't know what your application is connected to. An impact analysis system does. That's the gap worth closing. 

CIQ builds and operates AI-assisted development practices for enterprise engineering teams. If your team is testing pages but not change propagation, that's the conversation worth having — get in touch.