Why Architecture Matters In The Age of Agentic AI

Why Architecture Matters in the Age of Agentic AI

From a conversation between Scott Hebner and George Gilbert on The Next Frontiers of AI podcast

The Shift to Agentic AI: Digital Coworkers, Not Just Digital Assistants

We're witnessing a fundamental shift in artificial intelligence. AI is no longer just about assisting people in executing tasks—it's about helping people get actual work done. This includes knowledge work, planning, thinking, making decisions, and problem-solving. As George Gilbert, Principal Analyst at theCUBE Research put it, we're moving from digital assistants to digital coworkers.

Last week on the podcast, Haoyu Zha, CEO of successful agentic AI startup HOAi, shared three key success factors for the new agentic AI marketplace:

- Go deeply vertical

- Rethink what software is

- Target labor spend, not IT spend

The focus is shifting to domain-specific digital labor, where AI agents can perform meaningful work alongside humans.

Why Architecture Matters More Than Ever

Agentic AI is fundamentally a systems problem. The performance of AI agents depends on:

1. The quality of data about the business state

2. The agent's ability to reason about that data

3. The capability to operationalize decisions

This is precisely why architecture matters more than ever. As Gilbert explains, "You don't build a complex enterprise application because you wrote one microservice. There's a lot more to it. Agents are a little like microservices that think, but they've got to be part of a larger system."

This new architecture requires several emerging technologies beyond just LLMs and generative AI:

- **Vector databases and knowledge graphs** for better data retrieval and understanding

- **Agentic RAG (Retrieval-Augmented Generation)** for more dynamic knowledge integration

- **Multi-agent communication protocols** for coordination between agents

- **Tool APIs** for connecting to enterprise systems

- **Decision intelligence** for better reasoning and judgment

- **Learning loops** for continuous improvement

For most Fortune 500 companies, building this from scratch is unrealistic. This explains the projected 47% compound growth in agentic systems and platforms, as companies look to vendors who have already integrated these complex components.

The Evolution of RAG: From Traditional to Agentic

One key technology distinction is between traditional RAG and agentic RAG:

- **Traditional RAG** uses naively chunked vector embeddings with limited tool usage, primarily invoking a vector database and potentially reranking results.

- **Agentic RAG** represents a significant advancement. It can make plans, retrieve data, reflect on what it has retrieved, determine if it has all necessary information, and invoke additional tools as needed. It's more dynamic, making real-time decisions about what to invoke and what data is relevant.

Gilbert explains: "You're moving from putting all the control flow in symbolic code to more and more being dynamically planned and generated by the agent."

 The Power of Semantic and Causal Reasoning

The next frontier is enhanced reasoning capabilities:

- **Semantic reasoning** is based on knowledge graphs that help understand intent and context. It can fetch more contextually aware data and share semantic understanding within and among agents.

- **Causal reasoning** takes this to another level by understanding cause-and-effect relationships. This allows agents to make judgments based on understanding causal links between entities.

Gilbert places this in a larger context: "This is part of a movement where more knowledge of how the business works is learned by your data systems. You've always been setting up analytics to answer what happened, but the next questions are why did something happen, what's likely to happen, and what should we do?"

Types of Agents in an Agentic System

As organizations build agentic systems, they'll need different types of agents:

1. **Task Execution Agents** - Analyze data, execute tasks, identify patterns, detect anomalies, and make predictions

2. **Domain-Specific Agents** - Understand specialized fields like legal contracts with contextual knowledge and domain-specific LLM underpinnings

3. **Reasoning Agents** - Help think through problems, make better decisions, and perform counterfactual thinking

4. **Management Agents** - Plan and orchestrate how different agents work together, including humans

Gilbert notes that "We're mostly between levels one and two today," with domain knowledge typically added through RAG. The more sophisticated reasoning and orchestration capabilities are just beginning to emerge in products.

The Harmonization Layer: AI's Cognitive Core

One of the most critical components is what Gilbert calls the "harmonization layer" – where knowledge, reasoning, and goal setting converge.

He uses an analogy of autonomous vehicles: "Agents are like the self-driving cars we had in the desert 30 years ago. They could drive 100 miles alone but weren't coordinating with other vehicles or pedestrians. We still don't have pervasive autonomous driving because we don't have a digital four-dimensional map tracking everyone's trajectory."

In the enterprise, this harmonization layer becomes the shared database of the business state. Using the Disney theme park example mentioned by Salesforce CEO Marc Benioff, Gilbert explains: "You can't coordinate everyone's experience unless there's a shared database to say what the wait time is on all these different rides."

Getting Started with Agentic AI

For organizations just beginning their agentic AI journey, Gilbert recommends looking at your current platform providers:

1. Start with your enterprise application vendors (Oracle, Salesforce, SAP), who are building agents into apps to augment and automate tasks or workflows.

2. Next, explore your analytic data foundation, where you'll find agents to help data engineers build pipelines, assist data scientists with feature engineering, and enable business analysts to talk to data.

While these represent separate silos today, they're valuable starting points for organizations to begin integrating agentic AI capabilities.

The Golden Age of AI Agents

As Scott Hebner concluded, "We're entering the golden age of AI agents, a total reset on how software is designed, built, and used—perhaps the biggest since the e-business days 25 years ago with the advent of the internet."

This represents a fundamental shift where AI moves beyond assisting humans to becoming a true digital workforce that can think, reason, and act autonomously while coordinating with both humans and other AI agents.

The opportunity is no longer theoretical—it's real, growing, and available to those with the clarity of vision and urgency to act.

---

*This blog post is based on a conversation between Scott Hebner and George Gilbert on The Next Frontiers of AI podcast. For more insights, watch the AI Agent Builder Summit featuring 20 industry pioneers at thecube.net or on their YouTube channel, and don't miss the interview with Salesforce CEO Marc Benioff on how software and labor markets are changing due to agentic AI.*