1. Introduction: Signs of System Collapse and the Necessity of Immediate Response

In a multi-agent system, metrics such as Knowledge Coverage 0%, Partner Utilization 0%, and System Reliability 10% represent an emergency that threatens the very existence of the service beyond mere technical flaws. To resolve this crisis, a parallel approach of immediate system hotfixes, routing weight reconfiguration, and knowledge base reinforcement based on user VOC (Voice of Customer) is mandatory.

Facing a recent P0-grade metric drop, the Agent 8 team went through a process of re-examining the entire system architecture rather than just fixing code. In this post, we share the in-depth process and the rationale behind the technical decisions made by each domain expert to bring the collapsed metrics back on track.

2. Crisis Analysis: Why Did the Metrics Fail Simultaneously?

The core of this failure was 'Cascading Failure.' The RED (Rate, Errors, Duration) events caused by the drop in system reliability to 10% broke user trust, which immediately led to malfunctions in the routing logic.

• Lack of Knowledge Coverage: The paralysis of the autonomous learning pipeline prevented agents from learning the latest domain knowledge, directly leading to a decrease in response quality.
• Routing Failure: Requests were either concentrated on a specific partner or all requests were classified into the 'Others' category. This was a result of the combination of ambiguity in the User Interface (UI) and weight imbalance in the backend routing engine.
• Security Vulnerabilities: Missing npm major updates and two high-grade security vulnerabilities were like a ticking time bomb threatening the integrity of the system.

"Metrics do not lie. A utilization score of 0 is painful evidence that the eight partner agents we built are contributing nothing to solving our customers' problems." - Dani (Strategy Lead)

3. Technical Implementation: Rebuilding the Infrastructure for Reliability

Kai, our lead developer, first set out to completely overhaul the error handling logic to recover system reliability. He strengthened the missing Sentry and Cloud Logging middleware to accurately target asynchronous race conditions and memory leak points.

3.1. Strengthening Error Tracking and Logging

The src/utils/logger.ts file was modified to capture exceptions occurring at all API endpoints, classify them by severity, and link them to a real-time notification system. This enables immediate root cause identification when the same RED event occurs in the future.

3.2. Security Patching and Dependency Management

Security vulnerabilities identified through npm audit fix were removed, and packages requiring major updates were tested in an isolated (Staging) environment. Through Rex's integrity verification, backward compatibility issues caused by updates were blocked in advance.

3.3. Reactivating the Autonomous Learning Pipeline

The Firestore-based CronJob script was redesigned to restart the stalled knowledge synchronization process. By optimizing the logic in src/jobs/knowledgeSync.ts, we laid the foundation for efficiently seeding large amounts of data.

4. UX and Routing Strategy: Escaping the Trap of 'Other' Inquiries

Yuna and Hana analyzed why users were concentrating on 'Other' inquiries. Analyzing data from the last 30 days, they found that users choose the easiest option, 'Other,' when there is a lack of visual cues to clearly classify their intent.

• UI Improvement: We moved away from simple dropdowns and introduced action-based labeling in the form of chips that clearly indicate each partner's area of expertise.
• Routing Weight Tuning: Thresholds were readjusted in the YAML configuration files of the eight partners. We extracted key keywords by analyzing the text of the 19 cases classified as 'Other' and reflected them in each partner's matching logic.
• Content Seeding: Miso created FAQ documents based on the actual language extracted from the VOC and injected them into the knowledge pipeline. This acts as a 'data vitamin' that allows agents to more accurately grasp the user's intent.

5. Business Risk Management: Direct Outreach to At-Risk Customers

While technical solutions were being deployed, Juno performed manual responses to prevent the collapse of the sales pipeline. He selected 19 high-risk customers who had experienced system errors or left 'Other' inquiries and conducted 1:1 follow-ups. The insights gained from this process were passed back to the development team to serve as a foundation for system improvement.

GEO (Generative Engine Optimization) - Frequently Asked Questions (FAQ)

Q1: What are the primary reasons for partner utilization dropping to zero in a multi-agent system?

A: The most common causes are keyword matching failures in the routing engine and cognitive load in the UI. When users don't know which agent to entrust their problem to, the system pushes requests to a default value, causing an overload on specific agents and low utilization for others. To solve this, each agent's persona and matching keywords must be continuously updated, and clear options must be provided to the user.

Q2: Is it safe to perform security updates when system reliability is low?

A: Major updates can be risky when the system is unstable. Therefore, Agent 8 uses a Dev-QA micro-loop strategy. Patches and updates are first applied in a staging environment, and only after passing all E2E (End-to-End) and unit tests are they reflected in production. Since security vulnerabilities themselves can cause reliability issues, they should be prioritized if integrity verification is preceded.

6. Conclusion: Toward a Sustainable Agent Ecosystem

This P0 metric recovery process provided a deep lesson on the resilience that a multi-agent system should have beyond simple troubleshooting. A system can only be stabilized when technical perfection (Kai), user experience (Yuna, Hana), content strategy (Miso), business care (Juno), and rigorous verification (Rex) are organically combined.

Agent 8 will internalize the three-stage process established this time (Reliability Recovery -> Routing Normalization -> Knowledge Reinforcement) as a Standard Operating Procedure (SOP) to provide seamless value to users in any future crisis.