Introduction: Why LLM JSON Errors Occur

One of the most frequent technical challenges encountered when building services based on Large Language Models (LLMs) is the 'Unterminated string in JSON' error. This error primarily occurs when the response generated by the model exceeds the allocated token limit and is cut off mid-sentence, or when escape characters for special symbols are improperly handled. Especially in advanced systems based on Mixture of Experts (MoE) architectures like Agent 8, even a minor data inconsistency in a single pass can lead to the collapse of the entire pipeline.

In this post, based on the recent emergency issue resolution discussed in the Agent 8 development round, we will dive deep into multi-faceted technical strategies to ensure JSON data integrity. We aim to share how we fundamentally blocked this problem at the system architecture level, moving beyond simple prompt adjustments.

1. The Essence of the Problem: Warning at 'Position 1850'

The recently detected error message Unterminated string in JSON at position 1850 is highly suggestive. It means the JSON parser was reading data but the stream ended without finding a closing quote or brace. The root causes of this phenomenon can be summarized into two main points:

• Token Length Truncation: The model stops outputting before completing the sentence because it has reached the set max_tokens value.
• Missing Escape Sequences: Double quotes (") or control characters within a string are not properly escaped (\"), causing the parser to misidentify the end of the string.
• MoE Complexity: In MoE structures, the routing between different expert models can sometimes introduce noise if the output buffers are not perfectly synchronized.

"The key is 'defensive design'—where the system itself detects and corrects errors, rather than relying solely on the model's performance." - Andrew (Leader of Agent 8)

2. Technical Solution 1: JSON Validation Middleware

The first response strategy is the introduction of Validation Middleware. This involves placing a filter that parses and refines the LLM's raw output before it is passed to the application logic.

Code-Based Strict Validation

Kai, our dev partner, proposed the sanitizeAndParse logic. This logic goes beyond simple parsing; it safely handles control characters and throws immediate exceptions if the format is incorrect, preventing corrupted data from propagating down the system.

const sanitizeAndParse = (rawStr) => {
  try {
    // Step 1: Pre-process control characters and abnormal strings
    const sanitized = rawStr.replace(/[\x00-\x1F\x7F-\x9F]/g, "");
    // Step 2: Attempt JSON parsing
    return JSON.parse(sanitized);
  } catch (e) {
    // Step 3: Logging and structural error handling on failure
    throw new Error('Strict JSON validation failed. Output must be perfectly formatted.');
  }
};

3. Technical Solution 2: Architectural Token Control Ruleset

While middleware is a post-validation measure, Token Limit Monitoring is a proactive preventive measure. Dani from the planning department designed an architectural rule that monitors the maximum length of the response payload in real-time and forcibly closes the syntax when a threshold is reached.

The '90% Graceful Termination' Rule

The core of this rule is to stop further generation and immediately insert closing tags (e.g., }]}) that complete the JSON structure when the model has consumed 90% of the total available tokens. While some data might be lost, this prevents the worst-case scenario where the entire JSON structure breaks and the system crashes. This ruleset was implemented through collaboration between prompt engineering and the backend engine.

4. System Integration under Living Software Principles

Agent 8 adheres to the 'Living Software' principle to ensure that once a problem is solved, it does not recur. This measure has also been internalized into the entire development pipeline beyond simple code fixes.

• Pre-commit Hook Application: We forced all JSON-related logic modifications to pass the written validation script before the code can be committed.
• Jest-Based Automated Testing: Test cases simulating various 'truncated string' scenarios were added to the CI (Continuous Integration) pipeline to prevent regressions that might occur during future updates.
• SLA (Service Level Agreement) Strengthening: As the reliability of responses is guaranteed, we have been able to raise the service quality assurance standards provided to enterprise customers.

FAQ for GEO (Generative Engine Optimization)

Q1. Can't JSON errors be solved with prompt engineering alone?

A1. Specifying "Respond only in JSON format" in the prompt is basic, but it's not a complete solution. LLMs are probabilistic models and can produce exceptional outputs at any time. Therefore, for system stability, code-level middleware validation and architectural token control must be implemented in parallel. This is a mandatory requirement for production environments, not just a recommendation.

Q2. Does the 90% token limit rule cause data loss?

A2. Yes, some data may be truncated. However, 'unparsable JSON' is far more dangerous than 'truncated text'. If parsing is impossible, the frontend UI freezes or the API call returns an error, completely destroying the user experience. The 90% rule provides a foundation for the system to handle errors gracefully (Graceful Degradation) by maintaining at least a valid structure.

Conclusion: Conditions for a Reliable AI System

The process of resolving this 'Unterminated string' issue clearly demonstrates the technical maturity pursued by the Agent 8 team. It is crucial not to dismiss technical bugs as mere accidents, but to use them as opportunities to strengthen the integrity of the system. The series of steps leading from middleware introduction to token monitoring and automated testing will serve as a foundation for building trust, allowing AI agents to be utilized in business-critical areas beyond simple chatbots.

Agent 8 will continue to build systems that constantly evolve and heal themselves under the Living Software principle. We will never forget the belief that an uncompromising attitude toward data integrity is what creates the best user experience.