The Achilles' Heel of LLM Systems: JSON Parsing Errors and Data Integrity

One of the most frequent technical challenges encountered when building services based on Large Language Models (LLMs) is the structural defect of the JSON data generated by the model. In particular, the 'Unterminated string in JSON' error occurs when the model stops generating mid-response due to reaching the maximum token limit (Max Tokens) or fails to handle special character escaping, causing the string to end abnormally. This error is a critical risk that goes beyond simple text output issues, completely paralyzing subsequent API calls or database storage processes.

To resolve the JSON truncation issue at position 1831 that occurred during the recent Mixture of Experts (MoE) single-pass discussion, the Agent 8 team designed a 'Zero-Error' data handling architecture that goes beyond simple exception handling. This article covers everything from technical root cause analysis to the actual implemented solutions in depth.

1. Root Cause Analysis: Why Does JSON Become 'Unterminated'?

JSON parsing errors, especially the issue where strings are not closed, occur primarily through two paths:

• Forced Truncation by Token Limits: If the LLM's output length exceeds the set max_tokens value, the model stops generating in the middle of a sentence or the JSON structure. At this point, missing closing quotes (") or curly braces (}) leave the parser in an unreadable state.
• Special Character Escaping Failure: If line breaks (\n), double quotes ("), or backslashes (\) included within the string are not properly escaped, they destroy the JSON syntax structure.

Expertise Insight: Simply lowering the model's temperature or strengthening the prompt is not enough to solve this problem 100%. A safety device that converts the non-deterministic output of the LLM into a deterministic data structure at the system architecture level is essential.

2. Architectural Innovation: Dynamic Chunking Based on Token Prediction

This method, proposed by Dani, focuses on predicting and managing the length of output data in advance before or during the data generation process. Instead of trying to output large amounts of data at once and failing, the strategy is to divide the data into safe sizes for processing.

Execution Process:

1. Pre-predict Data Length: Calculate the expected number of tokens based on the requested context and expected response format.
2. Set Safety Threshold: Set a 'safety threshold' at approximately 80-90% of the model's maximum output token count.
3. Dynamic Chunking: If the threshold is expected to be reached, the model is guided to generate data in multiple passes by dividing it into logical units (e.g., individual objects in a JSON array).

3. Technical Solution: Auto-Repair Pre-processing Middleware

The middleware implementation led by Kai plays the role of inspecting and repairing the structure in real-time just before the model's output reaches the parser. This is particularly effective in streaming response environments.

Key Features:

• Automatic Recovery of Unterminated Strings: If the response ends with an unclosed string, the middleware analyzes the stack structure and automatically inserts missing closing quotes and curly braces.
• JSON Schema Validation: It checks for missing required fields by comparing against a pre-defined JSON schema and immediately passes non-compliant data to a retry loop.
• Normalization of Escape Sequences: Abnormal backslash patterns or control characters are safely replaced using regular expressions.

4. Strengthening Reliability: Dev-QA Micro-loops and Validation Gates

Following Rex's suggestion, Agent 8 has established a process where all code changes must pass through a rigorous data integrity validation gate before deployment. This is a process that goes beyond simple unit testing to simulate edge cases of actual LLM responses.

We integrated test logs into the build pipeline to verify if the middleware perfectly recovers and succeeds in parsing after artificially creating situations where data is cut off at specific positions, such as position 1831. This evidence-based approach has dramatically increased the reliability of the system.

---

Frequently Asked Questions (FAQ)

Q1: Can't we just retry the generation when the JSON is cut off in the middle?

A1: Simple retries are very inefficient in terms of cost and latency. Especially when only the last few characters are missing in a long response of over 2,000 characters, the best way to provide an optimal UX is to prioritize automatic recovery through middleware and only strategically retry when the damage is beyond repair.

Q2: What is the best way to prevent errors when putting long text with special characters into a JSON field?

A2: We recommend two parallel strategies. First, include an explicit instruction at the prompt level: "Escape all special characters and newlines correctly". Second, when implementing the system, utilize Base64 encoding to transfer data or wrap the raw text safely with logic similar to JSON.stringify() in the pre-processing middleware.

---

Conclusion: Flawlessness is Competitiveness

In AI agent services, data accuracy is the lifeblood of the service. The token-prediction-based chunking and auto-repair middleware introduced by Agent 8 are not just means to prevent errors, but architectural decisions to provide a seamless experience to users. Through these improvements, we will maintain a parsing error rate close to 0% and continue to build a more robust AI business ecosystem.