mirror of https://github.com/openai/codex.git synced 2026-10-18 17:18:52 +00:00

Files

T

jif a6f09397aa Compact the inline hidden tag buffer once per chunk (#49300 )

## Why

Parsing multiple tag delimiters in a chunk repeatedly drained the pending
buffer, shifting its remaining contents after each delimiter.

## What changed

Track a consumed offset in `InlineHiddenTagParser::push_str` and drain the
buffer once after processing the chunk, preserving partial delimiters for
the next chunk.

## Testing

Add a regression test comparing visible text and extracted tags against
single-chunk parsing at every UTF-8 character boundary. The input covers
non-ASCII text and delimiters, adjacent tags, an empty tag, and an unfinished
tag at end of input.

GitOrigin-RevId: 18bfbfdb81060ffda30901d5cb82ee5804878b09

2026-09-29 16:26:46 +00:00

src

Compact the inline hidden tag buffer once per chunk (#49300 )

2026-09-29 16:26:46 +00:00

BUILD.bazel

…

Cargo.toml

…

README.md

…

README.md

codex-utils-stream-parser

Small, dependency-free utilities for parsing streamed text incrementally.

Disclaimer: This code is pretty complex and Codex did not manage to write it so before updating the code, make sure to deeply understand it and don't blindly trust Codex on it. Feel free to update the documentation as you modify the code

What it provides

StreamTextParser: trait for incremental parsers that consume string chunks
InlineHiddenTagParser<T>: generic parser that hides inline tags and extracts their contents
CitationStreamParser: convenience wrapper for <oai-mem-citation>...</oai-mem-citation>
strip_citations(...): one-shot helper for non-streamed strings
Utf8StreamParser<P>: adapter for raw &[u8] streams that may split UTF-8 code points

Why this exists

Some model outputs arrive as a stream and may contain hidden markup (for example <oai-mem-citation>...</oai-mem-citation>) split across chunk boundaries. Parsing each chunk independently is incorrect because tags can be split (<oai-mem- + citation>).

This crate keeps parser state across chunks, returns visible text safe to render immediately, and extracts hidden payloads separately.

Example: citation streaming

use codex_utils_stream_parser::CitationStreamParser;
use codex_utils_stream_parser::StreamTextParser;

let mut parser = CitationStreamParser::new();

let first = parser.push_str("Hello <oai-mem-");
assert_eq!(first.visible_text, "Hello ");
assert!(first.extracted.is_empty());

let second = parser.push_str("citation>doc A</oai-mem-citation> world");
assert_eq!(second.visible_text, " world");
assert_eq!(second.extracted, vec!["doc A".to_string()]);

let tail = parser.finish();
assert!(tail.visible_text.is_empty());
assert!(tail.extracted.is_empty());

Example: raw byte streaming with split UTF-8 code points

use codex_utils_stream_parser::CitationStreamParser;
use codex_utils_stream_parser::Utf8StreamParser;

# fn demo() -> Result<(), codex_utils_stream_parser::Utf8StreamParserError> {
let mut parser = Utf8StreamParser::new(CitationStreamParser::new());

// "é" split across chunks: 0xC3 + 0xA9
let first = parser.push_bytes(&[b'H', 0xC3])?;
assert_eq!(first.visible_text, "H");

let second = parser.push_bytes(&[0xA9, b'!'])?;
assert_eq!(second.visible_text, "é!");

let tail = parser.finish()?;
assert!(tail.visible_text.is_empty());
# Ok(())
# }

Example: custom hidden tags

use codex_utils_stream_parser::InlineHiddenTagParser;
use codex_utils_stream_parser::InlineTagSpec;
use codex_utils_stream_parser::StreamTextParser;

#[derive(Clone, Debug, PartialEq, Eq)]
enum Tag {
    Secret,
}

let mut parser = InlineHiddenTagParser::new(vec![InlineTagSpec {
    tag: Tag::Secret,
    open: "<secret>",
    close: "</secret>",
}]);

let out = parser.push_str("a<secret>x</secret>b");
assert_eq!(out.visible_text, "ab");
assert_eq!(out.extracted.len(), 1);
assert_eq!(out.extracted[0].content, "x");

Known limitations

Tags are matched literally and case-sensitively
No nested tag support
A stream can return empty objects.