You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
The tokenize() function only matched [a-z0-9_]+ patterns, returning an empty set for CJK-only text. This made Jaccard similarity always 0 (or always 1 for two empty sets) for CJK content, effectively disabling MMR diversity detection. Add support for: - CJK Unified Ideographs (U+4E00–U+9FFF, U+3400–U+4DBF) - Hiragana (U+3040–U+309F) and Katakana (U+30A0–U+30FF) - Hangul Syllables (U+AC00–U+D7AF) and Jamo (U+1100–U+11FF) Characters are extracted as unigrams, and bigrams are generated only from characters that are adjacent in the original text (no spurious bigrams across ASCII boundaries). Fixes #28000 |
8 months ago | |
|---|---|---|
| .. | ||
| memory | 8 months ago | |
| cli.runtime.ts | 8 months ago | |
| cli.test.ts | 8 months ago | |
| cli.ts | 8 months ago | |
| cli.types.ts | 8 months ago | |
| flush-plan.ts | 8 months ago | |
| prompt-section.ts | 8 months ago | |
| runtime-provider.ts | 8 months ago | |
| tools.citations.test.ts | 8 months ago | |
| tools.citations.ts | 8 months ago | |
| tools.runtime.ts | 8 months ago | |
| tools.shared.ts | 8 months ago | |
| tools.test-helpers.ts | 8 months ago | |
| tools.test.ts | 8 months ago | |
| tools.ts | 8 months ago | |