Skip to content

Reject UTF-16 surrogate values in Character converter - #438

Merged
garydgregory merged 1 commit into
apache:masterfrom
rootvector2:character-surrogate-reject
Aug 12, 2026
Merged

Reject UTF-16 surrogate values in Character converter#438
garydgregory merged 1 commit into
apache:masterfrom
rootvector2:character-surrogate-reject

Conversation

@rootvector2

Copy link
Copy Markdown
Contributor

the Character converter in putDefaultMap accepts \uD800 through \uDFFF because Character.isBmpCodePoint only checks whether the value fits in 16 bits, not whether it is a valid Unicode scalar value; surrogates are reserved for UTF-16 encoding pairs and casting one to char produces an isolated surrogate that corrupts downstream UTF-8 encoding or text normalization.

  • Read the contribution guidelines for this project.
  • Read the ASF Generative Tooling Guidance if you use Artificial Intelligence (AI).
  • I used AI to create any part of, or all of, this pull request. Which AI tool was used to create this pull request, and to what extent did it contribute?
  • Run a successful build using the default Maven goal with mvn; that's mvn on the command line by itself.
  • Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Each commit in the pull request should have a meaningful subject line and body. Note that a maintainer may squash commits during the merge process.

@garydgregory

Copy link
Copy Markdown
Member

Closing; duplicate of #436 , same question.

@rootvector2

Copy link
Copy Markdown
Contributor Author

fair enough on the dup. to answer the question here too: you never type the raw character, the converter parses the literal six ascii chars \uD800 as an escape via the startsWith("\\u") branch, so a plain-ascii --opt \uD800 argument is enough to get an unpaired surrogate char back.

@garydgregory garydgregory reopened this Aug 12, 2026
@garydgregory

Copy link
Copy Markdown
Member

Reopening: I see your point, thank you for replying. Reviewing again.

@garydgregory
garydgregory requested a lite review from Copilot August 12, 2026 15:27
@garydgregory garydgregory changed the title reject UTF-16 surrogate values in Character converter Reject UTF-16 surrogate values in Character converter Aug 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@garydgregory
garydgregory merged commit de0bd57 into apache:master Aug 12, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants