Skip to content

Don't add a newline when normalizing long whitespace runs in PO values - #1482

Open
imsurya26 wants to merge 1 commit into
python-babel:masterfrom
imsurya26:fix/normalize-long-whitespace
Open

imsurya26 wants to merge 1 commit into
python-babel:masterfrom
imsurya26:fix/normalize-long-whitespace

Conversation

@imsurya26

Copy link
Copy Markdown

normalize() adds a trailing \n to a value when a line contains a whitespace run at least as wide as the wrap width, so write_po() followed by read_po() changes the message:

>>> from babel.messages.pofile import normalize, denormalize
>>> s = "abc" + " " * 80
>>> denormalize(normalize(s)) == s
False
>>> denormalize(normalize(s))[-1]
'\n'

The same happens for "abc" + " " * 80 + "\n" (gains a second newline) and for tab runs.

WORD_SEP.split(line) returns an empty chunk after a trailing separator. When the whitespace chunk before it is too wide, it is put on a line of its own and the empty chunk becomes one more (empty) line, which the "remove empty trailing line" step then mistakes for the trailing line and turns into lines[-1] += '\n'. Empty chunks are now dropped before wrapping.

This also stops normalize() from emitting a stray empty "" continuation line in some wrapped values, so a few .po files written by write_po change slightly (still equivalent PO). Output for inputs without an empty chunk is unchanged; I checked this with a differential fuzz over a few hundred thousand random strings and widths.

Added test_normalize_roundtrip_with_long_whitespace (fails without the change). tests/messages and the pofile doctests pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant