Skip to content

Do not raise ArgumentError on an argument with invalid bytes - #163

Open
ihamzatarar wants to merge 1 commit into
ruby:masterfrom
ihamzatarar:no-argument-error-on-invalid-bytes
Open

ihamzatarar wants to merge 1 commit into
ruby:masterfrom
ihamzatarar:no-argument-error-on-invalid-bytes

Conversation

@ihamzatarar

Copy link
Copy Markdown

Fixes #161.

Regexp#=== raises ArgumentError on a string that is not valid in its encoding, and parse_in_order matches every argument against the long and short option patterns before it can tell whether the argument is an option at all. So one bad byte anywhere in ARGV takes the parser down, including on an argument optparse would have returned untouched.

There are two sites, not one. The second is Switch#parse_arg, which matches the switch's own pattern against the value.

Change

Each site works on arg.b when the argument is not valid in its encoding. Option syntax is ASCII, so the structure still parses, and the bytes are passed on as they came in.

The only arguments this touches are ones that raise today, so no program that currently works can observe a difference.

Behaviour

With -n/--name=NAME and -v/--verbose defined, on Ruby 3.4.6:

argv before after
--bogus InvalidOption InvalidOption
caf\xE9.txt ArgumentError returned in the rest, bytes intact
--name=\xFF ArgumentError name set to \xFF
--name \xFF ArgumentError name set to \xFF
-n\xFF ArgumentError name set to \xFF
--verbose caf\xE9.txt ArgumentError verbose, and the file name in the rest
café.txt returned in the rest unchanged

Why pass the bytes through rather than raise ParseError

The issue asks for a ParseError, which would be an improvement over a backtrace, and I am happy to do that instead if you prefer. Passing the bytes through seemed closer to what Ruby already does: it hands arguments to ARGV as the bytes it was given, without validating them, and File.open takes them back. A file name from another system is a legitimate argument, so refusing it is a harder rule than optparse needs.

Either way this fixes the reported crash in ARGV.options { ... } and in ri, since both rescue ParseError and neither rescues ArgumentError.

One thing worth recording, which is why the second site needs the same treatment: on an invalid UTF-8 string, String#rindex does not raise, it simply fails to find a substring the string does contain. parse_arg relies on arg.rindex(s, 0) right after the match, so matching in binary and converting back before that line turns the crash into a spurious InvalidArgument.

Tests

test/optparse/test_invalid_encoding.rb, six cases. Five fail on master; the sixth asserts an unknown option still raises InvalidOption, and passes either way, so the change cannot swallow real parse errors.

Suite on Ruby 3.4.6, run standalone:

master   109 tests, 958 assertions, 0 failures, 2 errors
patched  115 tests, 985 assertions, 0 failures, 2 errors

Both errors are the same two, test_help_pager and test_ver, which need assert_in_out_err and assert_warning from ruby/ruby's test helpers and are not available when the suite runs outside the Ruby tree.

Regexp#=== raises ArgumentError for a string that is not valid in its
encoding, and parse_in_order matches every argument against the option
patterns before it can tell whether the argument is an option. A single
bad byte in ARGV therefore took the parser down, including on arguments
optparse would have returned untouched, such as a file name from a
system using another encoding. Switch#parse_arg had the same problem
when matching a switch's pattern against its value.

Match a binary copy at both sites. Option syntax is ASCII, so the
structure still parses and the bytes are passed on as they arrived.
Only arguments that raise today are affected.

String#rindex is the reason parse_arg needs this too: on an invalid
UTF-8 string it does not raise, it fails to find a substring the string
does contain, so converting back before that line would turn the crash
into a spurious InvalidArgument.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parse! raises ArgumentError: invalid byte sequence instead of a ParseError, even for non-option arguments

1 participant