Skip to content

Fix int overflow of the chunk offset in uploadLarge for files larger than 2 GiB - #395

Open
buzzcosm wants to merge 1 commit into
cloudinary:masterfrom
buzzcosm:fix/upload-large-int-overflow
Open

buzzcosm wants to merge 1 commit into
cloudinary:masterfrom
buzzcosm:fix/upload-large-int-overflow

Conversation

@buzzcosm

Copy link
Copy Markdown

Brief Summary of Changes

Uploader.uploadLargeParts computed the start offset of each chunk as bufferSize * partNumber with two ints and widened the result to long only afterwards. Once the offset passes 2^31 bytes, the multiplication overflows, the Content-Range header gets a negative start offset (for example bytes -2134900736--2113929217/... with 20 MiB chunks) and the upload fails with Invalid Content-Range header. Details and a standalone reproduction are in #394.

The fix widens to long before multiplying:

long currentLoc = offset + (long) bufferSize * partNumber;

The new test UploaderUploadLargeTest runs offline in cloudinary-core: it plugs a recording AbstractUploaderStrategy into Uploader, uploads 2.3 GB from an InputStream that reports bytes without allocating them, and checks the Content-Range of every 20 MiB chunk. It takes well under a second.

What does this PR address?

Fixes #394

Are tests included?

  • Yes
  • No

Reviewer, please note:

  • One line of production code changed. No public API or behaviour change other than correct offsets above 2^31 bytes.
  • Without the fix, the new test fails at chunk 103: Content-Range of chunk 103 should start with 'bytes 2160066560-' but was 'bytes -2134900736--2113929217/-1'.
  • Run locally with JDK 1.8 (Corretto 8) and JDK 11: all 276 cloudinary-core tests pass, and the cloudinary-core, cloudinary-http5 and cloudinary-taglib jars build. The cloudinary-http5 integration tests need a test environment and were not run locally, they run in CI.
  • The same fix (offsets calculated as long) was verified against a real product environment with a 2.26 GB video and 20 MiB chunks: the unpatched SDK fails with Invalid Content-Range header, the file uploads completely with long offsets.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I ran the full test suite before pushing the changes and all the tests pass.

…than 2 GiB

uploadLargeParts computed the start offset of each chunk as
bufferSize * partNumber with two ints and widened the result to long
afterwards. Once the offset passes 2^31 bytes the multiplication
overflows, the Content-Range header gets a negative start offset and the
upload fails with "Invalid Content-Range header".

Widen to long before multiplying and add an offline test that uploads
2.3 GB through uploadLarge and checks the Content-Range of every chunk.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

uploadLarge sends a negative Content-Range start offset for files larger than 2 GiB (int overflow in Uploader.uploadLargeParts)

1 participant