Skip to content

Fix distributed cloud performance, security, and robustness concerns - #152

Merged
tomdesair merged 5 commits into
masterfrom
bugfix/cloud-robustness-security-performance
Nov 15, 2026
Merged

tomdesair merged 5 commits into
masterfrom
bugfix/cloud-robustness-security-performance

Conversation

@tomdesair

Copy link
Copy Markdown
Owner

Summary of Changes

This PR addresses performance, security, and robustness concerns identified when using tus-java-server in distributed cloud environments with slow or unstable client connections across all storage backends (Disk, S3, Azure Blob) and protocol versions (Tus 1.0.0 and RUFH Draft-12).

1. Lock Acquisition & Watchdog Leak Prevention on POST Requests

  • Issue: During creation-with-upload (POST /files with payload), the initial upload was streamed without acquiring an upload lock for the newly generated UploadId. If the client disconnected and sent a resume/terminate request to another instance, concurrent execution occurred without contention resolution. Furthermore, the active interruptible stream leaked into AbstractLeaseLockingService.activeInputStreams, causing perpetual 500ms watchdog stat polling.
  • Fix: Wrapped stream execution in CreationWithUploadPostRequestHandler and RufhCreationPostRequestHandler to acquire an UploadLock for the generated UploadId via acquireUploadLockWithRetry, registering the interruptible stream with lock contention resolution and ensuring timely cleanup in finally blocks upon completion or failure.

2. Stream Safety & Uncommitted Byte Rollback

  • Issue: On client stream drops, UploadInfo.offset or uploadInfo.length could be left in an inconsistent tentative state.
  • Fix:
    • In RufhCreationPostRequestHandler and RufhAppendPatchRequestHandler, cleanly reset tentative uploadInfo.setLength(null) when client streaming terminates prematurely, enabling subsequent resume requests.
    • In TusFileUploadService.processTusException, cap rollback to (uploadInfo.getOffset() - initialOffset) so that uncommitted temp chunks do not truncate earlier valid committed data.

3. Safe Deduplication Handling in Cloud Storage

  • Issue: Write and truncation operations must not resolve duplicatesUploadId to parent uploads.
  • Fix: In S3StorageService.removeLastNumberOfBytes and AzureBlobStorageService.removeLastNumberOfBytes, target only the child upload's own physical files/blobs, preventing parent data truncation. Extracted expiration cutoff calculation into S3Utils.calculateExpirationCutoff and AzureUtils.calculateExpirationCutoff.

4. S3 Compose HTTP Timeout Calibration

  • Issue: OkHttp default 10-second timeout caused transient timeouts during server-side UploadPartCopy on large multi-hundred MB or multi-GB parts.
  • Fix: Increased S3ServerSideComposeHelper read, write, and call timeouts to 60 seconds while preserving the permanent feature degradation logic for genuine unsupported API errors.

5. Security & DoS Hardening

  • Smuggling Protection: Enforced strict CRLF line endings, prohibited unexpected chunk extensions, and bounded maximum chunk size (2GB) in HttpChunkedEncodingInputStream.
  • Metadata Bounding: Capped Upload-Metadata header length (64KB) in UploadMetadataValidator to prevent memory exhaustion DoS spikes, while preserving backward compatibility for non-ASCII characters without spaces or commas.
  • Regex Compilation Cache: Replaced unbounded regex compilation with thread-safe ConcurrentHashMap caching in Utils.java.

6. Preliminary Validation in getRawInterimResponse

  • Issue: Pre-creating uploads for HTTP 104 interim responses before validation left orphaned phantom uploads on storage when requests were invalid (e.g. path traversal or size exceeded).
  • Fix: Added preliminary validation in getRawInterimResponse across all protocol versions. When validation fails, null is returned without touching storage, allowing the request to fall through to normal servlet processing where formal RFC 7807 problem details or Tus error responses are generated.

7. Testing & Quality Gates

  • Added testChunkedPatchWithTrailingContentDigestValidation in AbstractITRufhProtocol, running end-to-end chunked streaming with trailing checksum validation across Disk, S3, and Azure backends.
  • Added comprehensive unit tests for all new boundary conditions, validators, and handlers.
  • Code style: 100% compliant with Google Java Style (fmt-maven-plugin:check).
  • Diff coverage: 97.68% line coverage against origin/master (exceeding the 95.00% threshold).
  • Conformity test suites: 336/336 tests passed (Tus 1.0.0 and RUFH Draft-12 across Disk, S3, and Azure Blob).

…concerns

- Fix POST creation-with-upload lock acquisition and watchdog leak: ensure
  distributed lease lock is acquired for newly created uploadId and released
  cleanly on stream completion or abortion across Tus and RUFH.
- Enforce stream safety and rollback integrity: rollback uncommitted bytes
  on client stream interruptions and reset tentative length for RUFH deferred
  length uploads.
- Prevent parent upload truncation on child error rollback in S3 and Azure
  Blob storage services.
- Increase S3 server-side compose HTTP timeouts to 60s for large uploads and
  preserve fallback behavior.
- Harden against HTTP request smuggling and DoS: enforce strict CRLF and chunk
  size validation in chunked transfer encoding stream and bound metadata length.
- Optimize pattern compilation using ConcurrentHashMap caching.
- Add preliminary request validation in getRawInterimResponse to prevent
  phantom upload creation on invalid requests.
- Add integration test for chunked streaming with trailing checksum verification
  across Disk, S3, and Azure backends.
@coveralls

coveralls commented Oct 31, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 94.898% (+5.0%) from 89.87% — bugfix/cloud-robustness-security-performance into master

- Evict pre-lock ThreadLocal cache entries on lock acquisition in ThreadLocalCachedStorageAndLockingService
- Harden protocol handlers, validators, stream/header parsing, and Disk/S3/Azure storage, locking, and deduplication
- Add unit tests covering new edge cases and error paths
- Clean up CHANGELOG.md for 2.0.0 and update AGENTS.md and storage/architecture docs
@tomdesair
tomdesair merged commit 4b9adcf into master Nov 15, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants