Waterfall

Multi-platform video downloader written in Rust, modeled after JDownloader's link-crawler and plugin architecture. Waterfall resolves page URLs into direct download links, verifies availability, and transfers files with retries, resume, and parallel HTTP segments.

Repository: https://git.corespace.de/AsP3X/waterfall.git

Features

• Plugin pipeline — decrypters turn page URLs into DownloadLinks; host plugins download them
• Native X (Twitter) extraction — syndication APIs (vxtwitter / fxtwitter) with yt-dlp fallback
• YouTube, Reddit, and generic URLs — via yt-dlp
• Format presets — best, 720p, 1080p, audio-only, and custom heights
• Link checking — HEAD / Range probe before download (skippable)
• Resilient HTTP downloads — retries, resume, parallel Range segments for large CDN files
• Persistent queue — batch URLs and process them later
• Cookie support — Netscape cookie files or browser import for authenticated extractions
• Health checks — waterfall doctor probes dependencies and syndication APIs

Architecture

Waterfall follows a JDownloader-style pipeline:

URL → LinkCrawler → DecrypterPlugin → DownloadLink → HostPlugin → file

Phase	Role	Examples
Decrypter	Resolve a page URL into one or more download links	x, youtube, reddit, generic
Host	Transfer a resolved link to disk	direct_http, yt_dlp_host
Extractor	Low-level backends used by decrypters	Native X API, yt-dlp subprocess

flowchart LR
  CLI[waterfall CLI] --> Engine
  Engine --> Crawler[LinkCrawler]
  Crawler --> Decrypters[Decrypter plugins]
  Decrypters --> Links[DownloadLink]
  Links --> Check[check_availability]
  Check --> Hosts[Host plugins]
  Hosts --> File[output file]

Requirements

Tool	Required	Notes
Rust	Yes	Edition 2021; stable toolchain recommended
yt-dlp	For YouTube, Reddit, and yt-dlp fallback	Auto-bootstrapped to ~/.cache/waterfall/yt-dlp on first use if not on PATH
ffmpeg	Recommended	Needed by yt-dlp for merging video+audio; doctor warns if missing

On macOS, install dependencies with Homebrew:

brew install rust yt-dlp ffmpeg

Installation

From source

git clone https://git.corespace.de/AsP3X/waterfall.git
cd waterfall
cargo build --release

The binary is at target/release/waterfall. Add it to your PATH, or run via Cargo:

cargo run -p waterfall-cli -- --help

Initial setup

1. Create a config file (optional but recommended):

   waterfall config init
   

   This writes ~/.config/waterfall/config.toml. Edit it to set defaults.

2. Verify dependencies:

   waterfall doctor --verbose
   

   All required checks should pass. A missing ffmpeg produces a warning only.

Configuration

Settings are merged in this order (later wins):

1. Built-in defaults
2. ~/.config/waterfall/config.toml
3. Environment variables
4. CLI flags

Config file

# ~/.config/waterfall/config.toml

output_dir = "downloads"
format = "best"                  # best | 720p | 1080p | audio-only
max_parallel_downloads = 2
http_segments = 4                # parallel Range requests for large CDN files
http_max_retries = 4
skip_link_check = false

# cookies_from_browser = "chrome"
# cookies_file = "/path/to/cookies.txt"

Environment variables

Variable	Description
WATERFALL_OUTPUT_DIR	Download directory
WATERFALL_YT_DLP_PATH	Path to yt-dlp binary
WATERFALL_FORMAT	Format preset (best, 720p, audio-only, …)
WATERFALL_MAX_PARALLEL	Concurrent downloads
WATERFALL_COOKIES_FILE	Netscape cookies file path
WATERFALL_COOKIES_FROM_BROWSER	Browser name for yt-dlp cookie import
WATERFALL_USER_AGENT	HTTP user-agent string
RUST_LOG	Log level (e.g. waterfall=debug)

Data paths

Path	Purpose
~/.config/waterfall/config.toml	User configuration
~/.cache/waterfall/yt-dlp	Auto-bootstrapped yt-dlp binary
~/.cache/waterfall/queue.json	Persistent download queue

Usage

Download a video

waterfall download "https://x.com/user/status/123/video/1"
waterfall download -o ./videos --format 720p "https://youtu.be/VIDEO_ID"

Inspect without downloading

Resolve links and print JSON metadata:

waterfall info "https://x.com/user/status/123"
waterfall download --dry-run "https://youtu.be/VIDEO_ID"

Batch input

Provide URLs from a file (one per line, # comments allowed), positional args, or stdin:

# File
waterfall download --file urls.txt

# Stdin
cat urls.txt | waterfall download

# Combined
waterfall download --file more.txt "https://x.com/extra"

Example urls.txt:

# My download list
https://x.com/user/status/123/video/1
https://youtu.be/VIDEO_ID

Format presets

Preset	Description
best	Highest available quality (default)
1080p	Cap at 1080p
720p	Cap at 720p
480p / 360p	Lower resolutions
audio-only	Audio stream only

waterfall download --format 720p "URL"

Cookies (authenticated content)

waterfall download --cookies-from-browser chrome "URL"
waterfall download --cookies-file ~/.cookies.txt "URL"

Skip link check

Skip the pre-download availability probe for faster batch runs:

waterfall download --skip-check --file urls.txt

List plugins

waterfall plugins

Health check

waterfall doctor
waterfall doctor --verbose   # JSON report including syndication API probes

Download queue

Queue URLs for later processing. State is stored in ~/.cache/waterfall/queue.json.

# Add URLs
waterfall queue add "https://x.com/user/status/123"
waterfall queue add --file urls.txt

# List queue
waterfall queue list

# Process pending items
waterfall queue run

# Dry-run queue (inspect only)
waterfall queue run --dry-run

# Remove completed/failed items
waterfall queue clear

Supported platforms

Platform	Primary method	Fallback
X / Twitter	Native syndication API → direct video.twimg.com URLs	yt-dlp
YouTube	yt-dlp	—
Reddit	yt-dlp	—
Other URLs	yt-dlp (generic decrypter)	—

For X URLs, Waterfall does not require yt-dlp. For YouTube, Reddit, and generic pages, yt-dlp is required.

Troubleshooting

yt-dlp not found

Install yt-dlp or let Waterfall bootstrap it automatically:

brew install yt-dlp
# or set explicitly:
export WATERFALL_YT_DLP_PATH=/opt/homebrew/bin/yt-dlp

On first download that needs yt-dlp, Waterfall may download a release binary to ~/.cache/waterfall/yt-dlp.

X video not found

1. Run waterfall doctor --verbose and confirm syndication APIs are reachable.

2. Try with browser cookies if the tweet is age-restricted or private:

   waterfall download --cookies-from-browser chrome "URL"
   

3. Check resolved links with waterfall info "URL".

Slow or failing CDN downloads

• Increase retries: http_max_retries = 8 in config
• Enable parallel segments: http_segments = 4 (default)
• Skip link check for batch jobs: --skip-check

Debug logging

RUST_LOG=waterfall=debug waterfall download "URL"

Project structure

waterfall/
├── Cargo.toml                 # Workspace root
├── crates/
│   ├── waterfall-core/        # Library: engine, plugins, extractors
│   │   ├── src/
│   │   │   ├── crawler/       # Link crawler
│   │   │   ├── decrypters/    # URL → DownloadLink
│   │   │   ├── hosters/       # DownloadLink → file
│   │   │   ├── extractors/    # X API, yt-dlp bridge
│   │   │   ├── download/      # HTTP transfer, engine
│   │   │   ├── plugin/        # Plugin traits and registry
│   │   │   ├── queue.rs       # Persistent queue
│   │   │   └── urls.rs        # Batch URL loading
│   │   └── tests/             # Integration tests (wiremock)
│   └── waterfall-cli/         # `waterfall` binary
└── .cursor/rules/             # Agent and contribution conventions

Contributing

Contributions are welcome. Please follow the workflow and conventions below.

Branch model

Branch	Purpose
master	Stable release line
dev	Integration branch for completed features
feature/<name>	Feature work; branch from up-to-date dev

Intended flow: feature/... → dev → master (via pull request)

Getting started

git clone https://git.corespace.de/AsP3X/waterfall.git
cd waterfall
cargo test
cargo clippy -- -D warnings

Commit messages

Use one of these prefixes:

Prefix	Use for
TASK:	New features or refactors
FIX:	Regression from an earlier change
BUGFIX:	Pre-existing incorrect behavior
DOCS:	Documentation-only changes
CHORE:	Tooling and maintenance

Example:

TASK: Add native Reddit video extractor.

Uses the public JSON API for v.redd.it URLs before falling back to yt-dlp.

Do not commit secrets, target/, or other generated artifacts.

Code guidelines

• Match existing patterns in the crate you are editing

• Keep changes focused; avoid unrelated refactors in the same PR

• Add inline documentation for non-obvious logic (see .cursor/rules/inline-documentation.mdc)

• Do not use #[allow(dead_code)] — remove or use dead code instead

• Run tests and Clippy before opening a PR:

  cargo test
  cargo clippy -- -D warnings
  

Adding a plugin

Waterfall plugins implement traits in crates/waterfall-core/src/plugin/traits.rs:

1. Decrypter — implement DecrypterPlugin:

   • host_matches(&Url) -> bool — whether this plugin handles the URL
   • decrypt(&PluginContext, &Url) -> Vec<DownloadLink> — resolve page URL(s)

2. Host — implement HostPlugin:

   • host_matches(&DownloadLink) -> bool — whether this plugin downloads the link
   • check_availability(...) — optional pre-download probe
   • download(...) — write the file to disk

3. Register the plugin in Engine::with_defaults() (crates/waterfall-core/src/download/mod.rs).

4. Add tests — unit tests alongside the module; integration tests in crates/waterfall-core/tests/ using wiremock for HTTP backends.

Example layout for a new platform example:

crates/waterfall-core/src/
├── decrypters/example.rs    # URL matching + link resolution
├── extractors/example.rs    # Optional low-level API client
└── hosters/http.rs          # Often reused for direct CDN URLs

Testing

# All tests
cargo test

# Core library only
cargo test -p waterfall-core

# Integration tests (X API mocks)
cargo test -p waterfall-core --test x_api_integration

New behavior should include tests where practical. Regression fixes should add a test that would have caught the bug.

Pull requests

1. Branch from dev (or confirm the target branch with maintainers)
2. Keep PRs small and reviewable
3. Describe what changed and how you tested it
4. Ensure CI checks pass (cargo test, cargo clippy -- -D warnings)

Library usage

Waterfall can be used as a Rust library:

use waterfall_core::{Config, Engine, RunOptions};

#[tokio::main]
async fn main() -> Result<(), waterfall_core::WaterfallError> {
    let engine = Engine::with_defaults(Config::load()?)?;
    let options = RunOptions::default();
    let paths = engine
        .download_with_options(&["https://x.com/user/status/123".into()], &options)
        .await?;
    for path in paths {
        println!("{}", path.display());
    }
    Ok(())
}

License

MIT — see workspace Cargo.toml.