Skip to content
DATA PROCESSING ENGINE
GitHub ↗

OPEN SOURCE / BUILT IN RUST

Complex
pipelines.
Clear execution.

Stream data through a graph of small, focused tools. Run it with Rust. See every stage in DPE Editor.

Start building Explore the editor ↓
npm install -g dpe
RUSTExecution engine
DAGExplicit dependencies
NDJSONStreaming contract
3 SDKsRust · Python · Bun

01 / THE PIPELINE, EXPOSED

See the flow.
Know where to look.

Open the graph, configure a stage, launch a run. Follow record counts and errors, then search the logs of the stage you need to understand.

DPE EDITORActual product capture / completed inventory run
DPE Editor displaying a completed inventory workflow with eighteen stages, branches and record counters.
01

Build visually

Typed stage settings and schema validation, backed by portable YAML variants.

02

Watch the run

Input counts, output counts, errors and execution state on each stage.

03

Inspect the evidence

Per-stage log search and session artifacts to understand the result.

Explore DPE Editor →

Desktop companion · early prerelease · runs your local DPE binary

02 / UNDER THE GRAPH

Small processes.
A powerful composition.

The graph describes the work. DPE connects tools over OS pipes and coordinates the run. Each tool receives structured input and produces the next stream.

Branch. Merge. Parallelize.

Route records by expression, merge upstream streams and run replicas for stages that benefit from parallel processing.

route / spread / replicas / fan-in

Process data as it flows.

Move NDJSON envelopes between stages. Use gates and disk-backed checkpoints wherever work needs an explicit barrier.

stdin → tool → stdout

Bring your language.

Write tools in Rust, Python or TypeScript with Bun. The SDKs provide output, metadata, tracing and diagnostics.

Rust / Python / TypeScript + Bun

Test before the full run.

Snapshot a stage in isolation. Compare emitted channels and file outputs, then check coverage across your variants.

dpe test / dpe coverage

03 / MAKE IT USEFUL

From raw files to usable data.

A / FILES & ROWS

Turn a directory into a dataset.

Scan files, read CSV or NDJSON, normalize records and write structured output.

B / CUSTOM PROCESSING

Make your domain logic repeatable.

Compose your classification, reconciliation or AI tools into a pipeline you can inspect and test.

04 / YOUR NEXT PIPELINE

Put your data
in motion.

Install the runner. Scaffold a pipeline. Start with the tools included, then add your own.

Get DPE on GitHub ↗

QUICK START / TERMINAL

npm install -g dpe

dpe init my-pipeline
cd my-pipeline

dpe run .:main \
  -i data/input \
  -o data/output \
  --seed '{"path":"$input"}'