FoxFlow

github.com/tedious-code/foxflow

High-performance data-import & pipeline engine — files, databases, and webhooks/APIs in, databases out. Think "small n8n," but built around batch streaming for throughput and reusing the FoxSchema dialect engine for schema-aware, auto-creating/evolving database targets across many SQL dialects.

Layout

npm-workspaces monorepo (mirrors FoxSchema's conventions).

• packages/engine (@foxflow/engine) — the workflow > pipeline > pipe model (zod), the DAG planners, and the batch-streaming runtime. A workflow orchestrates (triggers, overlap policy, completion edges between pipelines — with per-edge on: success|failure|always and gate settings, declarable inline via dependsOn); a pipeline streams batches through a DAG of pipes (source/transform/sink operators). Reuses @foxschema/core through the single seam in src/foxschema.ts (canonical type mapping + dialect DDL generation).
• packages/credentials (@foxflow/credentials) — encrypted credential management. Secrets are AES-256-GCM at rest and are never returned to clients; only the engine decrypts them, at run time.
• apps/api (@foxflow/api) — Fastify backend with zod validation/serialization. Workflow CRUD + validation (a bare pipeline document is implicitly wrapped in a single-pipeline workflow), credential management, run control.

Reusing @foxschema/core

Core ships as TypeScript source with no build step and lazy-loads native drivers, so it imports cleanly without any driver packages installed. During early development it is wired as a file: dependency pointing at the sibling FoxSchema checkout (../../../foxSchema/packages/core). Productionization = publish @foxschema/core to npm and replace the file: range with a version.

Develop

cp .env.example .env      # then set FOXFLOW_ENCRYPTION_KEY (see the file)
npm install
npm run typecheck         # tsc --noEmit across the monorepo
npm test                  # vitest
npm run dev:api           # start the Fastify API (default :3080)

Status

Initial scaffold. The engine's batch-streaming executor and the connector library are not yet implemented — see IMPLEMENTATION_STATE.md for the phased roadmap and what each milestone lands.