High-performance log normalization pipeline.
Raw logs go in — from any provider, any format. Structured, canonical, token-efficient events come out. Lumber ingests logs from Vercel, AWS, Fly.io, Datadog, Grafana, Supabase and more, semantically classifies each line with a local embedding model, and emits uniform structured events. Zero runtime dependencies. Fully deterministic. CPU-only.
Logs arrive from any provider in any format — Vercel, AWS, Fly.io, Datadog, Grafana, Supabase. Lumber takes them all, raw.
A local neural network reads each line by meaning, not regex, and files it into exactly one of 42 known categories.
Fifty thousand tokens of log storm collapse into five hundred tokens of canonical, LLM-ready events. Deterministic. CPU-only.
A local embedding model. Forty-two labels. One deterministic schema — every log line in, one canonical event out, no cloud in the loop.
Traditional log parsers use regex — fragile patterns that break when formats change. Lumber classifies by meaning. Every log line is converted into a 1024-dimensional vector by a local neural network, then compared against 42 pre-embedded taxonomy labels via cosine similarity. A "connection refused" log and an "ECONNREFUSED dial tcp" log produce similar vectors and land in the same category — even though they share no words. The same approach as CLIP, applied to log classification.
Every log is classified into exactly one leaf of a curated taxonomy. ERROR covers connection failures, auth failures, timeouts, OOM. REQUEST covers the HTTP lifecycle. DEPLOY covers builds and rollbacks. SYSTEM, ACCESS, PERFORMANCE, DATA, and SCHEDULED round out the spectrum. A finite, known label set — opinionated by design — makes downstream consumption predictable and machine-friendly. No ambiguity. No uncategorized noise.
Output designed for LLM consumption. Three verbosity tiers — minimal, standard, full — control token footprint. Stack traces are intelligently truncated: entry point and crash site preserved, middle frames replaced with "(N frames omitted)". High-cardinality fields — trace IDs, span IDs, request IDs — are stripped automatically. Identical events within a window collapse: fifty connection failures become one event with "count": 50. A log storm that would consume 50k tokens produces 500 tokens of canonical output.
The embedding model — 23M parameters, ~23MB on disk — runs on-device via ONNX Runtime. No cloud API calls. No usage fees. No data leaving the machine. Beyond the CLI, Lumber exposes a public Go library API: import the classification engine directly — no subprocess, no HTTP, no serialization overhead. Single texts or batches. Stream mode for live tails, query mode for historical fetch. Two external dependencies. Single binary. Apache 2.0.
Yes. Lumber is open source under the Apache 2.0 license, with source at github.com/hejijunhao/lumber.
Lumber is written in Go 1.24 and ships as a single static binary with just two external dependencies, onnxruntime-go and x/text.
Lumber classifies logs by meaning rather than by pattern: each line is embedded into a 1024-dimensional vector by a local model and matched against 42 taxonomy labels via cosine similarity, so differently worded messages for the same event land in the same category.
No. Lumber runs its 23M-parameter embedding model on-device via CPU-only ONNX Runtime; there are no cloud API calls, no usage fees, and no data leaves the machine.
Lumber emits uniform structured events as NDJSON — to stdout, a file, or a webhook — collapsing a 50k-token log storm into roughly 500 tokens of canonical, LLM-ready output.
Lumber ingests logs from providers including Vercel, AWS, Fly.io, Datadog, Grafana, and Supabase, normalising any format into one schema of 8 categories and 42 labels.