01
Append-optimized
Rows arrive as new rows, cheaply and one at a time. Corrections are real SQL — UPDATE and DELETE resolved at compaction — but they are not the design center, and that choice is what buys the ingest speed.
TallyDB
Embedded · SQL-native · numeric
The database that does the math for you.
TallyDB is a small, embeddable, SQL-native database for time-series and other append-ordered numeric data — with regression, correlation, and PCA running as SQL, inside the engine, on the same buffers your query already touched.
No server to run. No ETL between ingest and analysis. No copying rows out to a math library and back.
$
cargo install tallydb
tallydb ./market
tallydb> .import ticks.csv ticks 2,481,003 rows appended in 1.4s tallydb> SELECT symbol, regr_slope(px, ts) OVER w AS drift, corr(px, qty) OVER w AS flow FROM ticks WHERE symbol IN ('AAPL', 'MSFT') WINDOW w AS (PARTITION BY symbol ORDER BY ts ROWS 63 PRECEDING); symbol │ drift │ flow ────────┼────────────┼─────────── AAPL │ 0.0004182 │ 0.6113 MSFT │ -0.0001077 │ 0.2840 2 rows · 120µs
9.6×
faster rolling regressions than DuckDB + NumPy
120µs
to serve the newest window on a live query
~1µs
added per append at the default sync level
0
servers to run, ETL steps, or copies out to a math library
How it compares
A race to the same finish line: how much work each stack gets through in one millisecond, on rolling statistics over 20,000 rows with a window of 64.
ground covered in one millisecond — longer is better
Accurate at timestamp scale
The common fast idiom for rolling moments loses precision exactly where an ordering key lives. TallyDB holds 1e‑12-to-truth accuracy at those offsets, guarded on every change by a compensated reference over adversarial data.
Checked against the tools you already trust
Every query family and every window is re-derived independently by DuckDB and NumPy in continuous integration, over data that has round-tripped through storage on disk.
How we did it
Three assumptions, held strictly. General-purpose databases relax all three — which is what makes them bigger, slower to start, and harder to embed. Holding them is what lets TallyDB hand a column straight to a math routine with no copy and no conversion.
02
Ordered
Rows arrive roughly sorted on a declared ordering key — a timestamp, a sequence number, a ledger offset. Storage is partitioned on it, so a range predicate skips whole segments instead of scanning them.
03
Numeric-or-key
Every column is either a number — f64 or i64, used in arithmetic, aggregation, and windows — or a key: a dictionary-encoded label used only for filtering, grouping, and joining. Every value is therefore fixed-width — eight bytes for a number, four for a key code — so a column is a flat, densely packed run of machine words with no indirection and no variable-length anything.
Giving up strings is what makes the rest possible. Because no column can hold arbitrary text, the engine knows the exact size, shape, and order of every column before it reads a byte: how wide each value is, how many are in a segment, and what range of the ordering key they cover. Nothing has to be parsed, unpacked, chased through a pointer, or sized at runtime.
That is where the speed comes from — a query can skip whole segments that cannot match, and the segments that survive are already in the layout a numeric routine wants, so the statistics run on the buffers the read produced rather than on a copy of them. None of it trades away precision: the engine computes rolling moments in the numerically stable form rather than the fast-but-cancelling one, and holds accuracy to within 1e‑12 of truth even at timestamp-scale offsets.
The narrowness has a real cost, and it is a small one for this kind of data: labels come back as dictionary codes rather than text, and rendering them for display happens in your application.
Against the alternatives
What that trade looks like in practice
Row by row, against the two shapes most databases take: transactional engines built for writes, and analytical engines built for scans.
| TallyDB | OLTP engines | OLAP engines | |
|---|---|---|---|
| Single-row write speed | Fast | Fast | Slow — wants batches |
| Numeric compute speed | Fast | Slow — row storage | Fast |
| Storage layout | Ordered columnar segments | Row pages with in-place update | Columnar, batch-oriented |
| Ingest shape | One row at a time, queryable immediately | One row at a time | Large batches, or an ETL step |
| Analytical scans | Columnar, with segment pruning | Scans the whole row | Columnar and vectorized |
| Where statistics run | In the engine, on its own buffers | Outside, after an export | Outside, after an export |
| Scripting | Lua in SQL and SQL in Lua, zero-copy | Stored procedures | An out-of-process client |
| Column types | Numbers and keys only | Anything | Anything |
| Arbitrary transactional writes | Corrections only, at compaction | The whole point | Limited |
Read down the first two rows and the trade is plain: OLTP engines are fast to write and slow to compute, OLAP engines are fast to compute and slow to write. TallyDB is fast at both — not by being cleverer, but by giving up the general-purpose write path and moving the math inside the engine.
Reach for TallyDB when
Your data is an append-heavy ledger of numbers with labels attached — quantitative research, sensor and telemetry pipelines, event and metric streams, financial ledgers — and the analysis is rolling aggregates, grouping, lookups against small reference tables, and statistics.
Reach for something else when
You need general-purpose relational work: arbitrary text and blob columns, or joins between two large tables on an arbitrary key. TallyDB refuses those loudly rather than serving them slowly — Postgres, DuckDB, and SQLite are better at being general.
Install and import
Get the engine onto your machine, then get your data into it. Linux, macOS, and Windows.
Part one
Install the engine
TallyDB ships in two shapes, and they are the same engine. Use the tallydb console when you want a prompt to explore data at, and the library when the database belongs inside your own program — there is no server either way, so nothing to deploy, supervise, or authenticate against.
1 · The console
Download a release binary, or build it with Cargo.
cargo install tallydb
# open (or create) a database directory
tallydb ./market
2 · Embed the library
One dependency, no process to supervise.
# Cargo.toml [dependencies] tallydb = "0.1" // main.rs let db = Database::open("./market")?; let out = db.query("SELECT count(*) FROM ticks")?;
Part two
Import your data
Declare the table once, then load it. A CSV import is the fast way to bring history in — the file is read straight into ordered segments with no staging step — and the same table keeps accepting rows one at a time as new data arrives. Either way the rows are queryable the moment they land.
3 · Declare the table
Name the ordering key. Every other column is a number or a key.
CREATE TABLE ticks ( ts I64 ORDERING KEY, symbol KEY, px F64, qty I64 );
4 · Import a CSV
Header names are matched to columns; rows go straight into segments.
# ts,symbol,px,qty .import ticks.csv ticks → 2,481,003 rows appended in 1.4s # queryable immediately SELECT count(*) FROM ticks;
5 · Or append as data arrives
One row at a time from your application, or in bulk with SQL.
-- SQL INSERT INTO ticks VALUES (1719849600123, 'AAPL', 214.35, 100); // Rust db.append("ticks", &row)?;
Using it
Standard SQL for querying. Lua when SQL runs out of vocabulary — called from SQL, or driving SQL. Both run on the engine's own buffers, in-process, with nothing serialized across a boundary.
SQL, with the statistics built in
SELECT · WHERE · GROUP BY · HAVING · ORDER BY · LIMIT · DISTINCT · CASE · LIKE, star-schema joins against small reference tables, window frames, and UPDATE/DELETE for corrections.
On top of the standard aggregates, five statistics are native window functions: regr_slope, regr_intercept, covar_pop, corr, and eigen_max — the window's first principal-component variance.
Results come back in an Arrow-compatible columnar layout, so NumPy and other Arrow-aware tooling read them with no conversion step.
query.sql
-- rolling risk per desk, joined to a small
-- reference table, filtered on a key column
SELECT
d.desk,
covar_pop(r.pnl, r.exposure) OVER w AS cov,
corr(r.pnl, r.exposure) OVER w AS rho,
eigen_max(r.pnl, r.exposure) OVER w AS pc1
FROM returns r
JOIN desks d ON d.id = r.desk_id
WHERE d.region LIKE 'EU%'
WINDOW w AS (
PARTITION BY r.desk_id
ORDER BY r.ts
ROWS 63 PRECEDING
)
ORDER BY d.desk;
Lua in SQL — your own window function
Register a Lua kernel and call it from SQL like any built-in window function. The script reads the engine's numeric buffers directly as zero-copy views — nothing is copied out, and the curated native operations are callable from inside the script over those same buffers.
Write kernels in the vectorized vocabulary — whole-column operators and rolling combinators under one interpreter entry, the same model as NumPy. Element-by-element loops in Lua are legal but interpreter-bound; the vectorized form is the fast path.
Results are coerced exactly, or loudly, to the type declared at registration. NULL crosses the boundary as a sentinel and stays three-valued through arithmetic; keys arrive as dictionary codes, with text() and code_of() to move between code and label.
weighted_vol.lua → SQL
-- a window kernel over the engine's buffers function weighted_vol(px, w) local ret = (px - shift(px, 1)) / shift(px, 1) local m = wmean(ret, w) return sqrt(wmean((ret - m) ^ 2, w)) end -- register it, then call it from SQL .lua weighted_vol.lua → F64 window SELECT symbol, weighted_vol(px, decay) OVER w AS vol FROM ticks WINDOW w AS (PARTITION BY symbol ORDER BY ts ROWS 63 PRECEDING);
SQL in Lua — a driver script
The embed runs both ways. A driver script calls query(sql) and receives result columns as the same zero-copy views kernels consume, computes with the vectorized vocabulary, and feeds derived rows back through append(table, row) — an entire ingest-to-signal pipeline inside the engine, with no round trip through your application.
Run it from the console with .run FILE, or from your application with Database::run_script.
Script diagnostics route to a sink you install, so an embedded library never writes to your process's stdout.
signal.lua — .run signal.lua
-- read with SQL local r = query[[ SELECT ts, px FROM ticks WHERE symbol = 'AAPL' ORDER BY ts ]] -- compute over the same buffers local ret = (r.px - shift(r.px, 1)) / shift(r.px, 1) local sig = rolling_dot(ret, weights, 64) -- write the derived rows back for i = 1, #sig do append("signal", { ts = r.ts[i], v = sig[i] }) end log(("wrote %d rows"):format(#sig))
Try it on your own data
Point the console at a directory, import a CSV, and run a rolling regression in one line of SQL. TallyDB is MIT licensed and in active development.