Tinybird

@zanreal/medusa-usage-tinybird

The Tinybird sink for Copy to clipboard@zanreal/medusa-usage: the same append-only usage log, in a column store built to scan it.

Full documentation, in English and Polish, is published at https://zanreal.com/docs/oss/medusa-usage-tinybird and authored in Copy to clipboarddocs/.

The plugin ships a Postgres sink and works on a plain Medusa install with no account to open. This is for the other case: a meter counting billions of events, where the log stops fitting comfortably in the application's own database. Installing this package is the whole decision, and a deployment that does not install it is unaffected in every way.

1
pnpm add @zanreal/medusa-usage-tinybird

123456789101112131415161718
// medusa-config.ts
plugins: [
  {
    resolve: "@zanreal/medusa-usage",
    options: {
      providers: [
        {
          resolve: "@zanreal/medusa-usage-tinybird",
          id: "tinybird",
          options: {
            host: process.env.TINYBIRD_HOST,
            token: process.env.TINYBIRD_TOKEN,
          },
        },
      ],
    },
  },
]

The schema this sink talks to

The sink is one half of the design; the other half is the Tinybird schema it reads and writes, and the two only work together. That schema ships in this repository under Copy to clipboardtinybird/ - one data source and three endpoints, and nothing else:

Resource File What it is Copy to clipboardusage_events Copy to clipboardtinybird/datasources/usage_events.datasource The append-only log. Copy to clipboardReplacingMergeTree, sorted Copy to clipboardmeter, subject, occurred_at, key, partitioned by month of Copy to clipboardoccurred_at. Copy to clipboardusage_aggregate Copy to clipboardtinybird/endpoints/usage_aggregate.pipe The total behind an invoice. Copy to clipboardusage_events_list Copy to clipboardtinybird/endpoints/usage_events_list.pipe The events behind that total, keyset paged. Copy to clipboardusage_events_present Copy to clipboardtinybird/endpoints/usage_events_present.pipe Which keys the log already has.

Deploy it before pointing a sink at it, with Tinybird's own CLI:

12
tb login                 # or --host for a self-hosted instance
tb --cloud deploy

The read-time collapse described below lives in those Copy to clipboard.pipe files, so pointing this sink at a data source that was created some other way - a hand-written Copy to clipboardMergeTree, or an endpoint that does not Copy to clipboardGROUP BY key - gives back a sink that double counts every retry. Deploy the schema in this repository rather than reimplementing it.

The names are options, so a workspace that already uses them for something else can deploy under different ones and set Copy to clipboarddatasource, Copy to clipboardaggregatePipe, Copy to clipboardlistPipe and Copy to clipboardpresentPipe to match. The Copy to clipboardmedusa_usage token the four files declare carries exactly the grants the sink needs: Copy to clipboardAPPEND on the data source, Copy to clipboardREAD on the three endpoints.

At most one row per key

This is the guarantee the plugin rests on, and it is the one thing that does not port from Postgres for free. It is worth reading before trusting a number this sink produces.

Postgres gets it from a primary key. The deduplication key IS the primary key, Copy to clipboardINSERT ... ON CONFLICT DO NOTHING makes a retry a no-op inside the statement, and it is true the instant the statement returns.

ClickHouse, which is what Tinybird is, has no primary key constraint. Its Copy to clipboardORDER BY is a sorting key, not a unique one, and nothing refuses a second copy of a row. The nearest mechanism is Copy to clipboardReplacingMergeTree, which collapses rows sharing a sorting key during background merges - which run when the engine decides to, possibly hours later, possibly not at all for parts it does not choose to combine.

So the honest statement about the engine on its own is eventual. A sink built on Copy to clipboardReplacingMergeTree and nothing else double counts every retry until a merge happens to run, and for a number that becomes an invoice that is not a rough edge, it is the failure the plugin exists to prevent.

What this sink does instead

Deduplication is enforced where the log is read. Every endpoint collapses to one row per key before it sums anything:

1
SELECT key, argMax(quantity, version) AS event_quantity ... GROUP BY key

That is not eventual. It is computed over whatever rows exist at the moment of the query, so a duplicate written a millisecond ago is already collapsed: Copy to clipboardaggregate() taken immediately after a retried Copy to clipboardwrite() returns the number it returned before it. That is asserted against a live Tinybird, not argued (see Testing).

The engine's merge is then a storage optimisation and nothing more. Copy to clipboardENGINE_VER is a negated ingestion timestamp, so the merge keeps the same row the read path keeps: the earliest-ingested copy, which is also the copy Postgres keeps. The property that matters falls out of that - a merge can only ever remove a row the read path was already discarding - so an answer cannot change because a merge ran, and a number can still be re-derived in a year.

What is left eventual, stated plainly

• The physical log. Between a duplicate write and a merge, two rows exist on disk, and Copy to clipboardSELECT count() on the data source will say so. Do not bill from a direct query against the data source; a bare Copy to clipboardSELECT sum(quantity) counts every copy. The endpoints are the collapse, and they are the supported read path. Copy to clipboardaggregate() and Copy to clipboardlistEvents() are correct at any moment.
• The counters Copy to clipboardwrite returns. Copy to clipboardduplicates is counted by asking which keys are already stored, immediately before appending. That is a check-then-act, so two processes writing the same key at the same instant can both find it missing and both append, leaving the counters optimistic by one. Nothing else is affected: the copies are identical and the read path keeps one.
• Whose copy wins under a key collision. Both rules keep the earliest ingested, matching Postgres. It only becomes observable if a caller reuses an explicit Copy to clipboardidempotencyKey across events with different facts, which is a caller bug in either sink.

Options

Everything comes from the provider's Copy to clipboardoptions, with an environment fallback for the two values that belong to a deployment rather than to a repository. Nothing is hardcoded, and the token is never logged: it goes into an Copy to clipboardAuthorization header and nowhere else, never into a URL, and a Tinybird error body that quotes it back is redacted before it reaches a message.

Option Default What it is Copy to clipboardhost Copy to clipboardTINYBIRD_HOST The Tinybird API host, e.g. Copy to clipboardhttps://api.tinybird.co. Copy to clipboardtoken Copy to clipboardTINYBIRD_TOKEN A token with Copy to clipboardAPPEND on the data source and Copy to clipboardREAD on the endpoints. The Copy to clipboardmedusa_usage token the schema declares is exactly that. Copy to clipboarddatasource Copy to clipboardusage_events The usage log. Copy to clipboardaggregatePipe Copy to clipboardusage_aggregate The aggregate endpoint. Copy to clipboardlistPipe Copy to clipboardusage_events_list The listing endpoint. Copy to clipboardpresentPipe Copy to clipboardusage_events_present The key-lookup endpoint. Copy to clipboardcheckForDuplicates Copy to clipboardtrue Ask which keys are already stored before appending. Copy to clipboardtimeoutMs Copy to clipboard10000 How long one HTTP call may take before it is abandoned and retried.

Every one of them is validated by Medusa's provider loader before the service is constructed, so a missing token is a failed boot with a sentence explaining what to set, not a 401 six hours into a billing period.

Copy to clipboardcheckForDuplicates

On by default. It costs one extra round trip per batch and buys two things: a truthful Copy to clipboardduplicates count, and a retried batch that appends nothing at all rather than a second copy of every row.

Turning it off halves the round trips and cannot cause a double count - deduplication is in the read path either way. What it costs is honesty in the counters, which will read zero duplicates forever, and a log that accumulates physical copies until the engine merges them away.

Testing

1
pnpm test

Unit tests run against a fake Tinybird and cover the wiring: what goes on the wire, what comes back off it, that the token never reaches a URL, that a quarantined row throws rather than vanishing.

They are not sufficient, and the suite says so. The property this sink is hard to get right belongs to the engine on the other side, so Copy to clipboardsrc/live.test.ts writes to a real Tinybird and reads the answer back. It is skipped unless the environment names one:

123
tb local start
tb --local build
TINYBIRD_HOST=http://localhost:7181 TINYBIRD_TOKEN=... pnpm test

What it asserts there: that a write can be aggregated as soon as it returns; that a replayed batch does not move the total; that a duplicate which did reach the log as a second physical row still does not move the total; that the window is half open at millisecond resolution and consecutive periods tile; that a dimension filter is an equality and is typed; and that paging never skips or repeats a row.

Releasing

Publishing happens only from Copy to clipboard.github/workflows/release.yml, and there is no second path. npm provenance is a signed statement about where a tarball was built and from which commit, and only a cloud CI run holding an OIDC identity can produce one. An Copy to clipboardnpm publish from a laptop would put a version on npm carrying no provenance, and a published version cannot be replaced afterwards, only deprecated. Copy to clipboardpublishConfig.provenance in Copy to clipboardpackage.json makes that local publish fail rather than quietly succeed without it.

To cut a release:

1. Move the Copy to clipboard## [Unreleased] entries in CHANGELOG.md under a heading for the new version, dated.
2. Bump Copy to clipboardversion in Copy to clipboardpackage.json on Copy to clipboardmain.
3. Publish a GitHub Release whose tag is Copy to clipboardv<version>, exactly.

The workflow refuses to publish when the tag disagrees with Copy to clipboardpackage.json, or when that version is already on the registry. A release marked as a prerelease on GitHub publishes under the Copy to clipboardnext dist-tag, so Copy to clipboardnpm install @zanreal/medusa-usage-tinybird never resolves to a release candidate.

Publish Copy to clipboard@zanreal/medusa-usage before this package. It is declared here as a peer dependency, and until it is on the registry an install of this sink cannot resolve it.

Authentication is an Copy to clipboardNPM_TOKEN repository secret: a granular access token with write permission on this package. npm's trusted publishing (OIDC, with nothing stored in GitHub) cannot cover the first publish, because npmjs.com only offers the trusted publisher form on a package that already exists. Once the first version is up, add one under the package's settings on npmjs.com - GitHub Actions, owner Copy to clipboardzanreal-labs, repository Copy to clipboardmedusa-usage-tinybird, workflow Copy to clipboardrelease.yml, environment Copy to clipboardnpm - and then delete the Copy to clipboardNPM_TOKEN secret. The workflow needs no edit for that: npm attempts the OIDC exchange first and falls back to the token only when the exchange fails.

License

MIT