Version: 1.33.0

Storage & background tasks

This page covers four subsystems that share one trait: they all talk to something outside your process.

File storage — one API over the local filesystem, S3, GCS and Azure.
Background tasks, plus the jobs and webhooks a module declares for itself — recurring work and inbound callbacks.
Transactional outbox — events that become durable exactly when the transaction producing them commits.
Mail — SMTP, and vendor providers that install as plugins.

Because any of them can fail independently, storage and mail calls are wrapped in circuit breakers by default.

File storage (`pkg/storage`)

pkg/storage is a provider-agnostic file storage abstraction, with an interface designed to last through v1.x. The same code runs against:

the local filesystem — built in, no extra dependency,
AWS S3 (and anything S3-compatible: MinIO, R2, Spaces),
Google Cloud Storage,
Azure Blob Storage.

Cloud backends are separate modules

The framework carries local storage only. A cloud backend is one go get and one blank import away, and nothing else changes — the configuration is the same, and so is every line of code that uses a.Storage:

import _ "github.com/jcsvwinston/nucleus/providers/storage-s3"
// or providers/storage-gcs, or providers/storage-azure

They live outside the framework because a client library nobody opens is not free: the three cloud SDKs together weighed 42.6 MB of binary and 139 modules that every application paid for, whether or not it ever wrote to a bucket.

If you configure provider: s3 without the import, startup fails and the error prints the two lines above. It never quietly falls back to local storage — a silent fallback is how uploads end up on a container disk that disappears with the pod.

import "github.com/jcsvwinston/nucleus/pkg/storage"

// Get returns a ReadCloser and object metadata; always close the reader.
reader, info, err := a.Storage.Get(ctx, "uploads/avatar.png")

// SignedURL requires an opts argument (use zero value for defaults).
url, err := a.Storage.SignedURL(ctx, "uploads/avatar.png", 5*time.Minute, storage.URLConfig{})

// Put returns the stored ObjectInfo and an error.
info, err = a.Storage.Put(ctx, "uploads/avatar.png", body, storage.PutOptions{
    ContentType: "image/png",
})
_ = reader
_ = info

Configure the backend in nucleus.yml. The exact key shape is provider- specific — the snippet below is illustrative; the canonical schema is the Configuration reference:

# illustrative — see the Configuration reference for the full storage.* schema
storage:
  provider: s3            # local | s3 | gcs | azure
  s3:
    bucket: my-bucket
    region: eu-west-1

Per-driver credentials and endpoints are read from environment variables or platform credential providers — never embedded in the config file.

An application built WithoutDefaults() — the api template — builds no storage by default. It builds the storage block above when it also carries WithStorage(), which the api template does; without that option the block is ignored, and the boot log says so in one ERROR line naming it. From v2.0.0 that configuration refuses to start (DEP-2026-013). An application that declares no storage block gets no store either way.

Using a storage backend Nucleus does not ship

The four built-in providers — local, S3, GCS, Azure — are registered the same way anyone else registers one, so a backend this framework has never heard of is selectable by name without patching it:

package cephstore

import "github.com/jcsvwinston/nucleus/pkg/storage/provider"

func init() {
    provider.Register("ceph", New)
}

Note the import: pkg/storage/provider, not pkg/storage. It is a leaf package holding the contract and nothing else — the Store interface, its value types, the configuration and the registry — so writing a backend does not drag in the AWS, Azure and Google Cloud SDKs that the built-in implementations need. The names remain available from pkg/storage as aliases, so existing code keeps compiling.

Import that package for its side effects, set storage.provider: ceph, and the framework builds it. Everything it layers on top — the circuit breaker, tenant prefixing, the public-URL mapper — is applied around whatever your factory returns, so a provider never reimplements any of it.

Your backend reads its own settings from its own subtree:

storage:
  provider: ceph
  ceph:
    endpoint: http://ceph.internal
    pool: 32

func New(cfg storage.Config) (storage.Store, error) {
    var c struct {
        Endpoint string        `koanf:"endpoint" validate:"required"`
        Pool     int           `koanf:"pool" default:"8"`
        Timeout  time.Duration `koanf:"timeout" default:"5s"`
    }
    if err := cfg.BindProvider(&c); err != nil {
        return nil, err
    }
    …
}

The framework validates the namespace but not its contents — it cannot know the shape of a backend it has never seen, which is the point of the registry. What it does know is which providers are registered, so storage.ceph.* is accepted when ceph is and rejected when it is not: a misspelled section still fails as an unknown key rather than being ignored.

Inside BindProvider, a key your struct does not declare is an error too. Provider configuration is exactly the place a typo would otherwise sit unnoticed until the day the setting mattered.

Registering a name that is already taken is an error rather than a silent replacement: two packages claiming s3 would otherwise make the effective backend depend on import order.

An unconfigured or unknown provider name now fails with the list of registered ones. It used to fall through to the local filesystem, so a typo wrote your uploads to disk and said nothing.

Circuit breaker (storage)

App.New automatically wraps all remote provider operations (Put, Get, Delete, Exists, List, Copy, SignedURL) with a pkg/circuit.Breaker. The local provider and PublicURL (pure string composition) are never wrapped. storage.ErrNotFound is not counted as a failure — a missing object is a normal outcome.

When the breaker is open, wrapped operations return circuit.ErrOpen immediately. The default thresholds are:

storage:
  circuit_breaker:
    enabled: true
    failure_threshold: 5
    cooldown: 30s
    half_open_max_concurrent: 1

Set enabled: false to disable, or tune the thresholds for your workload. Full details: docs/guides/STORAGE_GUIDE.md.

Background tasks (`pkg/tasks`)

pkg/tasks runs one-off background tasks. Payloads are encoded as JSON and keyed by a task-type string; the manager handles enqueue, retry and dead-letter, and every provider records the same jobs.* metrics. Three providers ship in-tree:

pkg/tasks/providers/memory — in-process, no external dependency. Pending tasks are lost on restart, but nothing is lost while the process runs: see what the in-process queue holds.
pkg/tasks/providers/sql — durable on the database you already have, with no broker: see the durable queue.
pkg/tasks/providers/asynq — Asynq + Redis, durable.

For recurring work declared by a module, use module jobs (next section) — the framework schedules those for you. tasks.Manager is the lower-level surface for one-off tasks, and there are two ways to hold one.

The standalone cycle: build, register, run, close

tasks.Manager is not a field on App — you construct it from the provider you chose, register handlers, run the worker loop, and close it on shutdown. The complete cycle, compilable as shown:

package main

import (
	"context"
	"log"
	"os/signal"
	"syscall"

	"github.com/jcsvwinston/nucleus/pkg/tasks"
	asynqprovider "github.com/jcsvwinston/nucleus/pkg/tasks/providers/asynq"
)

const TypeSendWelcomeEmail = "email:welcome"

type SendWelcomeEmail struct {
	UserID int64
}

func main() {
	// memoryprovider.NewManager takes the same arguments for an
	// in-process, non-durable queue (no Redis needed).
	mgr, err := asynqprovider.NewManager(tasks.Config{
		RedisURL:    "redis://localhost:6379",
		Concurrency: 8,
	}, nil) // nil logger → slog.Default()
	if err != nil {
		log.Fatal(err)
	}

	// Register handlers before the first enqueue of each type.
	err = mgr.HandleFunc(TypeSendWelcomeEmail, func(ctx context.Context, t tasks.Task) error {
		var p SendWelcomeEmail
		if err := tasks.DecodeJSONPayload(t, &p); err != nil {
			return err
		}
		return sendWelcome(ctx, p.UserID)
	})
	if err != nil {
		log.Fatal(err)
	}

	// Run blocks until ctx is cancelled; run it in the same process as
	// the enqueuers or in a dedicated worker binary.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()
	go func() {
		if err := mgr.Run(ctx); err != nil {
			log.Printf("tasks: %v", err)
		}
	}()

	// Enqueue from anywhere that holds the manager (payload is
	// JSON-encoded for you):
	if _, err := mgr.EnqueueJSON(TypeSendWelcomeEmail, SendWelcomeEmail{UserID: 42}); err != nil {
		log.Printf("enqueue: %v", err)
	}

	<-ctx.Done()
	_ = mgr.Close()
}

func sendWelcome(ctx context.Context, userID int64) error { return nil }

EnqueueJSONWithPolicy and friends accept a tasks.EnqueuePolicy for queue, retry, timeout, delay and retention control.

The module path: `rt.Tasks()`

An application built on pkg/nucleus does not need to construct a manager at all: the framework builds one for the module-jobs runtime, and a module reaches it through its Runtime handle — rt.Tasks() returns that shared manager for enqueueing one-off tasks (and registering their handlers) beyond the cron jobs it declared.

Availability is explicit: the jobs runtime is built when at least one module registers a job, or when jobs_provider names a broker-backed provider (asynq) — configuring a broker is the opt-in for enqueue-only applications. rt.Tasks() returns nil inside OnStart (the runtime starts after every module's OnStart); resolve it lazily from request handlers, and treat nil as "no jobs runtime configured". rt.Outbox() follows the same degrade-to-nil contract for the transactional outbox.

Module jobs and webhooks

Modules declare recurring background jobs and inbound webhook receivers directly on their Module[C] definition; the framework schedules the jobs on pkg/tasks and mounts the webhook routes at boot.

nucleus.Module[BillingConfig]{
    Name: "billing",
    Jobs: func(j nucleus.JobRegistry, cfg BillingConfig) {
        _ = j.Register("reconcile", nucleus.JobSpec{
            Every:     15 * time.Minute, // or Cron: "0 3 * * *"
            Timeout:   5 * time.Minute,
            Singleton: true, // skip a tick while the previous run is live
            Handler: func(ctx context.Context) error {
                return reconcileInvoices(ctx)
            },
        })
    },
    Webhooks: func(w nucleus.WebhookRegistry, cfg BillingConfig) {
        _ = w.Register("/stripe", nucleus.WebhookSpec{
            Secret: cfg.StripeWebhookSecret,
            Handler: func(rw http.ResponseWriter, r *http.Request) {
                // Body is verified against X-Nucleus-Signature before
                // this handler runs; read it as usual.
            },
        })
    },
}

Jobs. Each registration needs exactly one schedule: Every for a fixed interval, or Cron for a standard 5-field expression or a descriptor such as @hourly. Cron expressions are validated at boot and mean the same thing on every provider.

The jobs_provider config key selects the runtime:

memory (default) — in-process. Pending jobs are lost on restart.
sql — durable on the application's own database. No broker to operate.
asynq — Redis-backed and durable. Set jobs_redis_url.

The durable queue (`jobs_provider: sql`)

A job accepted by one process runs even if that process never comes back. The queue is a table in the database the application already has — there is no broker to operate, which is what Solid Queue and Oban removed from Rails and Elixir.

nucleus add sql-queue writes jobs_provider: sql into nucleus.yml (and prints it instead when the key is already set). The queue is built at boot whether or not a module has registered a job yet, so a module that only enqueues one-off tasks through Runtime.Tasks() has it too. The keys it reads:

jobs_provider: sql
jobs_table: nucleus_jobs      # created if missing, per dialect
jobs_queues: [urgent, default, bulk]
jobs_concurrency: 8

What it guarantees. A job runs at least once. A worker claims it under a lease, renews that lease while the handler runs, and marks the outcome when it returns. If the process dies holding a job, the lease expires and another worker picks it up — that is what makes the queue survive a crash, and it is also why a handler must tolerate running twice for the same job. Make handlers idempotent; there is no exactly-once over a network.

Queues and priority. jobs_queues is an ORDER, not a set of weights: a worker serves the first queue before the second. A queue nobody lists is not served at all, so a worker cannot quietly run work meant for another fleet.

Retries. Each job carries its own curve — BackoffBase doubling up to BackoffMax on the enqueue policy — so a webhook that should back off for minutes and a thumbnail that should be retried in a second no longer share one. A job that spends its attempts goes to the dead letter with its last error, where retry-archived puts it back and purge-archived empties it.

Enqueueing inside a transaction. EnqueueTx writes the job in the caller's transaction, so the job exists exactly when the work that asked for it commits — and never when it rolls back.

Engines. PostgreSQL, MySQL and SQLite. SQL Server and Oracle are refused by name rather than silently treated as something else: the statements have not been exercised against them.

Workers and throughput. On PostgreSQL, MySQL 8 and MariaDB 10.6 or later, a worker claims a job with SELECT … FOR UPDATE SKIP LOCKED: it locks the row it takes and every other worker steps over it to the next one, so workers do not collide and adding them adds throughput until the database host runs out of CPU. On PostgreSQL 16 and MySQL 8.4 CI checks that claims do not come back empty while free work is waiting, and that sixteen workers drain a queue at least twice as fast as one. A twelve-core workstation measured 4.4 to 6 times; a four-vCPU runner saturates at about four workers. The claim reads the queue's index in order and stops at the first free row, so its cost does not grow with the backlog; CI checks that too. On MySQL and MariaDB it runs at READ COMMITTED: under the default REPEATABLE READ the claim's locking read also locks gaps in the index, and the workers end up waiting on each other.

On SQLite, and on a MySQL or MariaDB without SKIP LOCKED, the claim selects the next jobs and races a conditional UPDATE for each: correct everywhere, but every worker contends for the same rows, so more workers do not mean more throughput. The engine is asked once when the queue opens; on a server engine that refuses, the worker logs a warning when it starts. A worker that loses a race does not wait out PollInterval, which is reserved for a queue that has nothing due: it tries again after a random pause of up to a millisecond, which doubles with each further loss in a row and never exceeds PollInterval.

Stopping. The jobs runtime stops inside the application's shutdown, before the database closes: the scheduler stops ticking and hands its lease back, the workers stop claiming, and the handlers still running get the shutdown grace (write_timeout) to finish before what they hold is released for another process. With nothing running, stopping takes milliseconds.

Showing the queue

Whatever displays the queue — Orbit's admin panel, or a status page of your own — needs the provider's inspector, and Runtime hands out the manager. Ask for it:

if inspector, ok := nucleus.TaskInspectorFrom(rt); ok {
        snapshot := inspector.InspectRuntime()   // pending, active, dead, per queue
}

It is an optional interface rather than a method on Runtime, because Runtime is published and growing it would break every implementation outside the framework. All three providers satisfy it.

Every provider also records the same jobs.* metrics — enqueued, started, succeeded, retried, failed, held, and a duration histogram — labelled by provider, queue and task type. They used to live inside the asynq provider only, so an application without a Redis had nothing to alert on.

Scheduled jobs, with no Redis either. Every replica runs the schedule and contends for a lease row in the same database; the one holding it fires, and if it dies another takes over within one lease. Without that election each replica fires every entry on every tick.

The same job twice. UniqueKey on the enqueue policy reserves the work while it is queued or running, so a double-clicked button or a webhook delivered twice collapses into one job — and the second caller is told which id theirs became, rather than getting an error. The key is freed when the job finishes.

What the in-process queue holds

Restart durability is the asynq provider's job. What the in-process provider guarantees is narrower and worth stating exactly: while the process runs, an accepted job is not discarded. Three things used to end a job silently, and now each one keeps it instead:

What happens	Where the job goes
No handler is registered for its type	Held, waiting. Registering that handler with `HandleFunc` puts every job of that type back on the queue — so a producer deployed ahead of its worker does not lose what it enqueued in between.
The handler failed and the policy's retries ran out	Held in the dead letter, with the attempt count and the last error.
The process stopped between two retries, or with jobs still queued	Held in the dead letter, with that reason, and the shutdown logs how many.

Both stores are bounded (1 000 jobs each) and have separate budgets, so a mistyped task type cannot evict the jobs that actually died. When the bound evicts a job it is logged. Held jobs are reported through the queue snapshot as archived, and two queue actions operate on them:

retry-archived puts every held job back on the queue — the one to use after deploying the worker that was missing, or fixing what made the handler fail. Jobs that do not fit (the queue is bounded) stay held, each one in the store it came from.
purge-archived empties the dead letter only. Jobs still waiting for a handler are kept: they are not dead, and emptying the dead letter is not a way to lose them.

Registering a handler releases what was waiting for it, but if the queue is full at that moment those jobs stay held and nothing retries them on its own — the log says so, and retry-archived is the way back.

The other four queue actions (pause, unpause, retry, archive-retry) need a broker-backed provider and are refused by name, so an operator gets an answer rather than a click that does nothing.

jobs_concurrency caps the number of parallel workers. A broken registration — duplicate name, invalid cron, missing handler — fails boot rather than silently never running.

With asynq and more than one replica of your process, workers are safe to replicate (they consume a shared queue) but schedulers are not — each one would enqueue every cron entry on every tick. The framework therefore runs the scheduler under leader election over a Redis lock (SET NX + TTL): one replica ticks, the rest stand by, and a crashed leader is replaced within the lock TTL. That is the jobs_scheduler_lock key, on by default; setting it to false opts out and the boot log warns that every replica will fire every job.

Webhooks. Each registration mounts a real route at <webhooks_prefix>/<module-name><path>; the default prefix is /webhooks.

Incoming requests are checked in this order, all before your handler runs:

Method — POST only by default. Anything else gets 405.
Body size — 1 MiB by default. Larger bodies get 413.
Signature — with a Secret set, the request must carry an HMAC-SHA256 of the raw body in the X-Nucleus-Signature header, as sha256=<hex>. Unsigned or mis-signed requests get 401. Senders and tests can produce the value with nucleus.SignWebhookBody.

Two more rules apply at boot:

A webhook registered without a Secret is still mounted, but logs a WARN — its handler must authenticate callers itself.
Registration paths must be canonical. A path that path.Clean would rewrite (. or .. segments, duplicate or trailing slashes) fails boot with a clear error, instead of silently mounting a route that cleaned request URLs can never reach.

When csrf_enabled is on, the webhook prefix is exempted automatically: webhooks authenticate by signature, not by CSRF token.

Replay, declared honestly. The signature authenticates content, not freshness — a captured signed request verifies again if it is re-sent verbatim. If your handler's effect is not idempotent, deduplicate on an event ID carried in the payload.

To narrow the replay window, set TimestampTolerance on the spec. That changes the contract for senders:

The request must carry its send time as Unix seconds in the X-Nucleus-Timestamp header.
That time must fall within the tolerance of the receiver's clock.
The signature must cover <timestamp>.<body> rather than the body alone. nucleus.SignWebhookBodyWithTimestamp returns both header values.

A timestamp that is missing, malformed, stale, future-dated, or signed over the body alone is rejected with 401 before your handler runs.

Because it changes what senders must sign, the scheme is opt-in: leaving TimestampTolerance unset keeps the body-only behaviour. 5m is a sensible tolerance when senders have synced clocks, and event-ID deduplication still closes the window the tolerance leaves open.

Transactional outbox (`pkg/outbox`)

The naïve "enqueue inside a SQL transaction" pattern silently loses events when the transaction commits but the queue write fails. pkg/outbox solves this with the standard outbox pattern:

import (
    "database/sql"

    "github.com/jcsvwinston/nucleus/pkg/outbox"
)

// App.DB.Tx runs fn inside a transaction (tx is a *sql.Tx).
// App.Outbox is a *outbox.ManagedOutbox; EnqueueTx writes the event row
// in the SAME transaction, so the event is durable iff the commit lands.
err := a.DB.Tx(ctx, func(tx *sql.Tx) error {
    if err := repo.Save(tx, article); err != nil {
        return err
    }
    _, err := a.Outbox.EnqueueTx(ctx, tx, outbox.Entry{
        Topic:   "article.published",
        Payload: ArticlePublished{ID: article.ID},
    })
    return err
})

The outbox table is part of the migration set the framework manages. With outbox.enabled: true the framework starts a leasing dispatcher that polls the table and delivers committed events through the bridges declared under outbox.bridges (all keys in the Configuration reference).

Shutdown

Stopping the outbox is graceful: the dispatcher finishes the pass it is in — including the delivery in flight — and only then exits. A pass is not abandoned halfway, so a message is never left claimed by a delivery that was cut off mid-attempt (it would have waited for its lease to expire before anyone retried it).

Waiting is bounded. Stop(ctx) escalates to cancelling the pass when the context you pass expires, or after five seconds if it carries no deadline — a bridge that never answers cannot hold shutdown hostage. The escalation is logged at WARN so a slow shutdown is visible rather than silent.

Webhook bridge: delivery contract

A webhook bridge POSTs each outbox message as a JSON body to the configured URL:

{
  "id": "msg-01hzy4v7",
  "topic": "orders.placed",
  "payload": "eyJvcmRlcl9pZCI6NDJ9",
  "status": "pending",
  "attempts": 1,
  "available_at": "2026-07-22T12:00:00Z",
  "created_at": "2026-07-22T12:00:00Z"
}

Every delivery carries the header X-Outbox-Payload-Encoding declaring the shape of the payload field for that message, so the receiver never guesses:

base64 — the default: payload is a JSON string holding the base64 encoding of the raw payload bytes (the example above). Decode the string to get the payload document. This is the classic wire shape — byte-for-byte what every release up to v1.4.0 emits — so existing consumers keep working unchanged; the header is purely additive.
json — opt-in per bridge with payload_encoding: json: the payload document is embedded verbatim, e.g. "payload": {"order_id": 42}, and the consumer reads it directly with no base64 round-trip. Two edge cases: a payload that is not valid JSON (possible only for rows not written by the framework's own enqueue path) falls back to the base64 string form and declares base64 in the header for that delivery; a message with no payload puts JSON null in the field.

outbox:
  enabled: true
  bridges:
    - name: order-hooks
      type: webhook
      config:
        url: "https://consumer.example.com/hooks/outbox"
        pattern: "orders.*"
        payload_encoding: json   # opt-in; omit for the base64 default
        secret: "shared-webhook-secret"

Webhook bridge: HMAC signature

With secret set, the bridge signs every delivery: the header X-Nucleus-Signature carries sha256=<hex>, the HMAC-SHA256 of the exact body bytes under the shared secret. It is the same header and the same scheme module webhooks verify (previous section), so one verifier covers both directions — and nucleus.SignWebhookBody computes the expected value. Verify with a constant-time comparison, never ==/!=:

import (
    "crypto/hmac"
    "io"
    "net/http"

    "github.com/jcsvwinston/nucleus/pkg/nucleus"
)

func outboxHook(w http.ResponseWriter, r *http.Request) {
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }
    want := nucleus.SignWebhookBody(secret, body)
    got := r.Header.Get(nucleus.WebhookSignatureHeader)
    if !hmac.Equal([]byte(want), []byte(got)) {
        http.Error(w, "bad signature", http.StatusUnauthorized)
        return
    }
    // The body is authentic. The X-Outbox-Payload-Encoding header is
    // informational and travels UNSIGNED — the signature covers the body
    // alone — so decode by the encoding you configured for this bridge
    // (base64 unless you set payload_encoding: json), not by trusting the
    // header. Treat a header that disagrees with your configured encoding as
    // a bad request rather than as an instruction to switch decoders.
}

Without a secret the bridge delivers unsigned and logs a boot warning: the consumer must then authenticate deliveries itself (for example with a static header under config.headers) — and a static header does not authenticate the body, so prefer the signature.

Scope, honestly stated: the signature authenticates the body and nothing else. There is no anti-replay protection here — no timestamp in the signed material, no nonce — so a captured delivery verifies again if replayed.

In practice that is tolerable because outbox delivery is at-least-once anyway: consumers must already be idempotent, keyed on the message id.

Deliveries to plain http:// URLs send the body in clear. Use HTTPS outside loopback.

Plugin bridge: an external plugin delivers

A bridge of type plugin hands each message to an external capability plugin — an executable nucleus-plugin-<provider> on PATH — so a committed event can reach a broker or an endpoint the framework has no code for. The plugin serves queue.publish or webhook.deliver:

outbox:
  enabled: true
  bridges:
    - name: events
      type: plugin
      config:
        provider: relay             # runs nucleus-plugin-relay
        capability: queue.publish   # or webhook.deliver
        pattern: "orders.*"
        timeout: 10s
plugins:
  allowed:
    - provider: relay
      capabilities: [queue.publish]

queue.publish: the plugin receives the message's topic (or config.topic), its id as the key, and its payload as a JSON body.
webhook.deliver: the plugin receives the request the webhook bridge would have sent — the body above, X-Outbox-Payload-Encoding and, with config.secret, X-Nucleus-Signature — addressed to config.url. The consumer verifies it exactly as a direct delivery.

At boot the bridge checks that the plugins block allows the provider to run the capability, that the executable is on PATH and that it advertises the capability; if any check fails, the application does not start. With no allowlist it runs and logs one WARN (from v2.0.0 it must be listed, DEP-2026-014).

The plugin's exit code decides what happens to the message. A validation error (10) or a rejection (30) that the plugin does not mark retriable sends it to the dead letter on that attempt; anything else — a transient error, a timeout, a crash — is retried with the outbox's backoff until outbox.max_retries. nucleus outbox requeue brings dead-lettered messages back. Every envelope's metadata carries outbox_message_id, so a plugin that must not act twice can deduplicate: delivery is at least once.

The example plugin, nucleus-plugin-relay, serves both capabilities — a directory queue for queue.publish, an HTTP request for webhook.deliver — and its test runs it behind an application's outbox; its in-process counterpart, a module that registers an outbox bridge written in Go, is in Writing a module. The envelope and the exit codes are in the Plugin SDK reference.

Mail (`pkg/mail`)

Four drivers ship out of the box:

Driver	Use
`noop`	The default: accepts every message and delivers none.
`memory`	Tests: keeps every message, in order (`mail.MemorySender`); `nucleustest` selects it when the driver would be `noop`.
`log`	Development: writes every message, its links included, to the application log at `WARN` and delivers none. Refused outside `env: development`, at configuration load and at boot — a password-reset link in a shipped log is an account takeover.
`smtp`	Anything that speaks SMTP (`smtp_host`, `smtp_port`, `smtp_user`, `smtp_pass`).

An application built WithoutDefaults() — the api starter — builds no mail sender at all; WithMail() builds the one the configuration declares, and changes nothing on the default stack. Without that option a mail_driver other than noop is ignored, and the boot log says so in one ERROR line naming it; from v2.0.0 that configuration refuses to start (DEP-2026-015). A module whose flows depend on mail reaching a person asks mail.Discards(rt.Mailer()) at start-up and refuses to start rather than answering as if it had sent: the account flows do.

Vendor-specific HTTP providers (SendGrid, Mailgun, AWS SES, Postmark, Resend, …) install as nucleus-plugin-<provider> binaries on PATH and are discovered via the capability-style external bridge (pkg/plugins). The mail.send capability contract is documented in the Plugin SDK reference. Written in Go, a provider is one handler passed to plugins.Serve, which speaks the envelope and the exit codes for it; the repository's example plugin, nucleus-plugin-maildir, delivers into a Maildir and is built and run through the real runtime by CI — the starting point to copy. nucleus plugin test --provider <name> --execute sends a plugin a real request envelope and fails with its exit code and stderr.

The configuration can name the plugins that may run (opt-in until v2.0.0, when it becomes the rule):

plugins:
  allowed:
    - provider: sendgrid
      capabilities: [mail.send]

With an allowlist set, a mail_driver it does not list fails at boot before the binary is executed; without one, an application that selects an external mail plugin logs one WARN saying it will need listing.

HTML, attachments and templates

A Message carries a plain-text Body and, optionally, an HTML alternative and attachments:

msg := mail.Message{
    From:    "no-reply@example.com",
    To:      []string{"ana@example.com"},
    Subject: "Confirm your address",
    Body:    "Open https://app.example.com/verify?t=abc",
    HTML:    `<p>Open <a href="https://app.example.com/verify?t=abc">the link</a></p>`,
    Attachments: []mail.Attachment{{
        Filename:    "terms.pdf",
        ContentType: "application/pdf",
        Content:     pdfBytes,
    }},
}

Body stays required when HTML is set. A message with no text alternative is what a text-only client and a spam filter both receive badly, so the framework emits multipart/alternative with the text part first, per RFC 2046. Attachments wrap the body in a multipart/mixed; an attachment with Inline: true and a ContentID is rendered in place by an HTML body that refers to it as cid:<ContentID>.

A message that carries only Body is emitted exactly as before: one text/plain part, byte for byte.

Wording lives in templates rather than in string literals:

//go:embed mailtemplates
var mailFS embed.FS

tmpl, err := mail.ParseFS(mailFS, "mailtemplates/*.tmpl")
msg, err := tmpl.Render("verify", data, mail.Message{
    From: "no-reply@example.com",
    To:   []string{user.Email},
})

One name maps to up to three files — verify.subject.tmpl, verify.txt.tmpl (both required) and verify.html.tmpl (optional). The HTML half is parsed by html/template and the others by text/template, so a username rendered into an HTML mail is escaped per context while the plain-text half is left alone. A rendered subject containing a newline is refused, not trimmed: that is how header injection starts.

Queueing mail with the write it announces

A verification email sent straight from a handler has a window nothing closes: the database commits, the process dies, and an account exists whose owner never got the link — or the mail goes out and the transaction rolls back, and the link points at nothing.

mail.EnqueueTx puts the message in the outbox inside the caller's transaction, so the mail and the row commit together or neither does:

tx, err := db.BeginTx(ctx, nil)
// ... insert the user ...
if _, err := mail.EnqueueTx(ctx, outboxStore, tx, msg); err != nil {
    return err
}
return tx.Commit()

Delivery happens later, through a bridge registered for the mail.OutboxTopic topic, with the outbox's own retries and backoff:

router.Register(mail.OutboxTopic, mail.NewOutboxBridge(sender))

mail.Enqueue queues outside a transaction; it has the same window as sending directly, minus the retry, so prefer EnqueueTx whenever the mail accompanies a write.

Circuit breaker (mail)

App.New automatically wraps mail.Sender.Send with a pkg/circuit.Breaker. The noop driver and the Healthy SMTP HELO probe (used by /healthz) are never wrapped — so health checks can observe that a mail relay has recovered while Send is still short-circuited.

When the breaker is open, Send returns circuit.ErrOpen. The default thresholds are:

mail_circuit_breaker:
  enabled: true
  failure_threshold: 5
  cooldown: 30s
  half_open_max_concurrent: 1

Set enabled: false to disable. Config keys are documented in the Configuration reference.

Real time (`pkg/realtime`)

Pushing something to a browser used to mean bringing all of it: the upgrade, the framing, the ping/pong, the map of connections and the fan-out, in every application that wanted a progress bar or a notification.

hub := realtime.New(realtime.Config{Logger: logger})

r.Get("/live", func(c *nucleus.Context) error {
        return realtime.ServeSSE(c.Writer, c.Request, realtime.SSEConfig{
                Hub: hub, Topics: []string{"orders"}, User: currentUser(c),
        })
})

hub.Broadcast(ctx, realtime.Message{Topic: "orders", Event: "created", Data: payload})

Two transports. ServeSSE streams over plain HTTP — it crosses proxies that mangle upgrades, browsers reconnect on their own, and the keep-alive and the X-Accel-Buffering header that stops nginx turning a live stream into one lump are handled for you. ServeWS speaks WebSocket: the handshake, the accept key, the framing, fragment reassembly and the ping/pong are the framework's, and so are the three things that are easy to get wrong —

the origin check, which refuses cross-origin handshakes by default, because a browser sends cookies with a WebSocket handshake and does not apply CORS to it, so a permissive upgrade is a cross-site request carrying the user's session;
the message bound, refused from the frame header before the bytes are allocated, since that length is attacker controlled;
the masking rule, so an unmasked client frame is a protocol error rather than something accepted from whatever is not a browser.

Authorisation is the handler's, exactly as for any other route: decide first, then serve, and pass what you decided in as User. A channel does not get a second authorisation mechanism to drift from the first.

A slow client is disconnected, not waited for. Blocking the broadcaster makes one stalled browser everybody's problem, and buffering without a bound makes it the process's; disconnecting makes it the client's. What follows depends on the transport: an EventSource reconnects on its own and the new request resubscribes it, while a browser WebSocket does not and nothing in the package resubscribes it, so there the reconnection is code you write. Client.Dropped() is 1 after an eviction — a flag that this connection fell behind, not a count of the messages it missed.

Presence answers who is connected to a topic, one entry per connection — two tabs of one person are two entries, which is what a device list needs.

Across replicas. A hub with a relay carries broadcasts to the sockets the other instances hold, so live does not quietly become live-for-half-the-users when the second pod comes up:

relay, _ := realtime.NewRedisRelay(realtime.RedisRelayConfig{URL: cfg.RedisURL})
hub := realtime.New(realtime.Config{Relay: relay})
_ = hub.StartRelay(ctx)

Pub/sub rather than a stream, deliberately: a broadcast is only useful to the clients connected right now, and a replica that was down had none. Persisting messages nobody can receive is a queue, and the framework already has one.

File storage (pkg/storage)​

Using a storage backend Nucleus does not ship​

Circuit breaker (storage)​

Background tasks (pkg/tasks)​

The standalone cycle: build, register, run, close​

The module path: rt.Tasks()​

Module jobs and webhooks​

The durable queue (jobs_provider: sql)​

Showing the queue​

What the in-process queue holds​

Transactional outbox (pkg/outbox)​

Shutdown​

Webhook bridge: delivery contract​

Webhook bridge: HMAC signature​

Plugin bridge: an external plugin delivers​

Mail (pkg/mail)​

HTML, attachments and templates​

Queueing mail with the write it announces​

Circuit breaker (mail)​

Real time (pkg/realtime)​