Skip to main content
Version: 1.20.0

orbit/agent

The observability agent embeds in every framework process and ships events to a standalone admin server over a single Connect-RPC bidirectional stream. One agent per application node; many agents per server.

Wiring it into an app​

The agent owns its configuration type (agent.ExtensionConfig) — the framework carries no admin-specific config. Populate it directly, for example from your own config file, and pass it to agent.NewExtension together with the framework's state directory and your application's version string:

import (
    "context"
    "log"
    "os"

    "github.com/jcsvwinston/nucleus/pkg/app"
    "github.com/jcsvwinston/orbit/agent"
)

func main() {
    cfg, err := app.LoadConfig("nucleus.yml")
    if err != nil {
        log.Fatal(err)
    }
    a, err := app.New(cfg,
        app.WithExtensions(
            agent.NewExtension(agent.ExtensionConfig{
                Endpoints: []string{"https://admin.internal:9090"},
                Token:     os.Getenv("NUCLEUS_ADMIN_TOKEN"),
            }, cfg.StateDir, "v1.2.3"), // your app's version string
        ),
    )
    if err != nil {
        log.Fatal(err)
    }
    if err := a.Run(context.Background()); err != nil {
        log.Fatal(err)
    }
}

Endpoints is the switch. Leave it empty and the extension is a no-op: the framework runs exactly as it would without it. Set it and the agent starts in parallel with the framework's Run, and observability events flow from the framework's pkg/observability bus into the stream.

TLS​

The scheme of each endpoint picks the transport: http:// is cleartext HTTP/2 (h2c, for development), https:// performs a real TLS handshake and negotiates HTTP/2 through ALPN. By default https:// trusts the system store; for a private CA, or when the admin server requires client certificates (--agent-client-ca), pass a *tls.Config in TLS:

pool := x509.NewCertPool()
pool.AppendCertsFromPEM(caPEM)
cert, _ := tls.LoadX509KeyPair("agent.crt", "agent.key")

agent.NewExtension(agent.ExtensionConfig{
    Endpoints: []string{"https://admin.internal:9090"},
    TLS: &tls.Config{
        RootCAs:      pool,                     // private CA
        Certificates: []tls.Certificate{cert}, // client certificate (mutual TLS)
        MinVersion:   tls.VersionTLS12,
    },
}, cfg.StateDir, appVersion)

TLS itself cannot come from a config file, but the PEM files can: four fields name them, and the agent loads them on top of whatever TLS carries (nil included). Set them in code or, if your application unmarshals its own configuration into ExtensionConfig, through their koanf keys:

agent.ExtensionConfig{
    Endpoints:     []string{"https://admin.internal:9090"},
    TLSCertFile:   "/etc/orbit/agent.crt", // client certificate (mutual TLS)
    TLSKeyFile:    "/etc/orbit/agent.key", // both or neither
    TLSCAFile:     "/etc/orbit/ca.crt",    // private CA that signed the server
    TLSServerName: "",                     // when the endpoint host is not a name on the certificate
}
// koanf keys: tls_cert_file, tls_key_file, tls_ca_file, tls_server_name

agent.ExtensionConfig.TLSConfig() is the resolution the extension applies: the certificate file replaces TLS.Certificates, the CA bundle becomes TLS.RootCAs, the server name overrides the endpoint's host. A certificate without its key, a missing file or a CA bundle with no PEM certificate fail the boot with the field named.

The certificate rotates without a restart. The agent serves its client certificate from the files: every handshake checks whether the two files changed (size or modification time) and re-reads them when they did, so the next connection presents the new certificate. Under a live stream that is the next reconnect; a rotation is a write to two files. Write the certificate and the key as a pair — a key that does not match yet keeps the previous certificate in use, with one WARN, until the pair is whole. Keep the Common Name across rotations when the server binds node identity to the certificate: the node's name is fixed at boot. The CA bundle is read once. The /healthz probe the agent sends before opening a stream uses the same configuration and carries no token.

Data Studio through the contract​

The agent serves the fleet's Data Studio through Orbit's datasource contract — the same one the in-process panel speaks. By default it builds the Nucleus adapter over the application's model registry and database handles; an application on the Quark ORM hands its quarkdatasource adapter to ExtensionConfig.DataSource (set in code, as for orbit.Config.DataSource) and the fleet browses and edits Quark models.

Each request carries the operator the admin server resolved. The agent runs it as the panel would: the framework identity reaches your model hooks (auth.ClaimsFromContext), your policy applies per model and verb through the agent's Authorizer (*authz.Enforcer decides directly; a policy source that only exposes rows is compiled into one), and an operator scoped to a tenant is confined to the rows whose tenant column says so. A request without an operator — an admin server older than v0.15.0 — runs with no identity, no policy and no tenant, exactly as before; the server's own gates (mutation allowlist, read-only role) stay in front either way.

Node identity​

The agent resolves a stable NodeID: a UUIDv4 persisted at ${state_dir}/node_id, falling back to a hostname-derived ephemeral value when the state directory is unavailable.

This is the identity the agent registers under, and the value every fleet view keys on — the Nodes page, per-node stream filters, and the metrics cards.

With a client certificate from files and no node_id, the node is named after the certificate's Common Name. The server authenticates that name at the handshake (agent:<CN>), so writing it once, in the certificate, is what lets a server that binds node identity to the certificate (--agent-identity-from-cert) accept the agent. An explicit node_id still wins — and is then what such a server compares with the certificate: if the two differ, the registration is refused with PermissionDenied and the server log names both. A server without that flag registers the declared node_id and logs a WARN when it differs from the certificate. Events shipped over the stream carry the same NodeID, so an event's node_id always matches a registered node. (The agent stamps it over the in-process bus's own node label, which is host-local and does not correlate with the fleet registry.)

When the stream is down​

Losing the server does not lose the events. While the agent has no stream it keeps listening to the application's event bus under the subscriptions the server had, and parks what it hears in its per-type ring buffer; the next stream sends the parked events right after registering, before anything new. The buffer is bounded (default 64 per type, the same buffer that absorbs backpressure on an open stream), so an outage longer than it keeps the newest events per type and counts the rest in admin_agent_events_dropped_total{reason="buffer_full"}; what was parked is counted in admin_agent_events_parked_total. An agent with no subscriptions when the stream ended parks nothing: the server had asked for nothing.

Following the fleet​

An admin server may tell the agent that another server owns its node and where that server is. The agent goes — it ends the stream and dials that endpoint first from then on — only if the endpoint is in its own list. A redirect to a server the operator did not configure is logged and ignored, and the agent stays where it is: the servers decide the fleet's layout, the operator decides where an agent may connect. Redirects the agent followed are counted in admin_agent_redirects_total.

Hot-path cost​

The agent never blocks the framework's request thread. Every producer-side path — the HTTP middleware, the SQL observer — starts with a single atomic load on observability.Bus.HasSubscribers(kind) and short-circuits when nobody is watching.

What's inside​

The agent is layered:

  • Node identity — resolution and persistence, as described above.
  • Event pipeline — conversion and sampling, then a drop-oldest ring buffer that absorbs backpressure while the stream is open (it does not buffer across disconnects).
  • Transport — an endpoint-failover dialer with exponential backoff, and the bidirectional stream lifecycle: registration, recv/send/heartbeat, and replay on reconnect.
  • Metrics — the admin_agent_* Prometheus collectors.
  • RBAC snapshot — the read-only handler behind the fleet UI's Access control screen, wired automatically from the application's authorizer when the extension attaches. No extra configuration.

The top-level Agent exposes New, Run, NodeID, Connected, and Metrics.