# Deployment and operations

## SSH and Fly.io

### SSH endpoint

The OTP SSH listener verifies registered public keys and resolves their account
identity before accepting a command channel. Passwords, port forwarding, and
Erlang shell evaluation are disabled. SSH usernames are transport metadata,
not account identifiers. For local key generation, registration, and connection
instructions, see [local development](development.md).

Keep the host key durable and private. Production stores it with the translation
assets under `/data`; application records and member public keys live in PostgreSQL.

#### Public SSH on the standard port

For a public endpoint such as `ssh admin@bibleit.app`, the public TCP port
should be `22`. The application does **not** need to run as root to achieve
this: keep Bibleit listening on an unprivileged internal port (for example,
`2222`) and let the platform map public port `22` to it.

On Fly.io, map public port `22` to the OTP listener’s unprivileged internal
port `2222`. [`fly.toml`](https://github.com/bibleit-labs/bibleit-server/blob/main/fly.toml) includes this SSH service:

```toml
[[services]]
  internal_port = 2222
  protocol = "tcp"
  auto_stop_machines = false
  auto_start_machines = true
  min_machines_running = 1

  [[services.ports]]
    port = 22
    handlers = ["proxy_proto"]
    proxy_proto_options = { version = "v2" }
```

Do not attach TLS or HTTP handlers to this service—SSH already encrypts and
authenticates its transport. The Fly app needs an IP configuration that can
accept public TCP port 22; ensure its assigned addresses support that before
enabling the service. After DNS points `bibleit.app` at the app, clients use:

```sh
ssh bibleit.app
```

### Fly.io deployment

[`fly.toml`](https://github.com/bibleit-labs/bibleit-server/blob/main/fly.toml) deploys one OTP application as
`bibleit-server`. It serves HTTP internally on port `8080` behind Fly's public
HTTPS endpoint, maps public
SSH port `22` to internal port `2222`, and persists translation indexes and the
SSH host key at `/data`. Accounts and their SSH public keys are created through the web
dashboard; private keys never enter Bibleit.
Bind the internal SSH listener to `0.0.0.0` for Fly Proxy's IPv4 backend route.
The first deployed check found that binding only to `fly-local-6pn` left the
private listener running but public port 22 returned no SSH banner.

#### Public and internal ports

Keep the application on unprivileged internal ports and expose the standard
public ports through Fly Proxy:

| Protocol | Public port | Internal port / behavior |
| --- | --- | --- |
| SSH | 22 | PROXY v2 forwarding to 2222; SSH provides transport encryption |
| HTTP | 80 | Redirect to HTTPS on 443 with `force_https = true` |
| HTTPS and secure WebSockets | 443 | Fly terminates TLS and forwards HTTP/WebSocket traffic to 8080 |

The existing `[http_service]` in `fly.toml` already exposes public ports 80 and
443 with `internal_port = 8080`; no additional HTTP `[[services.ports]]` entries
are needed. Retain `force_https = true`, secure browser cookies and an HTTPS
`BIBLEIT_PUBLIC_URL`. Public port 80 remains available as the redirect entry point.
See [Fly's HTTP service configuration](https://fly.io/docs/reference/configuration/#the-http_service-section).

#### Earlier startup snapshot: local verification, 11 October 2026

The results in this section describe an earlier source snapshot. The current
image's behavior and remaining gates are documented below; the earlier UID 100
and architecture results do not validate the latest image.

The local Docker startup check passed after replacing the dashboard fallback's
undefined `bibleit_plan:free/0` call with `bibleit_plan:plan(starter)`. The
fallback's plan definition and access record now both select the starter plan.
The rebuilt OTP 29 release has no undefined-function or missing-function warning.

The check used a temporary source snapshot, isolated containers and network,
random loopback ports, disposable PostgreSQL 18 and separate named volumes.
It did not use the security assessment's database or build directories. All
test containers, volumes, the network and the tagged image were removed afterward.
No Fly resources or production databases were accessed during this check.

| Check | Result |
| --- | --- |
| Docker build and native library | Release assembled; translation NIF loaded; missing input returned the expected `open_failed` error |
| HTTP startup | `/`, `/auth/login`, `/docs`, `/plans`, `/healthz` and `/readyz` returned 200 |
| Database bootstrap | All seven packaged migrations applied using a non-superuser `bibleit` role; pool opened four connections |
| IPv6 database connection | IPv6-only test hostname established four live IPv6 connections; `/readyz` returned 200 |
| Runtime and storage | Container ran as UID 100; mounted `/data` was writable; SSH host key generated |
| SSH listener | Published local port returned an SSH banner; authenticated SSH was not exercised |
| Database outage and recovery | `/readyz` returned 503 while `/healthz` remained 200; readiness recovered without restarting the application |
| Shutdown and persistence | SIGTERM exited cleanly in approximately 0.21 seconds; database row, migration history, volume file and host key survived restart |
| Container recreation | IPv6 test recreated the application container and retained its SSH host key on the named volume |
| Dashboard fallback | Exact fallback source was exercised in a temporary probe for a missing disposable account; plan and access both selected starter |
| Environment isolation | Workspace `.env` was absent from the image |
| Idle memory | Approximately 102.7 MiB of the 512 MiB allocation; this is not a load benchmark |

These results cover local startup, not the completion of the separate security
assessment. OAuth, Resend, Turnstile, authenticated SSH, live WebSocket flows
and translation import were not exercised by this startup check. The first image
was built for the local Docker architecture; the additional amd64 verification
below covers the intended deployment architecture under local emulation.
Fly's proxy, TLS and actual volume permissions still need verification.

#### Earlier linux/amd64 snapshot and local emulation workaround

A subsequent Docker build and startup check explicitly targeted `linux/amd64`,
with disposable PostgreSQL 18 also running as amd64. Docker image metadata
confirmed `linux/amd64`; both containers reported `x86_64`, and the Erlang
runtime reported `x86_64-pc-linux-musl`. OTP 29 and the translation NIF loaded
successfully, with no undefined-function or missing-function release warnings.

The amd64 run repeated the HTTP, SSH banner, seven-migration, non-superuser role,
four-connection pool, IPv6-only database, outage/recovery, volume permissions,
shutdown and persistence checks successfully. SIGTERM exited cleanly in
approximately 0.20 seconds. Idle memory was approximately 139.7 MiB under
emulation with a 512 MiB limit; this is not a native-performance benchmark.
The workspace `.env` was absent, and all disposable test resources were removed.

The Apple Silicon host needed an emulation-only workaround. The unmodified
Dockerfile's first amd64 build failed before application compilation with
`prim_tty:tty_create/1` reporting `undef`; the same failure occurred in the bare
amd64 `erlang:29-alpine` image. Erlang maintainers document this user-space JIT
emulation problem and the `+JMsingle true` workaround in
[OTP issue 10355](https://github.com/erlang/otp/issues/10355).

Only the temporary test Dockerfile's build command was changed to
`RUN ERL_AFLAGS='+JMsingle true' make release`. The disposable application's
Compose environment also set `ERL_AFLAGS=+JMsingle true` for runtime checks.
The flag was not embedded in the final image environment. The repository
Dockerfile, `fly.toml`, security assessment files and production resources were
unchanged by this architecture check.

This earlier run verified an amd64 image under local emulation with that flag;
native hardware was not exercised by that run. The separate native verification
below supersedes its architecture limitation for the reviewed frozen snapshot.
The Dockerfile's failure under local emulation does not establish a failure on
native amd64. Keep the workaround
in local test settings; use the normal Dockerfile and runtime settings for the
native amd64 deployment build.

#### Current image: verified local behavior

The subsequent local release review rebuilt and tested the current image on
native ARM64 with disposable PostgreSQL 18. These results supersede the older
snapshot for packaging, permissions and runtime behavior. Native amd64 was then
verified separately as documented below; neither run validates deployed Fly boundaries.

The Dockerfile pins a reviewed multi-platform OTP base digest and upgrades Alpine
packages in both stages. The fixed-advisory comparison found no remaining matches
in the tested OS inventory; this does not establish that every dependency is free
of vulnerabilities. Build-time Python stages the selected public resources;
the runtime image excludes the source documentation tree and private assessment
records. Nested environment/key files are excluded from the build context.

| Boundary | Current locally verified behavior |
| --- | --- |
| Runtime permissions | UID/GID 10001; root-owned application directory/files; fresh data volume writable with capabilities dropped and no-new-privileges at 512 MiB / 1 CPU |
| PostgreSQL bootstrap | Restricted role without superuser, database-creation or role-creation privileges; seven migrations with filename/checksum identities; four connections |
| PostgreSQL TLS | Default `verify-full` succeeds with trusted CA and matching hostname; all four connections encrypted; unknown CA or wrong hostname prevents startup |
| Public origin | Secure-cookie mode requires an HTTPS `BIBLEIT_PUBLIC_URL` containing only an origin; browser Host/Origin must match that configured origin |
| HTTP and resources | HTTP/HSTS behavior checked locally; private resources denied |
| SSH | Registered-key protocol commands work; revoked keys denied; protocol-only execution and pinned host-key persistence verified |
| Host-key initialization | Listener system directory respected; private identity preserved; missing public key recoverable; restrictive permissions enforced; malformed, symlinked, partial or mismatched keys rejected as appropriate |
| Listening ports | Application listens only on 8080 and 2222; inbound distribution/EPMD disabled; NIF loads through an offline probe |
| Recovery | Database outage yields readiness 503 and liveness 200; recovery succeeds; SIGTERM exits normally in approximately 0.2 seconds; restart preserves the host key |
| Filesystem failure | Unwritable data mount fails closed; runtime does not switch to root to repair it |
| IPv6 | Private IPv6 database destination and IPv6 SSH tested locally |
| Local TLS relay and WSS | Real TLS termination into the cleartext Cowboy backend, canonical origin/secure cookies, authenticated management/private-viewer revocation and interruption/reconnect cleanup passed |
| Legacy volume recovery | UID 100 ownership causes safe startup failure; approved offline repair of explicitly listed paths to UID/GID 10001 preserves both host-key files and a persisted asset in a disposable fixture |

Local image checks and the full application suite passed. The review used only
disposable resources, which were removed. No Fly APIs, deployment tokens,
certificates, edge routing, actual private networks or shared database were tested.
Keep detailed assessment records private; this guide documents operator behavior
and verification limits rather than the assessment ledger.

`fly.toml` now declares TLS 1.2/1.3, ALPN `h2`/`http/1.1`, no self-signed
certificate fallback, HTTP request concurrency soft/hard limits of 64/128 and SSH
connection limits of 96/128. Local parsing and policy checks passed. Fly platform
validation subsequently passed for the deployed configuration. Enforcement of
these provisional concurrency limits is not a capacity or fairness guarantee.

Proxy headers remain untrusted by default. Admission uses the socket peer,
which can group unrelated clients behind Fly. The optional
`BIBLEIT_HTTP_TRUSTED_PROXY_IPS` is a comma-separated list of exact numeric
socket-peer addresses. Only those peers may supply a single, strictly parsed
`Fly-Client-IP`; `X-Forwarded-For` is always ignored. Missing, malformed or
duplicate client headers share the socket-peer budget. CIDRs, DNS names and
duplicate trust entries are rejected at startup. This affects HTTP/CLI
admission and WebSocket admission/frame limits, while global budgets remain.

Keep the list empty until public header-overwrite behavior, all accepted proxy
peers, independent-client fairness and direct/private bypass denial have been
verified on the selected deployment. Do not trust an entire private network.
Repeat this qualification after a Machine replacement, address or routing
change. The deployed configuration trusts only the qualified exact peer `172.16.41.130`
for HTTP and SSH. Paired tests from two owned public egresses verified that
forged client headers/preambles cannot evade a denied client budget while the
independent client remains admitted in the same window. Natural recovery passed
without a restart. This qualifies the observed route, not future proxy addresses.
Concurrency settings do not establish capacity or replace client admission.

SSH supports the separate opt-in `BIBLEIT_SSH_TRUSTED_PROXY_IPS`, with the same
exact numeric address validation. A trusted transport peer must send a binary
PROXY v2 TCP/IPv4 or TCP/IPv6 header before the ordinary SSH exchange. Configure
Fly's public SSH port with `handlers = ["proxy_proto"]` and
`proxy_proto_options = { version = "v2" }` together with this setting. The
application reads at most 512 payload bytes within a total two-second deadline;
missing, malformed, oversized, LOCAL, datagram and v1 headers close the socket.
Untrusted peers retain raw SSH and their real peer allowance. Forwarded clients
retain the existing global and per-client connection budgets. Clients and CLI
tools continue sending ordinary SSH; Fly injects the internal preamble.

Qualify public admission, spoofing resistance, independent-client denial and
natural recovery before accepting this configuration. Rollback must restore
both the port handler and the matching application trust configuration/image.
A new proxy address requires requalification; a prefix from an unlisted peer is
ordinary SSH payload and fails protocol negotiation. Do not widen trust to a
private CIDR to work around routing changes.

The local review and subsequent deployed checks passed. The historical native
snapshot below is retained for provenance; the final deployed artifact and
remaining operational limits are recorded in the current deployment section.

#### Native linux/amd64: verified frozen snapshot

The reviewed server snapshot at `f5f1736b723096e0e3bd80f817ea6701e7856cc1`,
with libbibleit at `f6fed37919c848b1aa0208900af992a5e8a507f2`, passed on a hosted
Ubuntu x64 runner with a real Linux x86_64 host and native amd64 Docker engine.
No emulation workaround was used. All 389 tracked server/native source hashes
matched the frozen checkouts. The eight validation stages passed, including the
native corpus, SSH bootstrap, release build, runtime checks, scoped APK advisory
comparison and all 17 isolated image/deployment checks. The updated official
CI actions ran on Node 24 with zero annotations.

The tested image identity was
`sha256:ab28d63404454a94ac1d891e3d0ae1e042a8ab559c314403c0de37f6adf42706`.
It was an ephemeral validation image, removed after the job, not a published
registry artifact available for deployment. The native architecture gate is
complete for this frozen snapshot. Subsequent documentation-only completion
changes are outside it; no main-branch merge or release selection is implied.

The native job exercised selected image/native checks, not the complete
integration/browser suite. The separate local suite passed 331 integration,
51 Erlang unit and 71 JavaScript tests, the native corpus and ten SSH-bootstrap
tests. Real local TLS/WSS fixtures strengthen the backend evidence but do not
verify Fly certificates, renewal, redirect routing, ALPN h2, private mesh or
public-client fairness. No Fly operation or registry publication occurred.

Select the immutable source and native commit for deployment, then build,
validate and record the actual registry artifact. A rebuild can change resolved
OS packages even with the base digest pinned, so do not treat a new image as the
tested digest without checking it. Keep detailed reports and assessment logs in
the private evidence store rather than the served public documentation.

#### Current deployment and verification, 11 October 2026

The first production deployment is complete. The current release is source
`7442010eebcfb1a2a9105db312ebaeae5f789003`, with immutable registry artifact
`registry.fly.io/bibleit-server@sha256:44f2278362bd73f26a2f1c4abb78bf4d72eb02a99c177629656b72391c79fc11`.
[GitHub Actions run 38106224843](https://github.com/bibleit-labs/bibleit-server/actions/runs/38106224843)
built and validated it on native Linux amd64: eight stages and all 20 actual
image checks passed, with 398 source hashes matched. The complete local suite
passed 334 integration, 56 Erlang unit and 71 JavaScript tests, the native corpus
and ten SSH-bootstrap tests. No emulation workaround was used in CI or production.

The app runs one 512 MiB Machine in `ams`, with four database connections,
dedicated IPv4 and IPv6, and its encrypted 1 GiB `/data` volume. Its canonical
origin is `https://bibleit-server.fly.dev`. The dedicated database/public-schema
owner is `bibleit_server_app`, with no superuser or cluster-wide read/write role
memberships. Verified client-to-Fly-gateway PostgreSQL TLS is described below.
Other databases on `mittel-pg18-5gb` were preserved.

Final deployed evidence passed 23 HTTPS boundary checks, 11 authenticated
HTTP/WebSocket checks and four registered-key SSH authorization/revocation
checks. Public HTTP redirects to HTTPS; health/readiness are positive. Exact
proxy trust, spoof resistance, independent-client fairness in the same denial
window, and natural recovery passed. Direct private listener access from the
owned database Machine was denied. Distribution/EPMD remain disabled.
Google/GitHub OAuth, email and Turnstile secrets were applied by normal deployment;
a separate “Deploy Secrets” action is unnecessary. Turnstile's real API and
challenge document loaded on signup with a propagated 32-character CSP nonce
and no observed CSP violation. Full human OAuth consent and email delivery are
operator acceptance tasks, rather than inferred from secret presence.

The on-Fly recovery drill verified a scoped `bibleit` logical archive in a new
offline database, using the exact production PostgreSQL image, actual PostgreSQL
18.6 Ubuntu, UTF8/libc/`en_US.utf8` and collation version 2.39. All seven migration
identities, constraints, application table ownership, non-superuser role and
restored COPY rows/sequences matched. The initial Alpine fixture used a different
locale; the matching-image rerun supersedes that result. The fixture had only a
Unix socket, and its restored database and plaintext scratch files were removed.
The archive and verification report remain on an independent encrypted Fly
volume with five-day snapshots; no production database or private key was
exported locally. This establishes scoped restore integrity, not a production
RPO/RTO or a complete application recovery/credential-quarantine rehearsal.
The app snapshot was separately restored on Fly: its sole persisted non-SSH
file, `available_translations.json`, matched byte-for-byte with UID/GID 10001
and mode 600. No persisted translation index/generation files were present.
The restored private host key derived the expected public fingerprint, with
`/data` mode 750, SSH directory 700 and private key 600. No private key bytes
left Fly. Temporary restored databases, helper Machines and app clone volumes
were removed after verification; the source snapshots and independent archive
volume were retained. Synthetic test-account/session/Live/key counts were
verified zero after the final authorization checks.

Before broad rollout or later infrastructure changes:

- Complete human OAuth consent/callback and email-delivery acceptance, and measure
  translation/import workload and storage growth before tuning RAM or concurrency.
- Independently review the CI deploy token's actual scope and expiry in Fly.
  Its secret value remains in GitHub; successful deployment alone cannot prove
  least privilege. The configured creation command requests an app-scoped token.
- Set durable backup scheduling, retention, loss/recovery objectives and protected
  integration-secret recovery. Five-day snapshots are current evidence, not a
  long-term retention policy. Keep production backups on Fly as selected by the operator.
- Requalify exact proxy peers after routing/address/Machine changes. Test the new
  route before widening trust; never replace exact peers with a private CIDR.
- Verify custom-domain DNS/certificates when adopting `bibleit.app`. Certificate
  renewal, declared concurrency enforcement and capacity have not been proven
  by these bounded tests. Single-Machine updates can interrupt live sessions.
- Keep a compatible recorded rollback image/configuration. The earlier nonce
  image `sha256:6036c47a04ba7fd3af93ed14af9bb7450de14aad3e3b54a2e94679fb8a809412`
  requires its matching raw-SSH/no-proxy-trust configuration. Image rollback does
  not undo migrations; never run development rebuild scripts on production.

#### Custom-domain migration: prepared, awaiting DNS and release validation

The operator selected `https://bibleit.app` as canonical, with legacy
`https://live.bibleit.app/<live_id>` redirecting to
`https://bibleit.app/lives/<live_id>`. Certificates for `bibleit.app`,
`live.bibleit.app` and the existing `www.bibleit.app` alias have been requested
on `bibleit-server`. The operator added the four apex/www validation records;
the updated DNS export and Cloudflare authoritative answers match the requested
values. The operator also replaced Live validation records; authoritative DNS
matches the new application and all three Fly certificates are issued and active.
Traffic routing and the existing legacy certificate remain unchanged. The running
release now uses `https://bibleit.app`, but alias acceptance failed
and traffic cutover is held. Do not remove the legacy certificate or GitHub
Pages until the repaired routes and public cutover checks pass.

The frozen candidate `5969f2f73547010cddd3ae5b0828839cad118f2e` passed
[native dry run 38127551591](https://github.com/bibleit-labs/bibleit-server/actions/runs/38127551591):
all 402 server/native source hashes matched the committed revisions; eight
stages and all 20 actual-image checks passed on native Linux amd64. The scoped
APK fixed-advisory comparison had zero alerts, and disposable-image cleanup
passed. Fly installation, publication and deployment were explicitly skipped.
This dry run did not publish a deployable registry artifact; the eventual
production build must validate and record its actual immutable digest again.
Unrelated uncommitted CSS/template edits were excluded. The organization-owned
production GitHub OAuth app has the correct HTTPS callback and both credentials
were staged on Fly and applied by the canonical release; Google's canonical
callback and the existing Turnstile
widget's canonical hostname were directly confirmed by the operator. Traffic
cutover and public acceptance remain pending. The repository configuration now
selects `https://bibleit.app` with exact live/fly.dev/www read-only redirect
aliases. The first canonical deployment, [run 38128299852](https://github.com/bibleit-labs/bibleit-server/actions/runs/38128299852),
passed native validation, actual-image checks and rollout from source
`0aaf26c88cfcdeb2982da31ab2f2301e8bab9f32`, with immutable image
`registry.fly.io/bibleit-server@sha256:6da61c8f3925173247e28b08aee50496e6ec932524f5ad37a61b1bf6882af80a`.
Pre-cutover canonical boundaries passed (23/23), but alias reads returned 421
instead of 308 because the runtime environment parser omitted
`BIBLEIT_HTTP_REDIRECT_HOSTS`. Traffic cutover is held pending a validated repair
and successful alias checks. Public routing, Pages and the legacy certificate
have not changed.

Cloudflare is authoritative. The apex and www currently use its proxy; live
points directly to the legacy Fly app. DNS-only routing is required for ordinary
SSH on public port 22 under the same canonical hostname. Preserve mail, Resend,
CAA and other unrelated provider records throughout the migration.

The operator's DNS export confirms the original origin records below. Public
Cloudflare A/AAAA answers conceal these upstream targets, so use the export
when preparing rollback; do not reconstruct the old origins from public lookup.

| Host | Original origin records | Original Cloudflare mode |
| --- | --- | --- |
| `bibleit.app` | A `185.199.108.153`, `185.199.109.153`, `185.199.110.153`, `185.199.111.153`; AAAA `2606:50c0:8000::153` through `2606:50c0:8003::153` | Proxied |
| `www.bibleit.app` | CNAME `mittel-labs.github.io.` | Proxied |
| `live.bibleit.app` | A `66.241.124.67`; AAAA `2a09:8280:1::115:f2bc:0` | DNS-only |

Replace all four old apex A records and all four old apex AAAA records during
cutover. Replace www's old CNAME with the new Fly CNAME
`mdonzjm.bibleit-server.fly.dev.` (DNS-only), or remove the CNAME before using
the new app's A/AAAA pair. Never mix old and new origins or put a CNAME alongside
A/AAAA for the same name. Preserve MX, SPF, DMARC, DKIM/wildcard DKIM,
Resend verification/sending CNAMEs, and zone NS/SOA records unchanged.

| Host | New ACME CNAME target | New ownership TXT value |
| --- | --- | --- |
| `_acme-challenge.bibleit.app` | `bibleit.app.mdonzjm.flydns.net.` | — |
| `_fly-ownership.bibleit.app` | — | `app-mdonzjm` |
| `_acme-challenge.live.bibleit.app` | `live.bibleit.app.mdonzjm.flydns.net.` | — |
| `_fly-ownership.live.bibleit.app` | — | `app-mdonzjm` |
| `_acme-challenge.www.bibleit.app` | `www.bibleit.app.mdonzjm.flydns.net.` | — |
| `_fly-ownership.www.bibleit.app` | — | `app-mdonzjm` |

Prepare apex/www validation records first, without changing traffic routing.
Replacing live's old ACME target and ownership value affects its legacy renewal
path: coordinate that replacement, keep its existing certificate during cutover,
and verify the new certificate before removing the old Fly association.

1. Register Google and GitHub callbacks at
   `https://bibleit.app/auth/google/callback` and
   `https://bibleit.app/auth/github/callback`; allow `bibleit.app` in Turnstile.
   Retain the old callbacks during the transition where providers permit it.
   Email links use the new public origin; changing the origin does not require
   changing the verified sending domain or exposing integration secrets.
2. Verify all required new Fly certificates are issued. Build/test the canonical
   host and exact alias redirect changes, then select the immutable native amd64
   artifact. Set `BIBLEIT_PUBLIC_URL=https://bibleit.app` and exact
   `BIBLEIT_HTTP_REDIRECT_HOSTS` matching the tested alias implementation.
3. Deploy the validated artifact, test the new TLS/SNI and Host routes against
   the new Fly addresses before switching DNS, then replace the selected
   apex/live A/AAAA routes (and www as described above) with `137.66.54.95` and
   `2a09:8280:1::1b0:958e:0`, DNS-only. Do not leave an old AAAA alongside the
   new A record. Record existing origin targets privately for rollback.
4. Verify public TLS, HTTP redirects, the legacy Live path mapping, canonical
   cookies/Origin, OAuth/Turnstile, WSS and pinned-key SSH. Requalify exact proxy
   peers, spoof resistance and independent-client fairness on the new route;
   never widen trust to a private CIDR or trust Cloudflare headers by default.
5. Once the new live hostname works and DNS has converged, remove only the
   `live.bibleit.app` certificate from legacy app `bibleit`. Preserve that app,
   its database, volumes and other resources. Roll back routing/configuration
   together if acceptance fails; do not downgrade TLS or reset schema/data.

#### First-deploy acceptance checks

After the first deployment is authorized and performed, verify Fly checks,
the public HTTP-to-HTTPS redirect on port 80, HTTPS on 443, `/healthz`, `/readyz`,
database migration identities and TLS policy, mounted-volume ownership for
UID/GID 10001, and authenticated SSH with host-key verification. Verify trusted
proxy admission and client fairness, declared concurrency limits, private-network
paths and the absence of inbound distribution/EPMD. Exercise sign-in, enabled OAuth/Turnstile/email
flows, translation import and live WebSockets, then restart the Machine and
confirm durable records, translation files and the SSH host key survive. Watch
memory during import and live traffic; adjust RAM and volume capacity from
measurements. Record the deployed image and confirm the backup procedure before
opening the service to general use.

The application creation command below is retained for reference; the operator
has already created `bibleit-server`. Confirm it is in the same organization as
the existing PostgreSQL cluster `mittel-pg18-5gb`. Both the cluster and
`fly.toml` use `ams`:

```sh
fly apps create bibleit-server --org mittel
fly volumes create bibleit_data --app bibleit-server --region ams --size 1
fly postgres attach mittel-pg18-5gb --app bibleit-server \
  --database-name bibleit --database-user bibleit_server_app --superuser=false
```

The attach command creates the new `bibleit` database and a dedicated role,
and sets the application's `DATABASE_URL` secret. Use it before the first
deployment; the server requires that secret to start. It does not reuse the
cluster's other databases. If you create `bibleit` manually first, ensure the
application role has database CONNECT and public-schema USAGE/CREATE privileges,
and owns the migration objects it creates. Fly attachment may retain `postgres`
as database owner; transfer only `bibleit` and its public schema to
`bibleit_server_app` when following this deployment's selected ownership policy.
The application does not need cluster-wide or database-creation
privileges to apply its packaged migrations.
Inspect role memberships after attachment: this cluster's attachment granted
`pg_read_all_data` and `pg_write_all_data` even with `--superuser=false`.
Revoke those memberships from the newly created application role before startup;
retain only the database/schema privileges needed for its own migrations.
Verify that the role cannot read or modify other applications' tables.
The attach flow currently generates `sslmode=disable` and prints the credential
URI. Review that result privately and keep it out of shared logs. Explicitly
decide the private database transport/trust policy before starting the app;
attachment alone does not verify TLS. Missing `sslmode` defaults to `verify-full`,
which requires a trusted CA and matching hostname; there is no fallback to
plaintext. Do not change TLS mode merely to suppress a startup failure.
The server applies its packaged SQL migrations at startup under an advisory
lock; no separate Fly release command is needed. Do not run the development
database rebuild script against this cluster. Bibleit's database client supports
the IPv6-only private hostname used by Fly Postgres.
Applied migrations are checked against their recorded filenames and SQL hashes.
Identity-less or mismatched journals fail closed; do not blindly backfill hashes
or reset an existing database. A new, dedicated `bibleit` database is the planned
bootstrap target.

`BIBLEIT_PUBLIC_URL` is set to `https://bibleit.app` in `fly.toml` and the running
canonical release. Traffic cutover awaits the alias repair.
Use the same public origin for OAuth callback registration and Turnstile's
allowed hostname. The URL must contain only an origin, with no application path, query or fragment;
HTTPS is required when secure cookies are enabled. Browser requests must use
the configured canonical Host/Origin, so align proxy/DNS and provider callbacks.

The selected database transport uses the cluster's existing private Flycast
proxy, with `sslmode=verify-full`. The connection URL retains the proxy's
`mittel-pg18-5gb.fly.dev` certificate hostname, while a numeric `hostaddr`
selects its private address. A credential-free TLS 1.3 probe verified the
managed certificate chain and hostname. TLS terminates at Fly's `pg_tls`
proxy; the origin PostgreSQL server has SSL disabled, and its connection leg
uses the encrypted private mesh. This verifies the client-to-proxy TLS boundary,
not end-to-end PostgreSQL-server TLS. Do not replace this policy with the
attachment's generated `sslmode=disable`, or expose the shared database publicly.
Keep the actual private address and credentials in the runtime secret.

Configure the integrations you enable through Fly secrets:

| Integration | Environment variables |
| --- | --- |
| Google OAuth | `BIBLEIT_GOOGLE_CLIENT_ID`, `BIBLEIT_GOOGLE_CLIENT_SECRET` |
| GitHub OAuth | `BIBLEIT_GITHUB_CLIENT_ID`, `BIBLEIT_GITHUB_CLIENT_SECRET` |
| Email | `BIBLEIT_RESEND_API_KEY`, `BIBLEIT_EMAIL_FROM`; optional `BIBLEIT_EMAIL_REPLY_TO` |
| Turnstile | `BIBLEIT_TURNSTILE_SITE_KEY`, `BIBLEIT_TURNSTILE_SECRET_KEY`; hostname defaults to the public URL's host |

Set each enabled integration's required variables together, with nonempty
values. For example, use `fly secrets set --app bibleit-server NAME=value ...`
with actual credentials. The repository's `.env` is excluded from Docker builds.
Register OAuth callbacks at `/auth/google/callback` and `/auth/github/callback`
on the configured public URL, and verify the email sender domain in Resend.

Deploy one Machine initially, since translation assets, SSH host keys and live
subscriptions currently belong to that Machine. Fly volumes are local storage;
additional Machines need an asset and live-subscription distribution strategy.

```sh
git submodule update --init --recursive
fly config validate --strict --config fly.toml
fly deploy --config fly.toml --ha=false
fly checks list --app bibleit-server
curl --fail https://bibleit-server.fly.dev/readyz
```

Use the selected canonical hostname in the smoke checks; substitute
`bibleit-server.fly.dev` if that is the configured initial public URL. Keep TLS
and Origin enforcement enabled while testing the eventual public route.

#### GitHub Actions deployment

`.github/workflows/fly-deploy.yml` provides a manual workflow with `dry_run`
enabled by default. In that mode it builds and validates on a native Ubuntu
amd64 runner using disposable PostgreSQL, without a Fly token, registry push
or Fly API operation. It first runs the existing native snapshot validation,
then builds and rechecks the actual deployment image because a rebuild can
resolve different OS packages. It does not replace the complete integration
and browser suite or the first-deploy acceptance checks above.

After the remaining prerequisites and release review are complete, create an
app-scoped deploy token locally and save its entire value as the repository's
GitHub Actions secret `FLY_API_TOKEN`:

```sh
flyctl tokens create deploy --app bibleit-server --expiry 720h
```

This example expires after 30 days; renew or choose an appropriate lifetime.
Keep the token out of commits and shared output. `FLY_API_TOKEN` authorizes CI
deployment; application runtime secrets such as `DATABASE_URL` must separately
be configured on Fly. Creating the app and token does not provision its database,
volume, public addressing or hostname.

Commit and review the workflow, then run it from GitHub Actions with `dry_run`
enabled first. A dry-run dispatch does not publish an image or deploy the app.
For a deliberately authorized deployment, run it from `main` with `dry_run`
disabled. That mode validates `fly.toml`, pushes the checked image to the app's
registry and deploys its registry digest with `--ha=false`; it does not rebuild
the image remotely. Deployments are serialized and an active deployment is
not cancelled by a newer run. No automatic push-triggered deployment is enabled
while first-deploy and security rollout checks remain open. Action revisions
and flyctl are pinned; review their updates periodically. Store detailed CI
validation artifacts privately according to repository access policy.

Fly's HTTP routing check uses `/readyz`, which includes a SQL canary; `/healthz`
remains available for process liveness during database outages. The 30-second
check grace period accommodates startup, migrations and translation loading;
increase it if measured startup takes longer. Keep the Machine running for
long-lived WebSocket and SSH sessions. The pool opens four database connections
per Machine; the existing databases share the cluster's connection budget.
The initial 512 MiB RAM allocation should be checked under actual translation
and import load and increased if necessary.

Point the HTTP and SSH hostnames at this Fly application. For SSH access from
IPv4 clients, allocate a dedicated IPv4 address with
`fly ips allocate-v4 --app bibleit-server`; shared IPv4 addresses do not route
raw TCP port 22. Add the custom domain certificate with
`fly certs add bibleit.app --app bibleit-server` and configure the DNS records
shown by Fly. Ensure `/data` and the configured SSH system directory are writable
by UID/GID 10001 on the mounted volume before serving traffic. The release files
remain root-owned, while the application stays non-root. An unwritable mount
fails startup rather than triggering a root bootstrap. For an existing UID 100
volume, perform only an approved offline ownership migration with the original
SSH private key preserved and checked; do not rotate its identity accidentally.
The entrypoint generates a host key only when the validated configured keypair
is absent and retains it on subsequent starts.

Inbound distributed Erlang and EPMD are disabled. Release `rpc`/`eval` diagnostics
that depend on distribution are not an operational prerequisite; use offline
release/native probes instead of reopening a listener or packaging a public cookie.

## Persistence and deployment

PostgreSQL is the only application database. On startup the server applies the
ordered SQL files in `priv/sql` transactionally, beginning with
`001_initial.sql` and retaining the tested revisions `001` through `007`. Applied
filenames and checksums must match. Development rebuild scripts are restricted
to disposable development databases; never use them to upgrade production.
It never creates an opaque key/value table or imports historical DETS files.

The relational model separates people from the resource containers they act
within. Users have provider identities, a personal account, and memberships in
explicit organizations. Organization roles grant permissions only within the
selected workspace; personal roles cannot bypass membership. Token scopes
may only narrow that authority. Plans and per-account overrides provide quota
limits. Lives belong to accounts while connected subscribers and OTP process
identifiers remain ephemeral in memory. See
[`docs/domain-model.md`](../domain-model.md) for the table-level contract.

| Data | Storage | Notes |
| --- | --- | --- |
| Users, identities, credentials, accounts and memberships | PostgreSQL | Credentials are Argon2 hashes. Providers are not linked merely because emails match. |
| Plans, account access, roles, permissions and quotas | PostgreSQL | Explicit relational rows; no serialized Erlang terms. |
| Access tokens, SSH public keys and browser/auth flows | PostgreSQL | Bearer values and action codes are SHA-256 hashed at rest. |
| Lives and presentation state | PostgreSQL | Subscriber sockets and process identifiers remain in memory. |
| Translation files and indexes | Filesystem/object storage | Large immutable assets stay outside PostgreSQL for the NIF. |
| SSH host private key | Secret filesystem/volume | Never store it in PostgreSQL or Git. |

Public keys are safe to store and are persisted with their fingerprint and
usage metadata. Private keys never enter Bibleit storage. Access tokens remain
hashed at rest and are not accepted as SSH credentials or browser sessions.

For a public repository, commit schema revisions and safe configuration
examples, but never commit `.env`, OAuth/client secrets, Resend keys,
`DATABASE_URL` credentials, database dumps, generated access tokens, SSH host
private keys, or persistent volumes. Use the deployment platform's secret
store and enable GitHub secret scanning. Confirm redistribution rights before
adding any Bible translation data to the repository or a release artifact.

## OTP release shape

The long-term deployment unit is one `rebar3` release, not an `erl -eval`
command. The release contains the BEAM runtime, the application, Cowboy, the
NIF, browser assets, and bundled translation metadata. Runtime configuration
and secrets remain environment variables.

```sh
make release

BIBLEIT_DATABASE_POOL_SIZE=4 \
DATABASE_URL=postgresql://bibleit:password@localhost/bibleit \
BIBLEIT_DATA_DIR=/var/lib/bibleit \
BIBLEIT_HTTP_PORT=8080 \
BIBLEIT_SSH_BIND_ADDRESS=127.0.0.1 \
BIBLEIT_HTTP_BIND_ADDRESS=0.0.0.0 \
_build/prod/rel/bibleit_server/bin/bibleit_server foreground
```

`make release` is the build-time command. Fly runs it inside the Docker build
stage; the running Machine receives only the assembled release and starts it
with `bin/bibleit_server foreground`.

`bibleit_server_app` applies environment configuration before its OTP
supervisor starts. The supervisor owns authorization, translation and Live
actors, the native SSH listener, and the Cowboy HTTP listener. This gives
browser and SSH clients the same LiveSession actors without a network hop.

The container [`Dockerfile`](https://github.com/bibleit-labs/bibleit-server/blob/main/Dockerfile) builds that release and runs
`bin/bibleit_server foreground`; it is the production entrypoint. Keep a
separate HTTP edge only if a future scaling, isolation, or independent
web-product boundary makes it useful. It is not needed for the current Live
presentation or a future dashboard.


### Remembered sign-in

After a successful browser sign-in, `/auth/login` offers the last used identity
with its provider icon. A 90-day HttpOnly cookie contains only a random shortcut
token; PostgreSQL stores its hash in `remembered_logins`. It is not a session or
credential. Google shortcuts pass the stored identity as `login_hint` without forcing
the account chooser, allowing Google to reuse an active session. Other-account
Google sign-in still requests the chooser. Provider authentication is always
verified on callback, and Google may require sign-in or consent. Email shortcuts
prefill the address and focus the password. Users can expand the other sign-in
options or forget the shortcut with a CSRF-protected POST. Logout keeps the
shortcut; expiry, identity/account deletion, or forgetting removes it. Labels
are localized in English, German, and Portuguese. The preference is stored in
`remembered_logins`.