ALMA

checking gateway…

Open the portal

LLM cluster governance · Universidad de Sevilla

Large language models, served from the cluster you already have.

ALMA gives every researcher an SLA, an API key and a list of endpoints with their own rate limits, then keeps the model behind them alive on SLURM: submitted, tunnelled, published to the gateway and restarted without anyone watching the queue.

Request access See how a request travels

Runs on
SLURM and vLLM, across A100 / A40 / A30 nodes
Speaks
OpenAI-compatible HTTP, one apikey header
Asks of you
No VPN, no sbatch, no node to babysit
cluster map t+00s · 0 served
jobmodelstate
    Illustrative view of the real lifecycle — queued, placed, tunnelled, served. Not live cluster data.

    02 — the path

    From your terminal to a GPU, in five hops

    None of it is hidden from you. This is the whole route a request takes, in order, and what each hop is allowed to do to it.

    1. your machine

      One HTTPS call, one header

      You send apikey: <your key> to a public URL. No VPN, no SSH session, no sbatch, and nothing to install beyond the HTTP client you already use.

    2. caddy · alma.us.es

      TLS ends here

      Caddy terminates TLS, forces HTTPS and sets the security headers, then splits the host by path: /api/models/* reaches the SLA gateway, /api/* the backend, and /portal/ the portal you sign in to.

    3. sla gateway · nginx

      Your SLA, compiled

      The YAML template an administrator assigned you is compiled into an nginx configuration: which models you may reach, which endpoints on them, at what rate. A model outside your SLA is not a permission error you have to interpret — it was never routed to you.

    4. alma-telemeter

      Counted per request

      nginx asks the telemeter on every call, through auth_request. It answers with what is left of your budget in X-RateLimit-* headers and turns an exhausted one into a 429 carrying Retry-After, never a silent drop. When its Redis is unreachable it omits the headers rather than reporting numbers it cannot stand behind.

    5. ssh tunnel → vllm

      Wherever SLURM put the model

      The gateway forwards to a local port tunnelled to the compute node SLURM chose. Models are not pinned to a node, so that node is only known once the job leaves the queue — which is why every open tunnel is reconciled, every 30 seconds, against what the running models actually need.

    the call your endpoint list hands you
    curl -X POST "https://alma.us.es/api/models/<model>/v1/chat/completions" \
  -H "apikey: $ALMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'
    what comes back (example)
    HTTP/2 200
content-type: application/json
x-ratelimit-limit: 60
x-ratelimit-remaining: 59
x-ratelimit-reset: 41

    Past the limit the same request returns 429 with Retry-After, so a client can back off instead of retrying into a wall.

    or point an OpenAI client at it
    https://alma.us.es/api/models/<model>/v1

    That is a complete OpenAI base URL: a client appends /models to check the connection and /chat/completions to talk. In Open WebUI it goes in Admin Settings → Connections → OpenAI, with your key in the API Key field — one connection per model. Those clients send the key as Authorization: Bearer, which the gateway accepts alongside the apikey header.

    03 — your account

    An SLA, not a quota you have to guess

    Everything you are allowed to do is written down, visible in the portal and exportable as JSON. This is what an account looks like from the inside.

    04 — the stack

    Four services, one job each

    ALMA is not one program. It is a small set of services with sharp edges between them, so a failure in one stays legible from the others.

    servicewhat it doeshow it ships
    alma-frontend The admin console and the user portal: models, users, SLA templates, cluster configuration, live SLURM logs, the OpenAPI spec. React 18 and Vite, served under /portal/. Docker and nginx
    alma-backend The REST API and the decisions behind it: model lifecycle, SLA generation and nginx configuration, SSH tunnels, the restart and tunnel schedulers, operator alerts, multi-device sessions. Push a v1.x.y tag
    slurm-controller A thin HTTP API over sbatch, squeue, scancel and the job logs, plus GPU availability per node. The only component that touches SLURM directly. Submitted to the cluster by hand
    alma-telemeter Rate-limit accounting behind nginx auth_request, backed by Redis, with the SLA limits watched on disk and the X-RateLimit-* headers computed per call. Docker

    The controller ships by hand on purpose: it runs as a SLURM job, inside the cluster, where no CI reaches. That asymmetry is the origin of the first failure mode below.

    05 — when it breaks

    The interesting part is the failure handling

    A cluster is not a cloud. Jobs get preempted, nodes fill up, a deploy lands on one side of the system and not the other. These are the cases ALMA is written around.

    1. Version skew is named, not guessed

      The backend asks the controller what it can do — GET /health returns a capability list, a version and a commit — and reports the answer as one of OK DEGRADED MISROUTED UNREACHABLE. A base URL pointing at the wrong service answers 404 exactly like an old controller does, so the two are told apart by whether the 404 carried the controller's own request id. An answer nobody can attribute is never read as a missing feature.

    2. A lost job is adopted, not duplicated

      Every launch is tagged alma-<modelHash>. If the record of a job goes missing, the next attempt finds the job in the queue by that name and rebuilds the record instead of submitting a second copy beside it. Nothing is ever submitted that cannot be named, and therefore cannot be stopped.

    3. Restarts have a budget, and it has to be earned back

      A model that dies while an SLA still references it is relaunched with exponential backoff and jitter, bounded by attempts and by elapsed time. Deferrals — the queue could not be read, something else is already launching it — do not spend the budget; only the model's own failures do. The budget returns after the model has stayed up, not after a single sighting of Running.

    4. Tunnels are reconciled, not assumed

      Every 30 seconds the tunnel units on the host are compared against the running models: orphans left by deleted models are torn down, two units claiming one port are separated, and a tunnel whose ssh process hung — active to systemctl, dead to a connection — is reinstalled. Gateway routes are published and withdrawn in the same tick, so a tunnel is never open onto a port nothing points at.

    5. Anything needing a person raises an alert

      A suspended model, an unresolvable port conflict, a scheduler blocked by a stale controller: each is recorded, deduplicated and pushed to Slack, Discord, Teams or e-mail, addressed to whoever holds the admin role at that moment. A dead webhook cannot fail the tick that raised the alert.

    06 — access

    Getting an account

    ALMA runs on academic infrastructure at the Universidad de Sevilla. Access is granted per person, with an SLA sized to what you are actually doing.

    1. Write to us

      Say who you are, your group or degree, the model or task you have in mind, and roughly how much traffic you expect.

    2. An administrator assigns an SLA

      You get a user, a role and an SLA template carrying the endpoints and rate limits that fit the request.

    3. Collect your key in the portal

      Sign in, copy the API key, copy the curl printed next to any endpoint, export the list as JSON, and start.

    Request access by e-mail

    alma.org.us.es@gmail.com

    Already have an account — open the portal