Deployment

Everything an operator needs to take TurboOCR from “I want to run this in prod” to “it’s running, monitored, and won’t fall over.” This page covers the supported Docker tags, the required GPU and driver baseline, the TensorRT cache volume, language switching, MPS-based scaling, monitoring endpoints, and a short production checklist.

Requirements

• OS — Linux. The container is built on the official NVIDIA TensorRT base image and is verified on Ubuntu 24.04.
• GPU — NVIDIA Turing or newer (RTX 20-series, GTX 16-series, or anything more recent).
• Driver — NVIDIA driver 595 or newer, plus the NVIDIA Container Toolkit so --gpus all is honored by Docker.
• VRAM — roughly ~4 GB for a text-only deployment and ~8 GB for the full pipeline (layout + tables + formulas). Each additional PIPELINE_POOL_SIZE replica adds about another full set.
• CPU fallback — a CPU-only image exists for development on machines without a GPU, but throughput is much lower (single-digit img/s) and it should not be used in production.

Docker images

Pre-built images are published to GitHub Container Registry under ghcr.io/aiptimizer/turboocr. Two flavors are produced from docker/Dockerfile.gpu and docker/Dockerfile.cpu in the source repo.

Tag	Purpose	Notes
:latest	GPU image, tracks the most recent release	Drogon HTTP on 8000 (via nginx), gRPC on 50051. Bakes every recognizer, PP-DocLayoutV3, and the table/formula weights. Fine for demos; pin in prod.
:v3.1.0	Pinned GPU image	The current release — use this in production.
turboocr-cpu:v3.1.0	CPU fallback	ONNX Runtime instead of TensorRT (own image: ghcr.io/aiptimizer/turboocr-cpu). Single-digit img/s — dev/CI only.

Caution

Pin a version in production. Use an explicit tag like :v3.1.0 rather than :latest. Pinning makes rollbacks trivial and stops a surprise upgrade from rebuilding TensorRT engines mid-incident.

Server binary names

Inside the image the server binaries were renamed in v3: GPU is turboocr-server (was paddle_highspeed_cpp) and CPU is turboocr-cpu-server (was paddle_cpu_server). Docker users are unaffected; this only matters for native builds and --print-config.

GPU image

The default image runs the full TensorRT pipeline. The first start compiles engines from ONNX (about 90 seconds on an RTX 5090, up to ~1 hour on older GPUs); the named trt-cache volume persists those engines so subsequent restarts are instant.

docker volume create trt-cache

docker run --gpus all -p 8000:8000 -p 50051:50051 \
  -v trt-cache:/home/ocr/.cache/turbo-ocr \
  ghcr.io/aiptimizer/turboocr:latest

HTTP listens on 8000 (nginx in front of Drogon, for connection buffering) and gRPC on 50051. Both protocols share the same GPU pipeline pool — you can run them simultaneously from the same container.

Speed up the first build with TRT_OPT_LEVEL

The engine build defaults to the highest optimization level (TRT_OPT_LEVEL=5, fastest runtime). On a slow GPU, set TRT_OPT_LEVEL=3 to cut the one-time build 3–5× for a small (<5%) runtime regression. The level is part of the engine cache key, so changing it triggers one rebuild.

CPU image

The CPU image is a drop-in fallback for hosts without an NVIDIA GPU. It uses ONNX Runtime instead of TensorRT, runs on plain ubuntu:24.04, and exposes the same HTTP API.

docker run -p 8000:8000 \
  ghcr.io/aiptimizer/turboocr-cpu:v3.1.0

Caution

Throughput is single-digit img/s versus up to ~556 img/s on a single RTX 5090. The CPU build exists for local development, CI, and integration tests — not production traffic.

TensorRT cache volume

TensorRT engines and language bundles live at /home/ocr/.cache/turbo-ocr inside the container. Mount this path as a named volume — never a host bind-mount.

docker volume create trt-cache
docker run --gpus all -p 8000:8000 -p 50051:50051 \
  -v trt-cache:/home/ocr/.cache/turbo-ocr \
  ghcr.io/aiptimizer/turboocr:latest

Why a named volume. The image symlinks /app/models/rec into the cache directory so non-default language bundles also live there. Bind-mounting an empty host directory at this path would shadow the baked-in bundles and leave the server with nothing to load. Named volumes auto-populate from the image on first use, which is exactly what you want.

If you delete the volume. The next start will rebuild every TensorRT engine from ONNX (~90 seconds) and re-extract the baked bundles. No data loss — just a slower cold start.

Model selection

Set the OCR_MODEL environment variable at start time. All models are baked into the image at build time from pinned PaddleOCR releases; there is no runtime download.

docker run --gpus all -p 8000:8000 -p 50051:50051 \
  -v trt-cache:/home/ocr/.cache/turbo-ocr \
  -e OCR_MODEL=medium \
  ghcr.io/aiptimizer/turboocr:latest

The PP-OCRv6 tiers tiny (default) / small / medium cover Latin + Chinese + Japanese and trade accuracy for speed. Other scripts use retained PP-OCRv5 recognizers via the same variable: arabic, eslav (Cyrillic), korean, thai, greek. OCR_LANG still works as a deprecated alias (warns on use). See the Configuration page for the full breakdown.

Web GUI (Studio)

The repository ships a browser UI — drop in an image or PDF, run OCR, see the layout overlay and reading order, select recognized text, and download a searchable PDF. It runs as a second container that proxies /api/* to the OCR server, so the browser stays same-origin and the server needs no CORS.

From the source repo root:

# GPU
docker compose -f docker-compose.demo.yml up --build

# CPU (dev/CI)
docker compose -f docker-compose.demo.cpu.yml up --build

Then open http://localhost:3000. The demo compose enables TABLE_BACKEND=slanext and FORMULA_BACKEND=ppformulanet_s, so tables and formulas are available end-to-end. The OCR service is still reachable directly on http://localhost:8000 for curl.

Persistent connections

Caution

Use HTTP keep-alive. Sending one short-lived connection per request — for example, a tight loop of curl — can overwhelm the server and cause it to stall. Every standard HTTP client (requests.Session, aiohttp, httpx.Client, JDK HttpClient, Go http.Client, libcurl with a long-lived handle) reuses connections by default if you reuse the client object. Don’t construct a fresh client per request.

The bundled image runs nginx in front of Drogon to buffer slow clients, but nginx cannot rescue a client that opens a fresh TCP connection per request. See the Clients guide for working examples in Python, Java, and C++.

Monitoring

Prometheus metrics

Scrape GET /metrics for Prometheus-compatible metrics. The server exposes per-route request counters, latency histograms, VRAM usage, and pipeline-pool saturation:

turbo_ocr_requests_total{route="/ocr/raw",status="2xx"} 1042
turbo_ocr_request_duration_seconds_bucket{route="/ocr/raw",le="0.025"} 980
turbo_ocr_request_duration_seconds_sum{route="/ocr/raw"} 12.345
turbo_ocr_request_duration_seconds_count{route="/ocr/raw"} 1042
turbo_ocr_gpu_vram_used_bytes 9052815360
turbo_ocr_gpu_vram_total_bytes 33661911040
turbo_ocr_pipeline_pool_size 5
turbo_ocr_pool_exhaustions_total 0
turbo_ocr_request_bytes_total 49493243
turbo_ocr_request_body_avg_bytes 9407

Tip

Watch turbo_ocr_pool_exhaustions_total. A non-zero value means clients are queueing on the pipeline pool — either raise PIPELINE_POOL_SIZE or scale out replicas. Latency histograms are bucketed per route, so each endpoint can be SLO’d independently.

Health probes

Endpoint	Use
GET /health	Basic liveness — returns "ok".
GET /health/live	Kubernetes liveness probe. Returns 200 once the process is up.
GET /health/ready	Readiness probe. Returns 200 only after the GPU pipeline passes a smoke test — gate traffic on this.

Building from source

For local builds, custom CUDA targets, or air-gapped deployments, see the build instructions in the GitHub repository. The Dockerfiles in docker/ are the canonical reference; the README’s “Building from Source” section lists every dependency and the CMake invocation for both GPU and CPU targets.

Configuration Every environment variable: language bundles, pipeline pool size, PDF parallelism, logging.

Clients Production-ready Python, Java, and C++ recipes — persistent connections, fan-out, gRPC.