Profiling a running deployment¶

Capture a CPU profile, a jemalloc heap profile or tokio runtime counters from a running ingester, compactor or query server. A released image cannot do this: the routes are compiled out of every published build, and the build that has them stays inert until you set an environment variable. Plan for a separate image, a restart of the role you want to profile, and a rollback afterwards.

The routes ride the metrics port (9100 ingester, 9101 compactor, 9105 query server) because that is the one HTTP listener every role shares. They are never mounted on the ingest or query API ports.

Before you start¶

• Either a Siglake checkout at the commit you want to profile and Docker, or access to the registry the profiling-image workflow pushes to.
• Permission to change the image and environment of one role and restart it.
• A way to reach the metrics port of the pod you profile, such as kubectl port-forward.
• Read Network policies first. The metrics port checks no token, and an armed profiler hands a CPU profile's stack traces and a heap profile's allocation sites to anything that can reach the pod.

Build the PROFILING=1 image¶

Run this from the root of a Siglake checkout:

docker build -f deploy/Dockerfile \
  --build-arg PROFILING=1 \
  --build-arg BUILD_CACHE_ID=profiling \
  -t siglake:prof-local .

PROFILING=1 compiles siglake-cli and siglake-query-server with the profiling feature, which is off by default, and rebuilds jemalloc with --enable-prof. It also adds --cfg tokio_unstable for the full tokio counter set, keeps frame pointers, and skips the strip step so profiles resolve to file and line. That debug information is most of why the image is about 1.95 GB, against 115 to 123 MB per stripped binary in the released build.

The profiling-image GitHub Actions workflow builds the same image on workflow_dispatch and pushes it to the project's own registry as siglake:prof-<short sha>. It never runs on a merge. A prof- tag is not a release artifact and no public registry carries one, so treat it as a throwaway for one investigation.

Arm the routes with SIGLAKE_PPROF_ENABLED=1¶

Before you change the Helm values, record query.image.repository, query.image.tag and the entries in query.extraEnv. To profile a query server, set both fields in the query-specific image override and add the variable:

query:
  image:
    repository: <registry>/siglake
    tag: prof-<sha>
  extraEnv:
    - name: SIGLAKE_PPROF_ENABLED
      value: "1"

This override restarts query pods on the profiling image. Ingester, compactor and schema-migration hook images keep their global image values. Set both query image fields because an empty field inherits its global counterpart.

The process reads SIGLAKE_PPROF_ENABLED once, while it builds the metrics router, so the setting takes effect on the restart and not before. 1 and true arm the routes; any other value, a misspelling included, leaves them unmounted.

The chart has no image override for the ingester or compactor. To profile one of those roles, put the variable in ingester.extraEnv or compactor.extraEnv and set the global image.repository and image.tag. That global change also selects the profiling image for the other role and the schema-migration hook. A query pod inherits it unless query.image already overrides both fields.

Start the process with heap sampling on¶

The heap route needs one more variable, and the process must already have it at startup:

query:
  extraEnv:
    - name: SIGLAKE_PPROF_ENABLED
      value: "1"
    - name: _RJEM_MALLOC_CONF
      value: "prof:true,prof_active:true"

The _ prefix is required: tikv-jemalloc-sys builds jemalloc with a prefixed symbol namespace and ignores a plain MALLOC_CONF. jemalloc reads the setting once at startup and a dump reports only allocations sampled while prof.active was true, so you cannot turn sampling on later and dump immediately. Get this wrong and /debug/pprof/heap answers 412 while the CPU and runtime routes look healthy. The CPU and runtime routes do not need it.

Capture a profile¶

In one terminal, open a path to the metrics port. Port forwarding gives you access; it restricts nobody else.

kubectl port-forward pod/<pod> 9105:9105

In a second terminal, call one route at a time. Each file stays under a temporary name until curl receives a complete, successful response. A failed capture removes its temporary file and leaves an earlier profile untouched.

set -eu

CPU_TMP=''
HEAP_TMP=''
cleanup() {
  [ -z "$CPU_TMP" ] || rm -f "$CPU_TMP"
  [ -z "$HEAP_TMP" ] || rm -f "$HEAP_TMP"
}
trap cleanup EXIT
trap 'exit 1' HUP INT TERM

CPU_TMP=$(mktemp ./cpu.pb.gz.XXXXXX)
curl -fSs --connect-timeout 5 --max-time 45 -o "$CPU_TMP" \
  "http://localhost:9105/debug/pprof/profile?seconds=30"
mv "$CPU_TMP" cpu.pb.gz
CPU_TMP=''

HEAP_TMP=$(mktemp ./heap.jeprof.XXXXXX)
curl -fSs --connect-timeout 5 --max-time 60 -o "$HEAP_TMP" \
  http://localhost:9105/debug/pprof/heap
mv "$HEAP_TMP" heap.jeprof
HEAP_TMP=''

curl -fSs --connect-timeout 5 --max-time 15 \
  "http://localhost:9105/debug/pprof/runtime?seconds=5"

trap - EXIT HUP INT TERM

The CPU and runtime routes hold the request open for the whole window, so set a client timeout above seconds. The CPU profile carries its own symbols and opens with go tool pprof cpu.pb.gz. The heap profile does not: jeprof needs the same binary, so record the image digest you captured from.

Read the response codes¶

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9105/debug/pprof/runtime

A 404 on every route is the check to run before a long capture: it says this build or this process cannot profile at all, which is different from a capture that returns nothing interesting.

Roll back when you are done¶

For a query server, restore the recorded query.image.repository and query.image.tag. If the query image inherited both global values before this procedure, remove the override or set both fields to empty strings. Remove the SIGLAKE_PPROF_ENABLED and _RJEM_MALLOC_CONF entries from query.extraEnv, but keep every entry that was there before profiling.

For an ingester or compactor, restore the recorded global image values and remove the two profiling entries from that role's extraEnv. Leaving the profiling image in place costs disk and pull time on every scheduling decision. Leaving the routes armed exposes an unauthenticated profiler on the metrics port. Neither opt-in is scheduled to become a default.