Skip to content

NoSublistCache option not respected by gwLeafSubs #8077

@nullboundary

Description

@nullboundary

Observed behavior

When server.Options.NoSublistCache: true is set, the gateway/leaf subscription sublist (gwLeafSubs) is still created with NewSublistWithCache() at server initialization in server.go:762. On leaf node deployments under sustained message traffic, this sublist's cache fills past the 1024-entry limit and continuously spawns reduceCacheCount goroutines to sweep it — consuming the majority of CPU despite NoSublistCache being explicitly set.

CPU profile from a leaf node running ~480 msg/s across 5 fixed subjects with NoSublistCache: true:

flat  flat%   sum%        cum   cum%
0.25s  3.34%        6.98s 93.32%  server.(*Sublist).reduceCacheCount
0.26s  3.48%        5.61s 75.00%  runtime.mapdelete_faststr
0.66s  8.82%        5.32s 71.12%  internal/runtime/maps.(*Map).Delete
1.72s 22.99%        3.86s 51.60%  internal/runtime/maps.(*table).Delete
1.85s 24.73%        1.85s 24.73%  internal/runtime/maps.probeSeq.next (inline)

The offending line:
server.go:762: gwLeafSubs: NewSublistWithCache() hardcoded, ignores NoSublistCache

This is pretty much the same as issue #8070 but for leaf nodes instead of mqtt subsystem. I disabled the mqtt subsystem to avoid issue #8070 but ran into the same problem for leaf nodes.

Expected behavior

When NoSublistCache: true is set, gwLeafSubs should be initialized with NewSublistNoCache(), consistent with the server options. No reduceCacheCount goroutines should be spawned.

Server and client version

nats-server v2.12.7

Host environment

macOS host, Linux container (ARM64). Embedded server via the Go server package (not standalone binary), configured as a leaf node with a remote connection.

Steps to reproduce

  1. Embed nats-server with NoSublistCache: true and leaf node remotes configured
  2. Publish ~480 msg/s across 4–5 fixed subjects via connected client
  3. Collect a CPU profile after several minutes (go tool pprof)
  4. Observe reduceCacheCount accumulating >90% of cumulative CPU samples despite NoSublistCache: true
  5. Confirm via -focus filter that the hot path originates from the server goroutine launcher, not account sublists or MQTT.

Suggested fix:

// server.go ~line 762
gwLeafSubs: func() *Sublist {
    if opts.NoSublistCache {
        return NewSublistNoCache()
    }
    return NewSublistWithCache()
}(),

Or a helper similar to setAccountSublist() line 1932.

Metadata

Metadata

Assignees

Labels

defectSuspected defect such as a bug or regression

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions