Operating LMC Exporter

Overview

LMC Exporter exposes LSEG/LMC shared-memory managed object data as Prometheus-compatible metrics. It can emit periodic stdout snapshots or serve metrics through an HTTP scrape endpoint, which defaults to /metrics when HTTP mode is enabled.

This article explains how to operate the exporter after it is built and configured. It focuses on:

• monitoring exporter and target health
• configuring HTTP scraping
• using scoped metric sets for operational views
• tuning render and worker behavior
• scaling from one target to multiple targets
• troubleshooting scrape readiness and routing errors

Prerequisites

Before operating lmc_exporter, ensure that you have:

• A Linux environment supported by the repository build targets: OL7_64_GCC482, RHEL8_64_GCC831, or RHEL9_64_GCC1141.
• A built src/lmc_exporter executable.
• An LMC install tree with the required headers and libraries.
• A valid license file. Startup validates the license before creating the LMC shared-memory pool.
• One or more shared-memory server_key values for the LMC applications you want to scrape.
• A Prometheus-compatible scraper, or direct access to the exporter HTTP endpoint for manual checks with curl.

To confirm the binary version:

./src/lmc_exporter --version

How It Works

At runtime, lmc_exporter:

1. Loads built-in defaults, an optional TOML config file, and then any CLI overrides.
2. Builds one or more scrape target definitions.
3. Creates the LMC shared-memory server pool.
4. Adds configured server keys to the pool.
5. Creates a target registry.
6. Creates a MetricBuilder and LMCExporterProbe for each target.
7. Starts either:
   • the HTTP scrape server, or
   • the periodic stdout metrics printer.
8. Discovers visible roots for each configured server_key.
9. Selects the best matching root, optionally using an instance hint.
10. Recursively binds child objects and variables under the selected root.
11. Renders Prometheus text once the tracked subtree is ready.

Before the first full sync completes, HTTP scrapes return 503 Service Unavailable with a readiness message. After initial sync, newly added child objects or variables do not take the target back to startup readiness. The exporter continues serving the last rendered snapshot and adds the new metrics after those paths sync.

Configuration

Core operational settings

The following TOML keys are directly relevant to operations.

Example operational TOML

context_id = "shmApp"
pool_name = "pool"

metrics_http_enabled = true
metrics_bind_address = "0.0.0.0"
metrics_port = 9464
metrics_path = "/metrics"

metrics_interval_seconds = 10
render_min_interval_ms = 250
target_workers = 0
worker_execution_mode = "logical"
worker_assignment_mode = "hash"
worker_rebalance_enabled = false
debug = false

license_file = "./scratch/example.lic"

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

[[targets]]
app = "ads"
server_key = "82"

[[targets]]
app = "ads"
server_key = "456"

[[targets]]
app = "adh"
server_key = "999"

Usage

Start HTTP scrape mode

./src/lmc_exporter \
  --server-key 82 \
  --app ads \
  --enable-metrics-http \
  --metrics-port 9464

./src/lmc_exporter --config ./lmcExporterConfig.toml

Scrape a single target

If only one target is configured, the base metrics path can resolve that target automatically:

curl "http://127.0.0.1:9464/metrics"

When multiple targets are configured, include routing parameters.

curl "http://127.0.0.1:9464/metrics?server_key=82"

You can also route by application and resolved instance:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads"

If more than one configured target shares the same app and instance, include server_key as well:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82"

Scrape a scoped metric set

Metric scopes expose named subsets of the rendered Prometheus output. This is useful for health checks, dashboards, or reducing scrape payload size for specific views.

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

Scrape the scope as a path segment:

curl "http://127.0.0.1:9464/metrics/process_health?server_key=82"

Or use the scope query parameter:

curl "http://127.0.0.1:9464/metrics?scope=process_health&server_key=82"

Monitoring LMC Exporter Health

Check scrape readiness

Use the base or scoped metrics endpoint and inspect the HTTP status code.

curl -i "http://127.0.0.1:9464/metrics?server_key=82"

Expected operational states:

A 503 response includes a message similar to:

scrape not ready: objects 10/12 synced, variables 120/140 synced, waiting on variable sync

Monitor internal exporter metrics

Each rendered target includes internal exporter metrics alongside the application metrics. Use these metrics to monitor readiness, sync state, queue pressure, worker utilization, refresh behavior, and cache effectiveness.

Recommended alert signals

Use the internal metrics to build operational alerts around target availability and exporter pressure.

lmc_exporter_internal_target_ready == 0

lmc_exporter_internal_target_initial_sync_complete == 0

lmc_exporter_internal_target_pending_sync_items > 0

rate(lmc_exporter_worker_rebalance_total[5m]) > 0

lmc_exporter_worker_dequeue_latency_p95_ms > 1000

Tune thresholds for your environment.

Operational Best Practices

Prefer TOML for production configuration

Use --config with a TOML file for production runs. TOML supports runtime defaults, metric scopes, and preloaded target tables in one file.

./src/lmc_exporter --config ./lmcExporterConfig.toml

CLI flags override TOML values. Use CLI overrides for temporary changes, not for long-lived production configuration.

Use server_key routing for multi-target deployments

In multi-target mode, server_key is the simplest and least ambiguous routing parameter.

curl "http://127.0.0.1:9464/metrics?server_key=82"

Use app and instance routing only when the resolved instance is known and unique.

Keep health scopes small

For operational dashboards and alerting, define a small process_health scope instead of scraping and evaluating the full application payload every time.

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

Use filters to limit subscription scope

For target definitions, use classes, seed_path_groups, or seed_paths to limit the tracked subtree. These filters limit both the subscribed subtree and the metrics exposed for that target.

[[targets]]
app = "ads"
server_key = "82"
classes = ["ManagedProcess.SinkDist", "ConsumerDataStream"]
seed_path_groups = ["core", "users"]

Use --debug only during investigation

Debug mode emits verbose sync, update, and lookup diagnostics. Enable it while investigating readiness, target selection, or variable update behavior, then disable it to reduce log volume.

./src/lmc_exporter --config ./lmcExporterConfig.toml --debug

Performance Tuning

Tune render coalescing

render_min_interval_ms controls the minimum spacing between demand renders for a single target.

Default behavior:

render_min_interval_ms = 250

Operational effects:

• Higher values reduce render CPU during bursts because multiple updates are coalesced into one deferred render.
• Lower values reduce the maximum delay between LMC updates and rendered scrape output.
• 0 disables spacing and restores render-per-update behavior.

Use a higher value when:

• update churn is high
• worker dequeue latency is increasing
• scrape output does not need sub-second freshness

Use a lower value when:

• freshness is more important than render CPU
• the target count is small
• internal latency metrics show the exporter has enough headroom

Use scoped metric sets to reduce scrape processing

Scoped metric sets are filtered from rendered Prometheus output and cached by scope. Repeated scoped scrapes can be served from the scope cache when the underlying snapshot and scope configuration have not changed.

Use scoped metrics for:

• health checks
• focused dashboards
• high-frequency operational scrapes

Watch cache effectiveness

Use these metrics to understand whether scrapes are using cached output or causing extra work:

rate(lmc_exporter_internal_scrape_render_cache_hits_total[5m])

rate(lmc_exporter_internal_scrape_render_cache_misses_total[5m])

rate(lmc_exporter_internal_scrape_scope_cache_hits_total[5m])

rate(lmc_exporter_internal_scrape_scope_cache_misses_total[5m])

Frequent cache misses during stable periods may indicate that target data is changing often, scopes are changing, or scrapes are reaching targets immediately after render updates.

Watch worker pressure

Monitor worker metrics when increasing target count or widening metric scope.

lmc_exporter_worker_busy_idle_ratio

lmc_exporter_worker_dequeue_latency_p95_ms

lmc_exporter_target_refresh_duration_p95_ms

Sustained increases in queue latency or refresh duration usually indicate that you should reduce target scope, increase worker capacity, or review scrape frequency.

Scaling Considerations

Scale target definitions deliberately

A single TOML file can preload multiple targets:

metrics_http_enabled = true
metrics_port = 9464

[[targets]]
app = "ads"
server_key = "82"
seed_path_groups = ["core", "admin", "sink_ripc", "users"]

[[targets]]
app = "ads"
server_key = "456"
seed_path_groups = ["core", "source_ssl", "service_generator"]

[[targets]]
app = "adh"
server_key = "999"
seed_path_groups = [
  "consumer_mqos_service_service",
  "mqos_service_attributes",
  "mqos_service_datastreams"
]

Each [[targets]] entry must define app and server_key. instance is optional when server_key is provided and can be resolved from the selected root at runtime.

Configure worker count

Use automatic worker sizing first:

target_workers = 0

Automatic sizing uses the configured target count, with a minimum of 1 and a cap of 64.

For explicit sizing:

target_workers = 4

Non-zero worker values must be between 1 and 64.

Choose an assignment mode

Use hash for stable target placement:

worker_assignment_mode = "hash"

Use least_loaded when adding targets and you want assignment based on the smallest current estimated load:

worker_assignment_mode = "least_loaded"

Enable rebalancing only when needed

Runtime worker rebalancing is disabled by default.

worker_rebalance_enabled = false

Enable it when worker metrics show persistent imbalance across workers:

worker_rebalance_enabled = true

When enabled, the exporter can move targets between workers and reports rebalance activity through lmc_exporter_worker_rebalance_total and lmc_exporter_worker_rebalance_last_duration_ms.

Examples

Minimal single-target HTTP scrape

app = "ads"
server_keys = ["82"]

metrics_http_enabled = true
metrics_bind_address = "0.0.0.0"
metrics_port = 9464
metrics_path = "/metrics"

license_file = "./scratch/example.lic"

Run:

./src/lmc_exporter --config ./lmcExporterConfig.toml

Scrape:

curl "http://127.0.0.1:9464/metrics"

Multi-target health scope

metrics_http_enabled = true
metrics_port = 9464
render_min_interval_ms = 250
target_workers = 0

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

[[targets]]
app = "ads"
server_key = "82"
seed_path_groups = ["core"]

[[targets]]
app = "ads"
server_key = "456"
seed_path_groups = ["core"]

Scrape each target’s health subset:

curl "http://127.0.0.1:9464/metrics/process_health?server_key=82"
curl "http://127.0.0.1:9464/metrics/process_health?server_key=456"

Legacy target file

Use the legacy format only when you need a minimal target-only file.

target app=ads server_key=82
target app=ads server_key=456

Run with:

./src/lmc_exporter --targets-config ./targets.txt --enable-metrics-http --metrics-port 9464

Troubleshooting

HTTP endpoint returns 503

Cause: the target is not ready for scrape. During startup, all tracked objects and variables must sync before the first scrape can succeed.

Resolution:

1. Inspect the response body for the readiness message.
2. Confirm the configured server_key is correct.
3. Confirm the selected root exists and matches the optional instance hint.
4. Run with --debug to capture sync, lookup, and update diagnostics.
5. Monitor lmc_exporter_internal_target_initial_sync_complete and lmc_exporter_internal_target_pending_sync_items.

HTTP endpoint returns 400

Cause: target routing is missing or ambiguous.

Resolution:

• If multiple targets are configured, include server_key.
• If omitting server_key, provide both app and instance.
• If multiple targets share the same app and instance, include server_key.

Example:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82"

HTTP endpoint returns 404

Cause: the requested target or scope does not exist, or the selected scope does not match any rendered metrics.

Resolution:

1. Verify the server_key, app, and instance values.
2. Verify the [[metric_scopes]] name.
3. Confirm that each metric name in the scope matches the rendered Prometheus metric names.
4. For wildcard scopes, verify that the wildcard pattern matches the rendered metric names.

No metrics are emitted

Cause: the app mapping or filters do not select any metrics.

Resolution:

1. Confirm the target app is correct.
2. Confirm metric mappings exist for that application.
3. Review configured classes, seed_path_groups, and seed_paths.
4. Temporarily reduce filtering to confirm whether the target emits metrics without the restrictive scope.

Multi-target scrape cannot disambiguate the target

Cause: more than one target matches the supplied route.

Resolution:

Use server_key in the scrape query.

curl "http://127.0.0.1:9464/metrics?server_key=82"

Startup fails before LMC discovery

Cause: license validation fails.

Resolution:

1. Confirm license_file points to the customer license JSON file.
2. Confirm the license has not expired beyond the 36-hour grace period.
3. Check startup stderr logs for license validation messages.
4. Re-run after updating the configured license file.

Worker latency or refresh duration is high

Cause: target refresh work is exceeding worker capacity or render output is too expensive.

Resolution:

1. Check lmc_exporter_worker_dequeue_latency_p95_ms.
2. Check lmc_exporter_target_refresh_duration_p95_ms.
3. Increase render_min_interval_ms to coalesce more updates.
4. Narrow classes, seed_path_groups, or seed_paths.
5. Use scoped metric sets for high-frequency operational scrapes.
6. Increase target_workers or enable worker_rebalance_enabled after confirming sustained pressure.