Diagnostic Endpoint Addresses

You can access detailed system metrics, JVM information, thread states, and performance data of Worker and Cache modules. You can use diagnostic endpoints via pod-to-pod communication from within Kubernetes or through API calls from outside, query all pods collectively, and analyze system performance. You can access critical information for troubleshooting, performance analysis, and capacity planning.

Overview

Diagnostic endpoints provide the ability to collect detailed information about the operational status of Worker and Cache modules. With these endpoints, you can access JVM metrics, thread information, connection states, environment variables, and more.

Use Cases

Performance Analysis

You can monitor JVM memory usage, thread counts, and system resources in real-time.

Troubleshooting

You can detect memory leaks and deadlock issues by taking thread dumps and heap dumps.

Capacity Planning

You can make scaling decisions by monitoring system resource usage.

Centralized Monitoring

You can perform cluster-wide status analysis by querying all pods collectively.

Access Methods

You can access diagnostic endpoints in two different ways:

Server Management Screen

You can access diagnostic information of all pods through the visual interface from the System → Server Management screen in API Manager. This method is suitable for manual control and quick analysis.

API Access

You can access endpoints directly with HTTP requests. This method is ideal for automation, monitoring systems, and script-based controls.

Workflow

The operational logic of diagnostic endpoints is shown in the diagram below:

Broadcast Mechanism

When internal=false (or when the parameter is not provided), the requested pod forwards requests to all other pods in the cluster and collects the results:

Authorization

All diagnostic endpoints require authorization. Environment ID is used in requests:

1. Add Authorization Header

   You must send the active environment ID in the Authorization header in the request.

   Authorization: <ENVIRONMENT_ID>
   
2. Validation

   Worker and Cache pods compare the incoming token with their own environment IDs.

3. Access

   If the token matches, the request is processed; otherwise, 401 Unauthorized is returned.

You can obtain the Environment ID from Environment settings in API Manager or from your system administrator. Requests made with incorrect or missing authorization information will be rejected.

Worker Diagnostic Endpoints

Available endpoints for Worker module:

JVM Metrics

JVM memory usage, heap/non-heap information, garbage collection statistics.

From Within Kubernetes

# Single pod
kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/jvm?internal=true"

# All pods (broadcast)
kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/jvm"

From Outside Kubernetes

# Single pod
curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/jvm?internal=true"

# All pods (broadcast)
curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/jvm"

Thread Information

Active thread count, thread states, thread pool usage.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/threads?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/threads"

Thread Dump

Detailed stack trace information of all threads. Used for deadlock detection.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/threaddump?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/threaddump"

Heap Dump

Binary dump of JVM heap memory. Used for memory analysis.

The heap dump endpoint does not support broadcast and returns a binary file (.hprof) as output. Therefore, the internal parameter is not used, and the result should be saved to a file.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/heapdump" \
  --output heapdump-$(date +%Y%m%d-%H%M%S).hprof

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/heapdump" \
  --output heapdump-$(date +%Y%m%d-%H%M%S).hprof

Connection Information

Active HTTP connections, connection pool states, backend connections.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/connections?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/connections"

Environment Variables

System and JVM environment variables, system properties.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/env?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/env"

Health Status

Pod health status, uptime, basic system information.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/health?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/health"

All Metrics

Collects all metrics except heap dump in a single call.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://worker-http-service.prod.svc.cluster.local:8091/apinizer/diagnostics/all?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<WORKER_ACCESS_URL>/apinizer/diagnostics/all"

Cache Diagnostic Endpoints

Available endpoints for Cache module:

JVM Metrics

From Within Kubernetes

# Single pod
kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/jvm?internal=true"

# All pods (broadcast)
kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/jvm"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/jvm"

Thread Information

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/threads?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/threads"

Thread Dump

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/threaddump?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/threaddump"

Heap Dump

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/heapdump" \
  --output cache-heapdump-$(date +%Y%m%d-%H%M%S).hprof

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/heapdump" \
  --output cache-heapdump-$(date +%Y%m%d-%H%M%S).hprof

Environment Variables

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/env?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/env"

Health Status

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/health?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/health"

Hazelcast Metrics

Hazelcast cluster information, cache statistics, distributed map metrics.

This endpoint is only available in the Cache module and shows the detailed status of the Hazelcast cluster.

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/hazelcast?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/hazelcast"

All Metrics

From Within Kubernetes

kubectl exec -it <any_pod_name> -n <namespace> -- curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://cache-http-service.prod.svc.cluster.local:8090/apinizer/diagnostics/all?internal=true"

From Outside Kubernetes

curl -X GET \
  -H "Authorization: <ENVIRONMENT_ID>" \
  "http://<CACHE_ACCESS_URL>/apinizer/diagnostics/all"

Endpoint Comparison Table

Endpoint	Worker	Cache	Broadcast	Description
/jvm	✅	✅	✅	JVM memory and GC metrics
/threads	✅	✅	✅	Thread information and states
/threaddump	✅	✅	✅	Detailed thread stack trace
/heapdump	✅	✅	❌	Binary heap dump file
/connections	✅	❌	✅	HTTP connection pool information
/env	✅	✅	✅	Environment variables
/health	✅	✅	✅	Health status and uptime
/hazelcast	❌	✅	✅	Hazelcast cluster metrics
/all	✅	✅	✅	All metrics (except heapdump)

Use Case Scenarios

Memory Leak Detection

1. Regularly monitor memory usage with the /jvm endpoint
2. Detect continuous increase in heap memory
3. Take a heap dump with /heapdump
4. Analyze with Eclipse MAT or VisualVM

# Monitor memory usage
curl -X GET -H "Authorization: <ENV_ID>" \
  "http://<WORKER_URL>/apinizer/diagnostics/jvm" | jq '.pods[].jvm.memory'

# Take heap dump
curl -X GET -H "Authorization: <ENV_ID>" \
  "http://<WORKER_URL>/apinizer/diagnostics/heapdump" \
  --output analysis.hprof

Deadlock Analysis

You can detect deadlock situations by taking thread dumps:

# Take thread dump
curl -X GET -H "Authorization: <ENV_ID>" \
  "http://<WORKER_URL>/apinizer/diagnostics/threaddump" \
  | jq '.pods[].threadDump' > threaddump.txt

# Search for "deadlock" in the file
grep -i "deadlock" threaddump.txt

Cluster-Wide Performance Analysis

You can check the status of all pods in a single call:

# Get all metrics of all pods
curl -X GET -H "Authorization: <ENV_ID>" \
  "http://<WORKER_URL>/apinizer/diagnostics/all" \
  | jq '.' > cluster-diagnostics.json

# List memory usage of each pod
cat cluster-diagnostics.json | jq '.pods[] | {pod: .podName, heapUsed: .jvm.memory.heap.used, heapMax: .jvm.memory.heap.max}'

Automated Monitoring Integration

You can integrate with Prometheus, Grafana, or custom monitoring systems:

#!/bin/bash
# monitoring-script.sh
while true; do
  RESPONSE=$(curl -s -X GET -H "Authorization: $ENV_ID" \
    "http://$WORKER_URL/apinizer/diagnostics/jvm")
  
  echo "$RESPONSE" | jq -r '.pods[] | "\(.podName) - Heap: \(.jvm.memory.heap.used)/\(.jvm.memory.heap.max)"'
  
  sleep 60
done

Connection Pool Monitoring

You can check connection pool states in Worker pods:

# Connection pool metrics
curl -X GET -H "Authorization: <ENV_ID>" \
  "http://<WORKER_URL>/apinizer/diagnostics/connections" \
  | jq '.pods[] | {pod: .podName, activeConnections: .connections.active, idleConnections: .connections.idle}'

Best Practices

Regular Monitoring

Perform proactive troubleshooting by periodically checking /health and /jvm endpoints.

Broadcast Usage

Use broadcast mode (without internal parameter) to check all pods in production environment.

Heap Dump Size

Heap dumps can create large files. Ensure sufficient disk space is available.

Authorization Security

Store Environment ID securely and do not display it in logs.

Performance Note: Heap dump and thread dump operations can create load on the pod. It is recommended to perform these operations during low-traffic hours in production environments.

Related Resources

For more information, you can check the following pages:

• Healthcheck and Version Addresses
• Prometheus and Grafana Integration
• Gateway Metrics Integration
• Cache Metrics Integration