> ## Documentation Index
> Fetch the complete documentation index at: https://docs.apinizer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Analytics Engine

> Analytics Engine (Analytics Engine), the component in the Apinizer platform that collects, analyzes, and visualizes API usage data. It processes log records from API traffic to extract analytics data, calculates performance metrics, and provides usage statistics.

## Analytics Engine Concept

<CardGroup cols={2}>
  <Card title="Data Collection" icon="database">
    Data collection from API traffic
  </Card>

  <Card title="Analysis" icon="chart-bar">
    Data analysis and metric calculation
  </Card>

  <Card title="Visualization" icon="chart-line">
    Dashboards and reports
  </Card>

  <Card title="Real-Time" icon="clock">
    Real-time metrics
  </Card>
</CardGroup>

## Analytics Engine Features

### Data Collection

API traffic and all logging are done **asynchronously**. This way, API Gateway's performance is not affected and logging operations are performed in the background.

<AccordionGroup>
  <Accordion title="Request Logs">
    * Request URL
    * HTTP method
    * Request headers
    * Request body (optional)
    * Timestamp
  </Accordion>

  <Accordion title="Response Logs">
    * Response status code
    * Response headers
    * Response body (optional)
    * Response time
  </Accordion>

  <Accordion title="Error Logs">
    * Error type
    * Error message
    * Error time
  </Accordion>
</AccordionGroup>

<Info>
  All logging operations are performed asynchronously. This way, API Gateway's response time is not affected and high performance is maintained.
</Info>

### Metrics

<CardGroup cols={2}>
  <Card title="Performance Metrics" icon="gauge">
    * Response time (average, min, max)
    * Throughput (requests/second)
    * Latency
    * Error rate
  </Card>

  <Card title="Usage Metrics" icon="users">
    * API call count
    * Unique user count
    * Endpoint usage
    * Method-based usage
  </Card>

  <Card title="Error Metrics" icon="triangle-exclamation">
    * Error count
    * Error rate
    * Error type distribution
    * Error trend
  </Card>

  <Card title="Traffic Metrics" icon="route">
    * Total traffic
    * Traffic trend
    * Peak hours
    * Geographic distribution
  </Card>
</CardGroup>

### Visualization

<AccordionGroup>
  <Accordion title="Dashboard">
    * Real-time metrics
    * Charts and graphs
    * Customizable widgets
    * Filtering and time range selection
  </Accordion>

  <Accordion title="Reports">
    * Ready reports
    * Custom report creation
    * Report scheduling
    * Report export (PDF, CSV, Excel)
  </Accordion>

  <Accordion title="Alerting">
    * Metric-based alarms
    * Threshold definition
    * Notification channels (Email, SMS, Webhook)
  </Accordion>
</AccordionGroup>

## Analytics Engine Components

<CardGroup cols={3}>
  <Card title="Log Collector" icon="file-lines">
    Component that collects log records from API Gateways

    * **Log Aggregation**: Collecting log records
    * **Log Parsing**: Parsing log records
    * **Log Storage**: Storing log records
  </Card>

  <Card title="Data Processor" icon="cog">
    Component that processes collected data and calculates metrics

    * **Data Transformation**: Data transformation
    * **Metric Calculation**: Metric calculation
    * **Data Aggregation**: Data aggregation
  </Card>

  <Card title="Storage Layer" icon="database">
    Layer where analytics data is stored

    * **Elasticsearch**: Log and metric data
    * **Time Series Database**: Time series data
    * **Data Retention**: Data retention policies
  </Card>
</CardGroup>

## Analytics Engine Usage Scenarios

<CardGroup cols={3}>
  <Card title="Performance Monitoring" icon="gauge">
    1. API response time monitoring
    2. Throughput analysis
    3. Latency trend analysis
    4. Bottleneck detection
  </Card>

  <Card title="Usage Analysis" icon="users">
    1. Most used APIs
    2. User-based usage
    3. Endpoint-based usage
    4. Time-based usage trend
  </Card>

  <Card title="Error Analysis" icon="triangle-exclamation">
    1. Error rate monitoring
    2. Error type analysis
    3. Error trend analysis
    4. Error resolution tracking
  </Card>
</CardGroup>

## Analytics Engine and Other Components Relationship

Analytics Engine receives log records from API Gateways **asynchronously**, processes them, and produces analytics data. Workflow:

```
API Gateway
   │
   │ Asynchronous Log Records
   ▼
Analytics Engine
   │
   │ Analytics Data
   ▼
Storage Layer (Elasticsearch)
   │
   │ Visualization
   ▼
Dashboard / Reports
```

<CardGroup cols={4}>
  <Card title="1. Asynchronous Data Collection" icon="database">
    Log records are collected asynchronously from API Gateways

    Request, Response, and Error logs are collected in the background. API Gateway's performance is not affected.
  </Card>

  <Card title="2. Data Processing" icon="cog">
    Log records are processed and metrics are calculated

    Data transformation and metric calculation are performed.
  </Card>

  <Card title="3. Data Storage" icon="hard-drive">
    Analytics data is saved to storage layer

    Log and metric data are stored in Elasticsearch.
  </Card>

  <Card title="4. Visualization" icon="chart-line">
    Dashboards and reports are created

    Users can visualize metrics.
  </Card>
</CardGroup>

<Info>
  Analytics Engine provides valuable information about API performance and usage by processing log records from API Gateways asynchronously. Thanks to asynchronous logging, API Gateway's response time is not affected and high performance is maintained.
</Info>

## Data Storage and Retention

### API Traffic Logs and ILM

API traffic logs **are stored in Elasticsearch** and **Index Lifecycle Management (ILM)** is directly supported. ILM policies can be managed through Apinizer. Additionally, logs can be sent to different targets through connectors:

<CardGroup cols={2}>
  <Card title="Elasticsearch Storage" icon="hard-drive">
    API traffic logs are stored in Elasticsearch

    * Real-time log storage
    * Search and analysis capabilities
    * Scalable storage
    * Direct viewing and management on Apinizer screen
    * Kibana visualization support
  </Card>

  <Card title="ILM Support" icon="database">
    Elasticsearch ILM policies are managed from Apinizer

    * Index lifecycle stages (hot, warm, cold, delete)
    * Automatic index rollover
    * Retention policies
  </Card>
</CardGroup>

<AccordionGroup>
  <Accordion title="Apinizer Analytics and Reporting">
    Apinizer provides special analytics and reporting features for API traffic logs stored in Elasticsearch:

    * **Analytics Dashboard**: Direct viewing on Apinizer screen
    * **Reporting**: Creating and managing custom reports
    * **Real-Time Metrics**: Live traffic and performance metrics
    * **Filtering and Querying**: Advanced filtering and querying capabilities

    <Tip>
      You can view, analyze, and create reports for API traffic logs from the Apinizer screen. All analytics and reporting operations can be managed through Apinizer.
    </Tip>
  </Accordion>

  <Accordion title="Kibana Integration">
    If desired, logs in Elasticsearch can be visualized with Kibana:

    * **Kibana Dashboard**: Creating dashboards on Kibana
    * **Visualization**: Kibana's visualization tools
    * **Advanced Analysis**: Kibana's analysis capabilities

    <Tip>
      Thanks to Kibana integration, logs in Elasticsearch can be used with Kibana's powerful visualization and analysis tools.
    </Tip>
  </Accordion>

  <Accordion title="Log Sending with Connector">
    API traffic logs can be sent to different targets through connectors:

    * **Kafka**: Sending logs to message queue
    * **Syslog**: Sending logs to syslog servers
    * **SIEM Tools**: Integration with security information and event management tools
    * **Relational Database**: Writing log records to relational databases

    <Tip>
      Thanks to connectors, API traffic logs can be sent to multiple targets in parallel. This way, logs are both stored in Elasticsearch and transferred to other systems.
    </Tip>
  </Accordion>
</AccordionGroup>

### Apinizer Application Logs

Automatic cleanup tasks can be configured for logs created by various components of the Apinizer platform:

<AccordionGroup>
  <Accordion title="Component Logs">
    * **Manager Logs**: Software logs of Manager application
    * **Gateway Logs**: Software logs of Gateway application
    * **Cache Logs**: Software logs of Cache application
  </Accordion>

  <Accordion title="Audit and Security Logs">
    * **Audit Records**: User change records
    * **ACL Audit Records**: API Proxy and group access records
    * **Token Requests**: Token request records
    * **API Manager User Token Blacklist**: Token records kept when logout is performed
  </Accordion>

  <Accordion title="Monitoring and Operation Logs">
    * **Alert Logs**: Alarm records
    * **Anomaly Detector Logs**: Anomaly detection logs
    * **Uptime Monitor Logs**: Uptime monitoring logs
    * **Task Flow Logs**: Task Flow operation logs
    * **Report Logs**: Report operation logs
    * **API Proxy Trace Logs**: Logs kept when trace feature is enabled
  </Accordion>

  <Accordion title="User and Metric Data">
    * **User Notifications**: User notifications
    * **Environment Metrics**: Environment (Namespace) metrics
  </Accordion>
</AccordionGroup>

<Tip>
  Application log cleanup tasks can be run hourly or daily. Records before the specified date are automatically deleted. For detailed information, see the [Application Log Cleanup Tasks](/en/admin/audit-compliance/application-log-cleanup-tasks) page.
</Tip>

### Unsent API Traffic Logs

When API traffic logs cannot be sent to the connector, a failover connector is used. If Apinizer configuration database is selected as the failover connector, accumulated data can be transferred to main connectors:

<Info>
  Unsent API traffic logs can be transferred manually. If an error is encountered during the operation, it is stopped halfway without data loss and can be restarted later. For detailed information, see the [Transfer Unsent API Traffic Logs](/en/analytic/transfer-unsent-api-traffic-logs) page.
</Info>

### Prometheus and Grafana Integration

Analytics Engine metrics can be integrated with Prometheus and Grafana:

<CardGroup cols={2}>
  <Card title="Prometheus Integration" icon="chart-bar">
    Apinizer metrics are exported to Prometheus

    * Gateway metrics
    * Cache metrics
    * Custom PromQL queries
  </Card>

  <Card title="Grafana Dashboard" icon="chart-line">
    Prometheus data is visualized in Grafana

    * Sample dashboards
    * Customizable graphs
    * Real-time metrics
  </Card>
</CardGroup>

<Tip>
  For detailed information about Prometheus and Grafana integration, see the [Integrating Apinizer Metrics with Prometheus and Grafana](/en/tutorials/how-to-guides/administrator-guides/prometheus-grafana-integration) page.
</Tip>

## Next Steps

<CardGroup cols={2}>
  <Card title="API Traffic" icon="route" href="/en/analytic/api-traffic">
    Viewing and analyzing API traffic logs
  </Card>

  <Card title="Prometheus and Grafana" icon="chart-bar" href="/en/tutorials/how-to-guides/administrator-guides/prometheus-grafana-integration">
    Prometheus and Grafana integration
  </Card>

  <Card title="Application Log Cleanup" icon="trash" href="/en/admin/audit-compliance/application-log-cleanup-tasks">
    Application log cleanup tasks
  </Card>

  <Card title="Data Log Layer" icon="database" href="/en/concepts/architecture#5-data-and-log-layer">
    Learn about Data Log Layer architecture
  </Card>
</CardGroup>