> ## Documentation Index
> Fetch the complete documentation index at: https://docs.apinizer.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Kafka
> Configures and manages Kafka connections. You can perform topic-based message sending and consumption operations with SSL/TLS security, store centralized topic access information for Integration Flow or Connector steps in a single location, and manage various Kafka producer/consumer properties
## Overview
Kafka Connection stores centralized topic access information for Integration Flow or Connector steps in a single location, making them reusable.
Manages various Kafka producer/consumer properties through propertiesMap to maintain standard configuration in the messaging infrastructure.
When Enable Secure is activated, it enforces SSL/TLS parameters, guaranteeing encryption of critical data traffic.
Since connection parameters are validated before Deployment through the Test Connection function, the error catching process is accelerated.
When a Kafka Message Queue connection is requested from within an Integration Flow or Connector, the system reads the configured connection parameters.
The Kafka producer/consumer pool reuses existing connections or opens new connections using values such as max.block.ms, reconnect.backoff.max.ms defined in propertiesMap.
If Enable Secure is active, mutual certificate-based Authentication is applied; optionally, SASL/SCRAM information is also read from propertiesMap.
Messages are sent to or consumed from the topic via Kafka wire protocol over TCP; serializer settings are taken from key.serializer and value.serializer fields in propertiesMap.
After the operation is completed, the connection returns to the pool; connections whose enable toggle is deactivated cannot be selected by any Flow.
In case of connection error, timeout, or authentication error, retry is performed according to retry.backoff.ms and max.block.ms values, and the result is transferred to the Apinizer message service.
Targeting uniform Kafka topics in "Send Message" or "Consume Message" steps within Integration Flow
Sharing centralized Kafka configuration in connector-based data collection processes
Sending telemetry or logs to Kafka at certain intervals in Scheduled Jobs
## Technical Features and Capabilities
Thanks to the mandatory topicName field, each Connection focuses on a specific topic and prevents incorrect topic selection in Integration Flow steps.
Desired Kafka client parameters are stored in key-value structure through propertiesMap, and string, integer, or secret data are separated thanks to MapValue types.
The debounce mechanism on the UI prevents creating a Connection with the same name and warns about conflicts early.
Ability to define separate connection parameters for each environment (Development, Test, Production).
Activating or deactivating the Connection (enable/disable toggle). In passive state, the connection cannot be used but its configuration is preserved.
SASL mechanisms are added to propertiesMap, enabling encrypted Authentication scenarios without code changes.
When Enable Secure is enabled, both KeyStore and TrustStore can be selected, and if necessary, new records are created instantly in Secret Manager.
Multiple SSLContext protocols (TLSv1.2, TLSv1.3, etc.) can be selected with MultiSelect and stored on the same Connection.
Ability to validate connection parameters before saving with the "Test Connection" button.
Exporting Connection configuration as a ZIP file. Importing to different environments (Development, Test, Production). Version control and backup capability.
Monitoring connection health, pool status, and performance metrics.
## Connection Parameters
**Description:** Connection name (must be unique)\
**Example Value:** `Production_KafkaTopic01`\
**Notes:** Should not start with space, special characters should not be used
**Description:** Environment ID where the connection will be deployed\
**Example Value:** `Development`\
**Notes:** Changing environment provides parametric management within the same Connection
**Description:** Target topic name on Kafka\
**Example Value:** `audit.events.v1`\
**Notes:** The same topic should be used in producer and consumer steps
**Description:** Key-value pairs for Kafka client (bootstrap.servers, etc.)\
**Example Value:** `bootstrap.servers=broker1:9092`\
**Notes:** At least one record must exist; MapValue types must be selected correctly
**Description:** Description about the Connection purpose\
**Default Value:** (Empty)\
**Recommended Value:** Prod audit topic publish connection
**Description:** Enables SSL/TLS usage\
**Default Value:** false\
**Recommended Value:** true (Production)
**Description:** Allowed SSLContext protocols when Enable Secure is enabled\
**Default Value:** (Empty)\
**Recommended Value:** TLSv1.2, TLSv1.3
**Description:** Keystore resource containing client certificate\
**Default Value:** (Empty)\
**Recommended Value:** ks-prod-clients
**Description:** Truststore resource verifying broker certificate\
**Default Value:** (Empty)\
**Recommended Value:** ts-shared-root
**Description:** Comes as true in connectionConfigKafka.model.ts\
**Default Value:** true\
**Recommended Value:** true
**Description:** Maximum wait time for connection establishment\
**Default:** 3000\
**Min:** 1000 | **Max:** 60000\
**Unit:** milliseconds
**Description:** Maximum wait time for request response\
**Default:** 3000\
**Min:** 1000 | **Max:** 120000\
**Unit:** milliseconds
**Description:** Maximum number of connections in Connection pool (client instance)\
**Default:** 10\
**Min:** 1 | **Max:** 200\
**Unit:** count
**Description:** retry.backoff.ms value, wait between retries\
**Default:** 3000\
**Min:** 100 | **Max:** 10000\
**Unit:** milliseconds
## Usage Scenarios
**Situation:** All services should publish messages to audit topic\
**Solution:** topicName=audit.events, bootstrap.servers=cluster-prod:9092\
**Expected Behavior:** Audit messages are collected in a single topic, audit team consumes from a single point
**Situation:** Same Connection should be used in different environments\
**Solution:** Environment=Development, Enable Secure=false\
**Expected Behavior:** Different broker URLs are managed with environment selection
**Situation:** Prod broker requires TLS\
**Solution:** Enable Secure=true, ProtocolTypes=TLSv1.3, KeyStoreId=ks-prod\
**Expected Behavior:** Certificate verification is ensured, messages are sent encrypted
**Situation:** Sudden traffic increase\
**Solution:** Pool Size=50, linger.ms=5, batch.size=32768\
**Expected Behavior:** Producer batches grow, throughput increases
**Situation:** Broker occasionally does not respond\
**Solution:** retry.backoff.ms=5000, retries=10\
**Expected Behavior:** No message loss with automatic retries
**Situation:** Message delays should be measured\
**Solution:** delivery.timeout.ms=60000, enable.idempotence=true\
**Expected Behavior:** Producer timeouts are logged, SLA reports are fed
## Connection Configuration
### Creating New Kafka
#### Configuration Steps
* Go to **Connection → Kafka** section from the left menu.
* Click the **\[+ Create]** button in the top right.
**Enable Status (Active Status):**
* Set active/passive status with toggle. New connections are active by default.
**Name - Mandatory:**
* Example: `Production_KafkaAudit`
* Enter a unique name, should not start with space.
* System automatically checks. Green checkmark: available. Red cross: existing name.
**Description:**
* Example: "Audit topic producer connection"
* Max. 1000 characters.
* Describe the purpose of the Connection.
In the action button area at the top of the page, you can use the **\[\<> Variable]** button to select dynamic values, and with global variables, you can manage connection parameters with variable-based values instead of fixed values. For detailed information, review the [Dynamic Variables](/en/concepts/core-concepts/dynamic-variables) page.
* Select environment from dropdown menu: **Development**, **Test**, or **Production**.
* Different connection parameters can be defined for each environment.
* Enter broker URLs, serializer settings, and timeout values in the propertiesMap table.
* Don't forget to select valueType for each record; use INTEGER for numeric values.
* Write the topic to connect to in the Topic Name field.
You can achieve high availability by adding multiple broker URLs.
* Enable TLS by turning on the **Enable Secure** toggle.
* Select supported SSLContext protocols in the **Protocol Types** field.
* Select KeyStore and TrustStore or create a new keystore.
Always use SSL/TLS in Production environment and select secure protocols.
* Add parameters such as max.block.ms, request.timeout.ms, retry.backoff.ms to propertiesMap.
* Determine settings such as linger.ms, batch.size, connections.max.idle.ms according to traffic volume.
* If using SASL/SCRAM, add sasl.mechanism, sasl.jaas.config keys.
* Ensure the correct TrustStore is selected for broker certificates.
* Store sensitive credential values as Secret type MapValue instead of plaintext.
Always store sensitive information as Secret type MapValue.
* Click the **\[Test Connection]** button.
* Test whether connection parameters are correct.
* **Success:** Green confirmation message
* **Failed:** Error details are shown
* Click the **\[Save and Deploy]** button in the top right.
**Checklist:**
* Unique name
* Mandatory fields filled
* Test connection successful (recommended)
**Result:**
* Connection is added to the list
* Becomes available for use in Integration Flow and Connector steps
* Becomes active according to environment
Connection created successfully! You can now use it in Integration Flow and Connector steps.
### Deleting Connection
Select **Delete** from the **⋮** menu at the end of the row or click the **\[Delete]** button on the connection detail page
**Check Before Deleting:** It may be used in Integration Flow or Connector steps. If necessary, assign an alternative connection. Back up with **Export** before deleting
**Alternative: Deactivation**
* Use the **Disable** option instead of deleting.
* Connection becomes passive but is not deleted.
* Can be reactivated when needed.
### Exporting/Importing Connection
In this step, users can export existing connections for backup, transfer to different environments, or sharing purposes, or import a previously exported connection again. This process is used to maintain data integrity in version management, transitions between test and production environments, or inter-team sharing processes.
Select **⋮ → Export** from the action menu. ZIP file is automatically downloaded.
Click the **\[Export]** button on the connection detail page. ZIP file is downloaded.
Format: `Date-connection-ConnectionName-export.zip`\
Example: `13 Nov 2025-connection-Production_Kafka-export.zip`
* Connection JSON file
* Metadata information
* Dependency information (e.g., certificates, key store)
* Backup
* Transfer between environments (Test → Prod)
* Versioning
* Team or project-based sharing
* Click the **\[Import Kafka]** button on the main list.
* Select the downloaded ZIP file.
* System checks: Is format valid? Is there a name conflict? Are dependencies present?
* Then click the **\[Import]** button.
**Scenario 1: Name Conflict** → Overwrite the old connection or create with a new name.
**Scenario 2: Missing Dependencies** → Create missing certificates or key stores first or exclude them during import.
### Usage Areas of Connection
**Steps:**
1. Create the Connection.
2. Validate the connection with **Test Connection**.
3. Save and activate with **Save and Deploy**.
4. Ensure the Connection is in **Enabled** status
Connection is selected in steps that require communication with external systems such as message queue (queue), topic, email, FTP/SFTP, LDAP, or similar. Example: Steps such as "Send Message", "Consume Message", "Upload File", "Read Directory". Connection selection is made from the **Connection** field in the configuration of these steps
In scheduled tasks (e.g., sending messages at certain intervals, file processing, etc.), access to external systems is provided by selecting the connection. When the connection changes, the job execution behavior is updated accordingly
The correctness of the connection can be checked independently of the Integration Flow with the Connection Test feature. This test is critical in the debugging process
## Best Practices
**Bad:** Writing all messages to a single partition.\
**Good:** Spreading traffic by increasing partition count.\
**Best:** Determining partition plan per topic according to consumer count and throughput needs
**Bad:** Adding values randomly.\
**Good:** Manually tracking changes.\
**Best:** Keeping propertiesMap changes under version control with export files
**Bad:** Not tracking keystore/truststore expiration dates.\
**Good:** Keeping a manual calendar.\
**Best:** Automatically planning certificate renewals with Secret Manager events and monitoring alarms
**Bad:** Not monitoring connection health.\
**Good:** Performing manual tests.\
**Best:** Tracking Connection Monitoring metrics with APM/Prometheus and generating automatic alarms
**Bad:** Using the same connection parameters in all environments.\
**Good:** Creating separate connections for each environment.\
**Best:** Managing all environments in a single connection using the Environment option, only changing environment when transitioning between environments
**Bad:** Saving and deploying the connection without testing.\
**Good:** Validating with Test Connection before saving.\
**Best:** Testing after every parameter change, performing full integration test in test environment before going to production
Store SASL username/password or token information as Secret type MapValue; do not leave inline credentials in JAAS config
Grant access to Kafka brokers only from whitelisted IP ranges, close unnecessary ports
Mask log lines containing bootstrap or credentials; do not leave plaintext credentials in debug logs
Store sensitive information such as usernames and passwords using environment variables or secret manager. Do not hardcode credentials in code or configuration files. Update passwords periodically
Always enable SSL/TLS in Production environment. Use self-signed certificates only in development environment. Track certificate expiration dates and renew them on time
Allow only authorized users to change Connection configuration. Store connection change logs. Apply change approval process for critical connections
**Why to avoid:** Connection breaks in broker failure.\
**Alternative:** Define multiple bootstrap.servers addresses
**Why to avoid:** Messages cannot be deserialized, error occurs.\
**Alternative:** Determine key/value serializers according to message format
**Why to avoid:** Risk of credential leakage.\
**Alternative:** Use Secret MapValue
**Why to avoid:** Test data may be written to production system, real users may be affected, security risk occurs.\
**Alternative:** Create separate connections for each environment, use environment parameter, separate connection names by adding prefix according to environment (Test\_, Prod\_)
**Why to avoid:** Connection constantly times out in network delays, Integration steps fail.\
**Alternative:** Adjust timeout values according to real usage scenarios, measure network latency and determine timeouts accordingly
**Why to avoid:** New connection is opened for each request, performance decreases, resource consumption increases, target system load increases.\
**Alternative:** Enable connection pool, adjust pool size according to traffic volume, set up pool monitoring
**Recommendation:** Adjust batch.size value according to message size (e.g., 32 KB).\
**Effect:** Throughput increases with less network collision
**Recommendation:** Select snappy or lz4 as compression.type.\
**Effect:** Bandwidth on broker decreases
**Recommendation:** Prefer asynchronous production by optimizing acks and linger.ms settings.\
**Effect:** Client wait time shortens
**Recommendation:** Adjust pool size according to peak traffic (recommended: concurrent request count × 1.5), set idle connection timeouts, perform pool health check.\
**Effect:** Connection opening cost decreases by 80%, response times decrease, resource usage is optimized
**Recommendation:** Measure real network latency, adjust timeout values accordingly, avoid very low or very high timeouts.\
**Effect:** Unnecessary waits are prevented, fast fail-over is provided, user experience improves
**Recommendation:** Monitor connection pool usage, track timeout rates, perform connection health check, set up alerting.\
**Effect:** Problems are detected proactively, performance bottlenecks are identified early, downtime decreases
## Troubleshooting
Topic name may be incorrect, topic may not be created on Broker, or ACL permissions may be missing.
Verify the topicName field.
Check topic status from broker administrator.
Add the relevant user to ACL policies.
Wrong SASL information, certificate invalid, or TrustStore missing.
Update SASL MapValues.
Check certificate expiration.
Ensure correct TrustStore is selected.
Network delay, target system responding slowly, or timeout value may be too low.
Check network connectivity.
Check target system health.
Increase timeout values.
Review connection logs.
Wrong username/password, expired credentials, or permission problem may exist.
Verify credentials.
Check that the user is active in the target system.
Check that necessary permissions are granted.
Check SSL/TLS certificates.
Pool size may be too low, connection leak exists, or traffic may be too high.
Increase pool size.
Check that connections are properly closed.
Set idle connection timeouts.
Monitor connection usage metrics.
A different connection may be selected in Integration/Connector step, the step may be misconfigured, or Flow/Job may not be redeployed.
Check that the Connection's enable toggle is active.
Verify that the correct connection is selected in Integration Flow.
Redeploy the Connection.
Redeploy Integration Flow or Job.
Check Gateway logs.
## Frequently Asked Questions (FAQ)
The same Connection focuses on a single topic, but can be quickly duplicated for different topics; this way permissions and monitoring are kept separate.
It is possible for Development environment but not recommended to close TLS in Production; keep it open according to security policies.
When adding MapValue, select SECRET as valueType; UI masks values and stores them encrypted during export.
Broker list is tried according to the first reachable server; DNS round-robin or bootstrap list is recommended for high availability.
You can, but it is not recommended; Integration Flow will continue to give errors after Deployment. Fix the error in the test first.
Yes, the same connection can be used in multiple Integration Flow or Connector steps. This provides centralized management and guarantees configuration consistency. However, changes made to the connection will affect all usage locations, so care should be taken.
Using Connection pool is not mandatory but strongly recommended in high-traffic systems. Reusing existing connections instead of opening a new connection for each request significantly increases performance.
Yes, it is recommended to create separate connections for each environment. Alternatively, you can manage all environments in a single connection using the environment parameter. This approach provides easier management and less error risk.
Several reasons may exist:
1. Connection enable toggle may be passive
2. A different connection may be selected in Integration step
3. Connection may not be deployed
4. Integration Flow may not have been redeployed yet