Enable follower fetching in Aiven for Apache Kafka®

Enabling follower fetching in Aiven for Apache Kafka® allows your consumers to fetch data from the nearest replica instead of the leader, optimizing data fetching and enhancing performance.

note

Follower fetching applies to consumers of classic Kafka topics.

For diskless topics, you can use the diskless_az client ID configuration or AZ-aware listeners to route producers and consumers to brokers in the same availability zone. Aiven enables AZ-aware listeners on request, and they don't support PrivateLink connections.

See Configure AZ-aware listeners for diskless topics.

Prerequisites

• Aiven for Apache Kafka service version 3.6 or later.
• Availability zone (AZ) information for your Aiven for Apache Kafka service.
• Aiven CLI client.
• Aiven Provider for Terraform.

note

Follower fetching is supported on AWS (Amazon Web Services) and Google Cloud.

Identify availability zone

Before configuring follower fetching, identify the AZ where each Kafka consumer runs. Set client.rack to that AZ using the same identifier format as the Kafka brokers.

For the formats used by each cloud provider, see Zone ID formats.

• AWS: Availability zone (AZ) names can vary across different accounts. The same physical location might have different AZ names in different accounts. To ensure consistency when configuring client.rack, use the AZ ID, which remains the same across accounts.

  To find the AZ ID used by your brokers, check the broker rack information in your client logs or service connection details.

  To map AZ names to AZ IDs, see the AWS Knowledge Center article and the AWS documentation on AZ IDs.

• Google Cloud: Use the AZ name directly as the client.rack value. For more information, see Google Cloud's regions and zones documentation.

Enable follower fetching

Use one of the following methods to enable follower fetching on your Aiven for Apache Kafka service.

Console

1. Access the Aiven Console, and select your Aiven for Apache Kafka service.
2. Click Service settings.
3. Scroll to Advanced configuration and click Configure.
4. Click Add configuration options.
5. Select follower_fetching.enabled from the list and set the value to Enabled.
6. Click Save configurations.

Enabling follower fetching at the service level allows Kafka clients and Aiven-managed services to use rack-aware fetching.

You must still configure client.rack on Kafka consumers. Aiven for Apache Kafka® Connect and Aiven for Apache Kafka® MirrorMaker 2 configure this automatically based on the availability zone where each node runs.

CLI

Enable follower fetching on an existing service with the Aiven CLI:

avn service update <service-name> -c follower_fetching.enabled=true

Parameters:

• <service-name>: Name of your Aiven for Apache Kafka service.
• follower_fetching.enabled=true: Enables the follower fetching feature.

API

Use the ServiceUpdate API to enable follower fetching on an existing service:

curl --request PUT \
    --url https://api.aiven.io/v1/project/YOUR_PROJECT_NAME/service/YOUR_SERVICE_NAME \
    --header 'Authorization: Bearer YOUR_BEARER_TOKEN' \
    --header 'content-type: application/json' \
    --data '{
        "user_config": {
            "follower_fetching": {
                "enabled": true
            }
        }
    }'

Parameters:

• YOUR_PROJECT_NAME: Name of your project.
• YOUR_SERVICE_NAME: Name of your service.
• YOUR_BEARER_TOKEN: API token for authentication.
• follower_fetching={"enabled": true}: Enables the follower fetching feature.

Terraform

Use the follower_fetching attribute in your aiven_kafka resource.

Client-side configuration

To enable follower fetching at the client level, configure the client.rack setting in the Apache Kafka client. Set the client.rack value to the corresponding AZ ID for AWS or AZ name for Google Cloud for each client. This allows the consumer to fetch data from a replica in its AZ when an eligible replica is available.

Example configuration for a consumer on AWS:

client.rack=use1-az1

Example configuration for a consumer on Google Cloud:

client.rack=europe-west1-b

Example scenario: follower fetching in different AZs

Assume you have an Aiven for Apache Kafka cluster running in two AZs in the us-east-1 region for AWS and in the europe-west1 region for Google Cloud:

Cluster setup and consumer distribution

Cloud	Region	AZs for brokers	AZs for consumers
AWS	us-east-1	use1-az1, use1-az2	use1-az1, use1-az2, use1-az3
Google Cloud	europe-west1	europe-west1-b, europe-west1-c	europe-west1-b, europe-west1-c, europe-west1-d

Consumer configuration

Set the client.rack value to the respective AZ ID for AWS or AZ name for Google Cloud for each consumer:

# AWS consumers in use1-az1
client.rack=use1-az1

# AWS consumers in use1-az2
client.rack=use1-az2

# AWS consumers in use1-az3
client.rack=use1-az3

# Google Cloud consumers in europe-west1-b
client.rack=europe-west1-b

# Google Cloud consumers in europe-west1-c
client.rack=europe-west1-c

# Google Cloud consumers in europe-west1-d
client.rack=europe-west1-d

Fetching behavior

Cloud	Consumer location	Fetching behavior	Notes
AWS	use1-az1	Can fetch from a local replica	Reduced latency and network costs
AWS	use1-az2	Can fetch from a local replica	Reduced latency and network costs
AWS	use1-az3	Fetch from the leader (no matching broker.rack)	No follower fetching possible
Google Cloud	europe-west1-b	Can fetch from a local replica	Reduced latency and network costs
Google Cloud	europe-west1-c	Can fetch from a local replica	Reduced latency and network costs
Google Cloud	europe-west1-d	Fetch from the leader (no matching broker.rack)	No follower fetching possible

Example scenario: incorrect client.rack value

A consumer runs in use1-az1 but is configured with the following value:

client.rack=use1-az2

Kafka treats the consumer as running in use1-az2 and can direct it to a replica there. Because the consumer is in use1-az1, this increases cross-AZ network traffic instead of reducing it. Set client.rack to the AZ where the consumer runs.

Use follower fetching with Kafka Connect and MirrorMaker 2

Aiven for Apache Kafka® Connect and Aiven for Apache Kafka® MirrorMaker 2 use follower fetching when it is enabled on your Aiven for Kafka service. The client.rack values that these services set apply to classic topics only. For diskless topics, see Configure AZ-aware listeners for diskless topics.

Kafka Connect

When follower fetching is enabled on the Aiven for Apache Kafka® service, rack-aware fetching is enabled by default for Kafka Connect sink connectors.

Kafka Connect sets consumer.client.rack based on each node’s availability zone.

Sink connectors use this value when consuming data from Kafka. Source connectors do not use follower fetching. To disable rack awareness for a specific sink connector, set:

{
  "consumer.override.client.rack": "noop"
}

MirrorMaker 2

When follower fetching is enabled for a replication flow, MirrorMaker 2 reads from in-sync follower replicas in the same availability zone as the MirrorMaker 2 node.

Follower fetching is enabled by default for new replication flows. You can disable it per replication flow if needed.

For details about how rack awareness works in MirrorMaker 2, see Configure rack awareness in MirrorMaker 2.

Verify follower fetching

After enabling follower fetching, monitor cross-availability-zone network traffic and data transfer costs to assess its effectiveness.

Related pages

• Follower fetching in Aiven for Apache Kafka®
• Configure AZ-aware listeners for diskless topics