Meeting of Worlds – Kafka Adapter

The meeting of two worlds – 3 Vs (volume, variety, and velocity) of big data and the back office has brought in tremendous advantages to the business world and has technical challenges to be dealt with.

Companies use data collection systems for nearly everything from business analytics to near-real-time operations, to executive reporting. As organizations are moving towards integrating IoTs to the regular operations, the data integration to the back office and operations in real-time (or near-real-time) is more in demand now. Apache Kafka is a leader in streaming the data and can also manage the 3 Vs very well.

While streaming the data from IoT devices, data generating software with the CRM or ERP systems using Kafka and lot of other out of the norm use cases, where customers want a little different or the business process needs a little more than the regular/normal usage, it is a challenge or a gap that is filled by Advantco’s adapter.

Internet of Things to backend SAP system

A large advertising company uses Apache Kafka to stream data collected from the advertisement in the mobile apps. But then the data needs to be transmitted to the backend processes in SAP ERP system for further processing and usage.

Data collection software to backend SAP system

A retailer with online and in-store presence use Apache Kafka to collect order details from a Point of Sale (PoS) software but then needs to transmit to the SAP ERP system in a real-time or near-real-time frequency.

Time-bound Cleanup process

For a specific regulatory purpose, a manufacturing organization needs to delete customer data that has been placed in the Kafka cloud system after a specified amount of time.

Advantco adapter is being used for all these and more use cases. It enables to exchange data between non-SAP systems and the Integration Server or the PCK by means of Kafka APIs.

Benefits of the Advantco Adapter for Kafka

The key advantage of Advantco adapter is that it is fully integrated with the SAP Adapter Framework, which means the Kafka connections can be configured, managed and monitored using the standard SAP PI/PO & CPI tools. With SAP scalability and stability, Advantco Kafka Adapter ensures the highest level of availability. It is also fully integrated with the SAP logging/tracing framework.

• Integration with SAP NetWeaver®: The adapter is built on the SAP NetWeaver® Adapter Framework
• Easy and quick installation by importing an SWCV transport (.tpz) into the Integration Repository/Enterprise Service Builder and deploying the J2EE software archive (SDA or SCA) via the J2EE SDM/JSPM.
• Integration with SAP NetWeaver® Runtime Workbench and SAP NetWeaver® J2EE Logging/Tracing. Supporting SAP NetWeaver PI/PO 7.5, PI 7.4, & SCPI.
• Configuration for the target topics and how to read streams of data
• Configuration for polling
• Adapter Specific Message Attributes
<li </li

Key features of the Advantco Adapter for Kafka

Sender Adapter

The Sender Kakfa adapter must be configured as sender channels in the Integration Builder or the PCK. In a multi-cluster environment, the Sender Kafka adapter implicitly guarantees that there is only one session active among all server nodes at a time.

A sender Kafka adapter is used to consume data from Kafka brokers then it sends data to the Integration Server or the PCK.

Receiver Adapter

The Receiver Kafka adapter must be configured as receiver channels in the Integration Builder or the PCK. Receiver Kafka channel sends message payloads received from the Integration Server or the PCK to Kafka Server.

Content Transformation

Content transformation or conversion parameters are for converting main message payload between Plain, JSON and XML formats.

Producer API

Producers are used to publish messages to one or more Kafka topics. Producers send data to Kafka brokers. Every time a producer publishes a message to a broker, the broker simply appends the message to the last segment file. Actually, the message will be appended to a partition. Producer can also send messages to partition of their choice.

Variable Substitution

To enable the use of message descriptor apart from the payload for enhanced processes, the variable substitution is available varied use cases.

Advance Kafka Configuration

If the basic parameters provided do not fulfill the requirements, advance Kafka configuration can be enabled using the checkbox to specify the supported configurations for Kafka producers. The configuration key is the same as supported by the native Kafka documentation.

Logging and Tracing Configuration

The adapter uses standard SAP Logging API to write log and trace messages to standard log and trace files.

• Log messages help users to get information about adapter execution, including processing errors
• Trace messages are for Advantco’s developers to look deeply into adapter execution, which is helpful for debugging purposes.

Conclusion

The Advantco adapter for Kafka enables exchange streams of data with the integration server or PCK by means of Kafka APIs.

Other benefits include

• Supports a variety of security options for encryption and authentication
• Supports both free-structure payload and Avro-structure payload
• Fully compatible with 0.11.0 and 1.0.0 brokers
• It is supported by a dedicated team with 24X7 support and comes with maintenance and upgrades etc.

Please reach out to our sales team at sales@advantco.com if you have any questions.