Change Data Capture (CDC) is a cornerstone of modern cloud-native architectures, enabling real-time event-driven applications. Debezium, a leading CDC platform, provides powerful Single Message Transformations (SMTs) to filter, route, and modify data streams before they reach their destination.
While Debezium is a Java-based project, many cloud-native applications are built in Go. With this latest enhancement, you can now write SMT filters and routers in Go, allowing Go developers to fine-tune data processing within a Java technology thanks to TinyGo and WebAssembly.
This blog post explores how to use Go plugins in Debezium, demonstrating how this feature enhances flexibility and customization for cloud-native teams.
Go SMT Filters and Routers in Debezium
Introduction
Change Data Capture (CDC) is at the heart of modern event-driven architectures, enabling applications to react to database changes in real-time. Debezium has long provided a robust set of Single Message Transformations (SMTs) to filter, route, and modify events before they reach their final destination. But what if you wanted to extend Debezium’s capabilities using a language other than Java?
With the latest enhancement, Debezium now allows developers to write SMT filters and routers in Go, opening up new possibilities for cloud-native applications. Since Go is widely used in cloud environments, this feature empowers teams to customize their CDC pipelines without leaving their primary development ecosystem.
Why Go for SMTs?
Debezium runs on the JVM, and its SMTs have traditionally been Java-based, leveraging JSR 223. While this is great for Java developers, who might want to write filters in Groovy or similar, many teams building cloud-native applications rely on Go. With this new feature, you can now:
-
Write SMT filters and routers in Go, leveraging its simplicity, performance and type system.
-
Avoid Java dependency overhead when Go is your primary language you don’t need to add extra jars to the classpath.
-
Customize event processing logic closer to your existing Go-based microservices.
This makes it easier for Go developers to fine-tune how change events are filtered and routed without needing Java expertise.
How It Works
The integration is based on Go plugins, allowing Debezium to dynamically load and execute Go-based SMTs. These plugins act as lightweight, customizable event processors that filter and route CDC events before they reach Kafka or another downstream system.
Here’s an example of how you can define a simple Go SMT filter:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
package main
import (
"github.com/debezium/debezium-smt-go-pdk"
)
// Example filter function
//export process
func process(proxyPtr uint32) uint32 {
var topicNamePtr = debezium.Get(proxyPtr, "topic")
var topicName = debezium.GetString(topicNamePtr)
return debezium.SetBool(topicName == "mytopic")
}
func main() {}
This function should to be compiled with TinyGo:
tinygo build -ldflags='-extldflags --import-memory' --no-debug -target=wasm-unknown -o filter.wasm filter.goThis function could be loaded as a plugin into Debezium, allowing the event processing pipeline to execute custom filtering logic in Go. A sample configuration on how to achieve this looks as follows:
transforms=filter
transforms.filter.type=io.debezium.transforms.Filter
transforms.filter.language=wasm.chicory
transforms.filter.expression=file://filter.wasmWhere This Fits Best
This feature is particularly useful for:
-
Cloud-native applications where Go is already the dominant language.
-
Teams that want to extend Debezium without Java expertise.
-
Environments with lightweight deployments, where Go’s efficiency and lazy access to the data can help keep resource usage low.
A Glimpse into the Future
While this feature currently focuses on Go-based SMTs, it also hints at a broader trend: enabling Debezium users to bring their own languages to the CDC processing pipeline. WebAssembly (Wasm) could further extend this concept, allowing transformations to be written in a variety of languages while maintaining portability and security. Chicory is being used as a zero dependencies Java Runtime for Wasm.
For now, if you’re working with Go-based cloud-native systems and want to fine-tune your CDC pipeline, this new capability in Debezium is an exciting step forward.
Andrea Peruffo
With nearly two decades of coding experience, I'm fueled by passion as I continue to type away daily. As a Principal Software Engineer at Red Hat, I actively contribute to diverse Open Source projects, driven by both personal fulfillment and professional advancement. My not-so-secret passion lies in programming languages, developer tools, compilers, and beyond. Come and spot me on a project near you!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
