Cloudflare Logpush GA: SQL Transformers Cut Costs via Pre-Export Filtering Powered by DataFusion

Cloudflare announced that Logpush now supports Transformers, a feature that allows security and observability teams to filter, redact, and reshape log streams using SQL queries before the data is exported. According to the company’s blog post, this update makes Transformers generally available, enabling users to apply changes directly to log data within the export pipeline before it reaches external destinations like SIEMs or object storage.

How do Transformers change the log export workflow?

Previously, teams often had to deploy separate processing services to clean or modify logs after they arrived at their destination. Cloudflare states that this required additional effort to scale, monitor, and maintain. With Transformers, this processing moves into the Logpush export workflow. Teams define a SQL query to filter records, remove sensitive fields, or reshape the output structure. Logpush applies these queries to each batch before export, ensuring that destinations receive data in the specific format required.

What technical components power the Transformers feature?

The Transformers feature is built on Apache DataFusion and Apache Arrow, according to Cloudflare. These components allow the system to execute SQL queries against log data while maintaining the high throughput and predictable resource usage required by the Logpush service. Cloudflare notes that its data pipeline ingests more than one billion events per second, with the Bufstream delivery service handling over 110 million Logpush records per second. The new feature was designed to meet these operational requirements without introducing additional failure scenarios.

Why might teams use SQL queries for log processing?

Teams can use Transformers to reduce downstream data volume and associated costs by filtering out records they do not need, such as internal health checks. According to the source material, organizations collecting logs from multiple CDNs can add static provider fields to normalize records before they reach analytics platforms. The SQL queries can rename, nest, remove, or add fields, and redact parts of client IP addresses. The transformed results are exported in NDJSON format, matching team-defined schemas.

How do users manage and deploy Transformers?

Customers can create, preview, and manage Transformers through Transformer Studio in the Cloudflare dashboard or programmatically via the API. After saving a Transformer, users attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update to a Transformer creates a new version, allowing teams to review changes over time. This functionality was part of eight major updates to Cloudflare Observability announced during the company’s recent Birthday Week.

What to do and how to stay safe: Cloudflare

• Review current log export jobs to identify high-volume streams that could benefit from pre-export filtering to reduce storage costs.
• Test SQL queries in Transformer Studio to ensure they correctly redact sensitive fields like IP addresses before attaching them to production jobs.
• Monitor the performance of Logpush jobs after enabling Transformers to verify that throughput remains consistent with operational requirements.
• Check if existing downstream analytics platforms require schema normalization that can be handled upstream via Transformers.

Step-by-step guide: Browser Security Settings for Small Business Teams

General security guidance from the TechSec Feed newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

Is the Transformers feature available on all Cloudflare plans?

According to the source material, Logpush and Transformers were made generally available on all self-serve plans during Cloudflare's Birthday Week.

What format is the transformed log data exported in?

The transformed results are exported in NDJSON format, with both input and output using this serialization standard.

Can Transformers convert data between different serialization formats?

No, the source material states that Transformers do not convert between serialization formats; both input and output use NDJSON.