A pipeline processing millions of events daily started taking a full day to complete. Scaling workers to their maximum allocation barely moved the needle. CPU and memory utilization remained surprisingly low despite the backlog growing. The root cause wasn't insufficient infrastructure—it was a classic stream processing pitfall that the right architecture change solved in under an hour of processing time.
Key facts
- Our pipeline joins main event data with enrichment sources before publishing results and persisting to BigQuery
- CoGroupByKey caused data skew where sparse key partitions overwhelmed single workers
- Replacing the approach with Apache Beam's State feature cut processing from 24 hours to 30 minutes
- The change dramatically improved CPU and memory utilization without adding infrastructure
Background: Our Pipeline Architecture
The pipeline processes millions of events daily from two primary data sources, enriched with four additional sources. Cross-charge amounts are calculated by joining these sources, then published to an output Pub/Sub topic. All output messages land in Google BigQuery for downstream analysis and reporting.
We integrated one small enrichment source using Apache Beam's State and CoGroupByKey to pair this dataset with the main flow. The approach worked initially. As data volume grew, processing time stretched from hours to a full day, and costs mounted. The solution became economically unsustainable.
Our Analysis: Why CoGroupByKey Bottlenecked Here
The problem wasn't CoGroupByKey itself—it's a reliable primitive for keyed joins. The issue emerged from how our data distribution interacted with the pattern.
Data skew and high fan-out occur when key partitions become uneven. In our case, certain keys generated sparse but numerous key-value pairs. Apache Beam's State and CoGroupByKey assigned these sparse partitions to individual workers. With millions of events flowing through, a single worker became the bottleneck for the entire pipeline.
Critically, increasing worker count didn't help. With maximum workers deployed, CPU and memory utilization stayed low. The bottleneck wasn't resource availability—it was a serialization point where one worker processed what should have been distributed load.
This matters for teams on Google Cloud because Dataflow pricing ties to worker uptime. A pipeline running for 24 hours with idle resources wastes budget. Beyond economics, longer processing windows delay downstream reporting and create backlogs that cascade into dependent systems.
What We Changed
We replaced the CoGroupByKey-based enrichment with Apache Beam's Stateful processing. The key difference: Stateful processing keeps enrichment data in memory per key, avoiding repeated joins across the worker fleet.
The tradeoff is upfront complexity. Stateful DoFns require careful key design and state management. You must handle late data, clear state appropriately, and ensure keys remain reasonably distributed. The investment pays off when keys are stable and enrichment data fits reasonably in memory.
For teams evaluating this approach, consider: how volatile is your enrichment data? Stateful processing works best when enrichment sources are relatively static within a pipeline run. If enrichment data changes frequently, CoGroupByKey's recomputation per bundle may be more appropriate.
Bottom Line: Profile Before Scaling
Adding workers to a bottlenecked pipeline rarely solves the problem. In our case, the bottleneck was algorithmic—a single worker serializing work that should have been distributed. Profiling revealed low CPU and memory utilization as the signal. For teams seeing similar patterns, examine your key distribution before scaling horizontally. The fix may be architectural rather than infrastructural.
Who Should Care
| Role | Why It Matters |
|---|---|
| Data engineers managing Dataflow pipelines | Avoid the common mistake of scaling out of a skewed problem |
| GCP cost managers | Identify pipelines running longer than necessary on idle resources |
| Architecture leads reviewing beam patterns | Evaluate Stateful processing for stable-key enrichment scenarios |
Key takeaways
- Data skew in CoGroupByKey can bottleneck a pipeline despite available worker capacity
- Stateful processing eliminates repeated joins for stable enrichment sources
- Profile utilization patterns before assuming you need more infrastructure
- The switch from 24 hours to 30 minutes required no additional compute resources
- Stateful DoFns require more upfront design but reduce operational cost at scale
Frequently asked questions
When should I prefer Stateful processing over CoGroupByKey in Apache Beam?
Stateful processing works best when enrichment data is relatively stable within a pipeline run and your keys are evenly distributed. CoGroupByKey suits volatile enrichment sources or when recomputation per bundle is acceptable.
How do I detect data skew in my Dataflow pipeline?
Monitor worker-level CPU and memory utilization. If utilization remains low while pipeline progress stalls, you likely have a serialization bottleneck. Cloud Monitoring provides worker-level metrics.
Does Stateful processing require more memory per worker?
Yes. State is stored in memory per key, so workers need capacity for your largest key's state plus operational overhead. Ensure your worker machine type accommodates this.
Can I combine Stateful processing with other Beam primitives?
Yes. Stateful processing integrates within ParDo, allowing you to combine enrichment with other transformations in your pipeline.
What monitoring should I add after switching to Stateful processing?
Track state size per key, late data rates, and state clearing metrics. Uneven state growth across keys can indicate recurring skew patterns.