How to Set Up Multiple OWOX BI Streamings for Accurate Data Consumption

Introduction

Proper configuration is essential when using multiple Streamings in OWOX BI to ensure accurate data consumption and avoid unexpected costs. This article provides a step-by-step guide on setting up multiple Streamings to prevent duplicate data entries and ensure precise billing.

Understanding the Challenge

Setting up multiple Streamings that write to the same dataset can lead to inflated data consumption calculations. Here's why:

1. Independent Calculations: OWOX BI calculates data consumption separately for each Streaming instance.
2. Data Duplication: Writing to the same dataset causes overlapping data entries, leading to duplicate counts.

For instance, if Streaming #1 and Streaming #2 each write 1,000 unique owox.user_id entries daily to the same dataset, OWOX BI will count these entries twice, resulting in an inflated consumption total of 4,000 instead of 2,000.

Steps to Configure Multiple Streamings Correctly

To ensure accurate data consumption calculations, follow these steps to configure each Streaming with a unique BigQuery dataset:

1. Create and Configure Datasets During Streaming Setup:
   • When setting up a new Streaming instance in OWOX BI, you can create a unique dataset directly within the setup interface.
   • Ensure the dataset you select or create does not already contain events_intraday_* tables to avoid any data overlap.
   • For instance, assign each Streaming to its corresponding unique dataset, such as Streaming_dataset1, Streaming_dataset2, etc., making sure they are empty of any events_intraday_* tables before linking.
2. Verify Configuration:
   • Double-check that each Streaming is correctly configured to its unique dataset.
   • Monitor data entries to ensure no duplication occurs across datasets.

Benefits of Using Unique Datasets

• Accurate Data Consumption: Each Streaming's data is calculated independently, preventing inflated consumption totals.
• Transparent Billing: Avoid unexpected costs by ensuring accurate data tracking and billing.
• Efficient Data Management: Unique datasets help organize and manage data more effectively.