Self-Service Data Quality
DQOps includes built-in data quality checks that will verify the most common data quality issues that could make the data unusable for machine learning. You just need to connect to the data source, enable the required quality checks, and verify the source data.
- Profile the data quality of new datasets or flat files with 150+ data quality checks.
- Verify the data quality status of training data sets.
- Design custom data quality checks and rules.
Automated Monitoring for ML Success
Automatically monitor the quality of your data to avoid retraining machine learning model with poor data
- Validate your training data daily and get notified about issues.
- Detect outliers in your data using anomaly detection checks.
- Compare seasonal data to a reference value.
Data Quality and ML in 1 Place
All data quality checks are stored in the YAML files, which you can store in Git along with your machine learning scripts. Data quality checks can be easily edited using popular code editors like VSCode with code completion support.
- Store data quality checks in Git.
- Edit data quality checks with a code editor.
- Get auto suggestions (autocomplete) of data quality checks.
Effortless Integration
The DQOps platform stores all data quality configurations in human-readable YAML files. Using REST API Python client, you can run data quality checks from data pipelines and integrate data quality into Apache Airflow.
- Store data configuration in Git.
- Edit data quality configuration in popular code editors.
- Automate any operation visible in the user interface with a Python client.
Data Lineage Protection
Monitor data quality throughout the entire data journey. Use built-in dashboards for quick issue review and root cause identification.
- Ensure each step of data ingestion is functioning properly
- Identify the root cause of the issue.
- Fix problems with data sources.
Granular Pipeline Control
Integrate DQOps easily with scheduling platforms to halt data loading when severe quality issues arise. Once issues are resolved, resume processing seamlessly.
- Monitor data quality directly in the data pipelines.
- Integrate DQOps with Apache Airflow or Dbt.
- Prevent bad data from entering your pipelines.
Data Observability
DQOps applies data observability by automatically activating data quality checks on monitored data sources. You can also monitor data quality in CSV, JSON, or Parquet files.
- Monitoring data ingestion, transformation, and storage processes.
- Detect anomalies, errors, or deviations from expected behavior.
- Proactively address potential issues before they escalate.
Unhealthy Partition Detection
DQOps proactively identifies corrupted or unavailable partitions within your data lake, safeguarding the reliability of your data.
- Detect partitions that are unavailable due to corrupted Parquet files.
- Detect tables and partitions whose files are stored on offline or corrupted HDFS nodes.
- Identify unhealthy partitions and ensure your data lake remains a reliable source of insights.
Seamless Data Ingestion
DQOps safeguards data integrity during the data ingestion process by validating incoming files against defined expectations.
- Detects missing columns in new files, preventing data from being loaded into incorrect locations.
- Analyzes average values to identify reversed or missing columns in CSV files, preventing data from being loaded incorrectly.
- Ensure that the external table always meets the data format and data range checks.
Data Observability at Petabyte Scale
DQOps platform was designed to support analyzing the data quality of large tables. Special partitioned checks analyze data by grouping by a date column, enabling incremental analysis of only the most recent data
- Observe data quality at a petabyte scale.
- Analyze only new or modified data to avoid data lake pressure or high query processing costs.
- Configure the time window for the execution of partitioned checks.
