Role Summary
Data Science Tester with strong Databricks(goo to have), SQL, and Python skills (must have) to validate data pipelines, ensure data quality, and support end-to-end data validation in a cloud-based analytics environment
Key Responsibilities
1. Data Pipeline Validation (Must Have)
· Validate end-to-end data pipelines, including ingestion, transformation, and output layers.
· Ensure correctness of data movement across systems (source → staging → target).
· Perform validation of batch and streaming pipelines in Databricks environments.
2. Data Quality & Validation Testing (Must Have)
• Design and execute test cases to validate:
• Data completeness
• Data accuracy
• Data consistency across systems
3. SQL-Based Validation (Must Have)
• Write and execute advanced SQL queries for:
• Data validation and reconciliation
• Data profiling and comparison
4. Python-Based Testing (Must Have)
• Develop validation scripts using Python (e.g., Pandas, PySpark).
• Automate data validation checks where feasible.
• Validate transformation logic implemented in PySpark workflows.
5. Databricks Testing & Validation (Good to Have)
• Execute and validate data processing jobs within Databricks.
• Validate notebooks, workflows, and scheduled jobs.
• Perform test execution and result validation in Databricks workspaces.
• Support testing of Spark-based data transformations
6. Data Pipeline & Workflow Testing (Must Have)
• Validate orchestration workflows and dependencies (pipeline triggers, job sequencing).
• Ensure resilience and correctness of pipeline execution.
• Support validation of integration points across data systems.
7. Collaboration & Analysis
• Work closely with data engineers and data scientists to:
• Understand data models and transformations
• Validate analytical outputs
• Participate in requirement analysis and test strategy definition.
• Provide insights on data quality risks and improvements.
Nice to Have (Optional / Advanced)
• Exposure to AI/ML workflows or LLM-based applications (optional)
Responsibilities
Role Summary
Data Science Tester with strong Databricks(goo to have), SQL, and Python skills (must have) to validate data pipelines, ensure data quality, and support end-to-end data validation in a cloud-based analytics environment
Key Responsibilities
1. Data Pipeline Validation (Must Have)
· Validate end-to-end data pipelines, including ingestion, transformation, and output layers.
· Ensure correctness of data movement across systems (source → staging → target).
· Perform validation of batch and streaming pipelines in Databricks environments.
2. Data Quality & Validation Testing (Must Have)
• Design and execute test cases to validate:
• Data completeness
• Data accuracy
• Data consistency across systems
3. SQL-Based Validation (Must Have)
• Write and execute advanced SQL queries for:
• Data validation and reconciliation
• Data profiling and comparison
4. Python-Based Testing (Must Have)
• Develop validation scripts using Python (e.g., Pandas, PySpark).
• Automate data validation checks where feasible.
• Validate transformation logic implemented in PySpark workflows.
5. Databricks Testing & Validation (Good to Have)
• Execute and validate data processing jobs within Databricks.
• Validate notebooks, workflows, and scheduled jobs.
• Perform test execution and result validation in Databricks workspaces.
• Support testing of Spark-based data transformations
6. Data Pipeline & Workflow Testing (Must Have)
• Validate orchestration workflows and dependencies (pipeline triggers, job sequencing).
• Ensure resilience and correctness of pipeline execution.
• Support validation of integration points across data systems.
7. Collaboration & Analysis
• Work closely with data engineers and data scientists to:
• Understand data models and transformations
• Validate analytical outputs
• Participate in requirement analysis and test strategy definition.
• Provide insights on data quality risks and improvements.
Nice to Have (Optional / Advanced)
• Exposure to AI/ML workflows or LLM-based applications (optional)
Salary : As per industry standard.
Industry :IT-Software / Software Services
Functional Area : IT Software - Application Programming , Maintenance
