dataScore: 90/100For Claude Code

Automated ETL pipeline with monitoring and logging

Creates a robust Extract-Transform-Load pipeline with error handling and data quality tracking

Fill in the variables

Your personalized prompt

You are a data engineer specialized in ETL pipelines. Create a complete, production-ready pipeline.

# SPECIFICATIONS
Data sources: {{sources_donnees}}
Update frequency: {{frequence_update}}
Required transformations: {{transformations_requises}}
Final destination: {{destination_finale}}

# EXPECTED DELIVERABLES

## 1. Pipeline Architecture
- Data flow diagram (Extract → Transform → Load)
- Recommended technologies (e.g., Airflow, Prefect, dbt)
- Technical choices justification

## 2. Extraction Code (Extract)
```python
# Code to extract from {{sources_donnees}}
# Include:
- Source connections
- Credentials management (environment variables)
- Pagination if needed
- Timeout and retry logic handling
- Extracted data validation
```

## 3. Transformation Code (Transform)
```python
# Implementation of {{transformations_requises}}
# Include:
- Data cleaning
- Type validation
- Deduplication
- Enrichment if necessary
- Missing values handling
- Quality checks
```

## 4. Loading Code (Load)
```python
# Loading to {{destination_finale}}
# Include:
- Upsert strategy (insert/update)
- Transaction management
- Appropriate indexing
- Post-load verification
```

## 5. Monitoring and Logging
```python
# Tracking system
- Structured logs (with timestamps)
- Performance metrics (duration, volume)
- Failure alerts
- Monitoring dashboard (key indicators)
- Log storage
```

## 6. Testing and Validation
```python
# Test suite
- Unit tests per function
- End-to-end integration tests
- Data quality tests
- Performance tests
```

## 7. Orchestration
```python
# Configuration for {{frequence_update}}
- DAG/workflow definition
- Dependencies management
- Retry policy
- Notifications (success/failure)
```

## 8. Documentation
- README with setup instructions
- Required environment variables
- Deployment procedure
- Troubleshooting guide

Code must be modular, tested, and production-ready.

Go further: optimize this prompt →