automatisationScore: 90/100For GitHub Copilot

Ethical and resilient web scraping bot

Create an automated bot to extract and process web data with anti-blocking management, robots.txt compliance and structured export.

Fill in the variables

Your personalized prompt

You are an expert in web scraping and automation. I want to create a bot to automatically collect data.

**Target platform/site:** {{plateforme_cible}}

**Actions to automate:** {{actions_a_automatiser}}

**Data to extract:** {{donnees_a_extraire}}

**Execution frequency:** {{frequence_execution}}

**Desired output format:** {{format_sortie}}

# Mission
Create a complete, ethical and robust scraping bot including:

1. **Initial configuration**
   - Technology choice (Puppeteer, Playwright, Selenium, BeautifulSoup...)
   - Technical choice justification
   - Setup and required dependencies
   - Environment variables to configure

2. **Best practices compliance**
   - robots.txt verification
   - Appropriate and identifiable user-agent
   - Delays between requests (rate limiting)
   - Retry policy with exponential backoff
   - No server overload

3. **Anti-detection strategies**
   - User-agent rotation
   - Cookie and session management
   - Delay randomization
   - Proxy support (if necessary)
   - Human behavior simulation

4. **Extraction logic**
   - Navigation and authentication if needed
   - Robust CSS/XPath selectors
   - Dynamic content handling (waits)
   - Automatic pagination
   - Data extraction and cleaning

5. **Advanced error handling**
   - HTML structure change detection
   - Captcha handling (notification)
   - Automatic retry with conditions
   - State saving on interruption
   - Screenshots on error for debugging

6. **Storage and export**
   - Extracted data validation
   - Deduplication
   - Export to requested format (CSV, JSON, DB...)
   - Data timestamping and versioning
   - Incremental vs full refresh

7. **Scheduling and deployment**
   - Cron or scheduler configuration
   - Dockerization for portability
   - Environment variables for configuration
   - Structured logs

8. **Monitoring and alerts**
   - Success metrics (extraction rate)
   - Anomaly detection
   - Failure notifications
   - Recommended monitoring dashboard

9. **Documentation**
   - Installation guide
   - Parameter configuration
   - Usage examples
   - Maintenance and troubleshooting

Provide complete code with file structure, ready for production deployment. Include legal and ethical considerations specific to the use case.

Go further: optimize this prompt →