Titanic ML Project Pipeline

A complete end-to-end machine learning workflow using the Titanic dataset. This project demonstrates modern ML engineering practices, including cloud storage, ETL pipelines, feature stores, model training, versioning, and real-time monitoring.

---

🛠️ Project Overview

This project is structured to simulate a real-world machine learning pipeline. It covers everything from raw data ingestion to model deployment and monitoring.

Workflow Steps:

1. Database Setup 🗄️

   • What it is: Creating a home for your raw data.
   • Step: Upload the Titanic CSV to a Google Cloud Platform (GCP) bucket.
   • Example: Data in companies lives in cloud storage or streams through services like Kafka, not on local machines.

2. Project Setup 🏗️

   • What it is: Preparing the "skeleton" of your code.
   • Step: Create a virtual environment, setup.py, and requirements.txt.
   • Structure: Organize folders into src (source code) and pipelines.

3. ETL Pipeline with Astro Airflow 🌪️

   • What it is: Extract, Transform, Load. Automates data movement.

   • Steps:

     • Extract: Pull CSV from GCP.
     • Transform: Convert CSV into SQL table format.
     • Load: Push the table into a Postgres database.

   • Example: Imagine a "one-click" button that moves data to a professional database automatically.

4. Data Ingestion 📥

   • What it is: Bringing data from the database into your ML project.
   • Step: Pull SQL data, convert to CSV, and split into Train, Test, and Validation sets.

5. Jupyter Notebook Testing 📓

   • What it is: Research phase for data exploration.
   • Step: Perform data cleaning, EDA (Exploratory Data Analysis), and feature engineering to validate approaches before automation.

6. Feature Store Setup (RedisAI) ⚡

   • What it is: High-speed storage for clean features.
   • Step: Use Redis (via Docker) to store processed features.
   • Example: Redis enables lightning-fast access (~1ms) for millions of rows compared to static files.

7. Data Processing & Feature Extraction 🧼

   • What it is: Cleaning and preparing features for model training.

   • Steps:

     • Remove duplicates and missing values.
     • Store features using store_data() in Redis.
     • Retrieve features using extract_data() for model training.

8. Model Training 🧠

   • What it is: Training AI models using the features stored in Redis.
   • Step: Extract features from the Feature Store and run the training script to generate model files.

9. Training Pipeline 🔄

   • What it is: Automates ingestion, processing, and training in a single step.
   • Step: Run a unified script that executes the complete workflow from data ingestion to model creation.

10. Versioning 📂

    • What it is: Keep a history of code and data.
    • Step: Use GitHub to manage code and dataset versioning (suitable for small datasets like Titanic).

---

🌐 User Interface & Monitoring

11. Flask App & UI 💻

    • What it is: A web application for user interaction.
    • Step: Build a Flask app where users input passenger information. Style with HTML/CSS.

12. Data & Concept Drift Detection 📉

    • What it is: Monitor changes in data or model relationships.

    • Types:

      • Data Drift: Input data changes (e.g., self-driving car data changes).
      • Concept Drift: Relationship between input and output changes (e.g., new fraud patterns).

    • Tool: Levy Detect library to detect drift.

13. ML Monitoring (MVP) 📊

    • What it is: Dashboard to track model health and performance.

    • Tools:

      • Prometheus: Collect metrics like app usage and drift alerts.
      • Grafana: Visualize metrics with charts and dashboards.

    • Example: Alerts can trigger automatic model retraining when drift exceeds thresholds.

---

🗂️ Project Structure

project/
│
├── src/                # Source code for ML project
│   ├── data/           # Scripts for data ingestion & ETL
│   ├── features/       # Feature engineering and extraction
│   ├── models/         # Model training scripts
│   └── pipelines/      # Orchestration scripts (Airflow/ETL)
│
├── notebooks/          # Jupyter notebooks for EDA and testing
├── docker/             # Docker configuration (Redis, Postgres)
├── app/                # Flask application
├── requirements.txt    # Python dependencies
├── setup.py            # Project setup configuration
└── README.md           # Project documentation

---

⚡ Technologies Used

• Cloud & Storage: GCP Buckets, Postgres SQL
• ETL & Orchestration: Astro Airflow
• Feature Store: RedisAI (via Docker)
• ML & Data Science: Python, Pandas, Scikit-learn
• UI & Deployment: Flask, HTML, CSS
• Monitoring: Prometheus, Grafana, Levy Detect
• Version Control: GitHub

---

🚀 How to Run

1. Clone the repository:

   git clone <repo-url>
   cd project

2. Set up the environment:

   python -m venv venv
   source venv/bin/activate   # Linux/Mac
   venv\Scripts\activate      # Windows
   pip install -r requirements.txt

3. Start Redis and Postgres via Docker:

   docker-compose up -d

4. Run the ETL pipeline:

   python src/pipelines/etl_pipeline.py

5. Train the model:

   python src/models/train_model.py

6. Launch the Flask app:

   python app/main.py

7. Access monitoring dashboards at configured Grafana and Prometheus URLs.

---