Complete Python data science setup with Jupyter, pandas, machine learning libraries, and visualization tools

Data Science Environment

This example provides a comprehensive data science development environment with Python, Jupyter, machine learning libraries, visualization tools, and data engineering capabilities.

Quick Start

Download Configuration

# Download the data science configuration
curl -fsSL https://raw.githubusercontent.com/jameswlane/devex/main/examples/data-science.yaml > ~/.devex/data-science.yaml

# Preview what will be installed
devex config apply --file ~/.devex/data-science.yaml --dry-run

Apply Configuration

# Apply the configuration
devex config apply --file ~/.devex/data-science.yaml

# Install all tools
devex install

Verify Installation

# Check Python and key libraries
python --version
jupyter --version
pip show pandas numpy matplotlib

Complete Configuration

# ~/.devex/data-science.yaml
# Complete Data Science Development Environment

applications:
  development:
    # Version Control
    - name: git
      description: "Distributed version control system"
      category: "Version Control"
      default: true
      priority: 10
      
    - name: git-lfs
      description: "Git Large File Storage for datasets"
      category: "Version Control"
      default: true
      dependencies: ["git"]
      
    # Code Editors
    - name: vscode
      description: "Visual Studio Code with data science extensions"
      category: "Editors"
      default: true
      post_install:
        # Data Science VS Code extensions
        - shell: "code --install-extension ms-python.python"
        - shell: "code --install-extension ms-toolsai.jupyter"
        - shell: "code --install-extension ms-python.pylint"
        - shell: "code --install-extension ms-python.black-formatter"
        - shell: "code --install-extension ms-python.isort"
        - shell: "code --install-extension charliermarsh.ruff"
        - shell: "code --install-extension ms-vscode.vscode-json"
        - shell: "code --install-extension redhat.vscode-yaml"
        - shell: "code --install-extension ms-vscode-remote.remote-containers"
        - shell: "code --install-extension GitHub.copilot"
      config_files:
        - source: "~/.devex/assets/vscode/data-science-settings.json"
          destination: "~/.config/Code/User/settings.json"
          
    # Jupyter Environment
    - name: jupyter
      description: "Jupyter Notebook and JupyterLab"
      category: "Data Science"
      default: true
      dependencies: ["python"]
      install_method: pip
      install_command: "jupyter jupyterlab notebook"
      post_install:
        # Jupyter extensions
        - shell: "pip install jupyterlab-git"
        - shell: "pip install jupyterlab-drawio"
        - shell: "pip install jupyterlab-variableinspector"
        - shell: "pip install jupyterlab-code-formatter"
        - shell: "pip install jupytext"
        - shell: "jupyter labextension install @jupyter-widgets/jupyterlab-manager"
        
    # Alternative Data Science IDEs
    - name: spyder
      description: "Scientific Python IDE"
      category: "Data Science"
      default: false
      dependencies: ["python"]
      install_method: pip
      install_command: spyder
      
    - name: rstudio
      description: "RStudio IDE for R"
      category: "Data Science"
      default: false
      linux:
        install_method: apt
        install_command: rstudio
      macos:
        install_method: brew
        install_command: rstudio
        cask: true
      windows:
        install_method: winget
        install_command: RStudio.RStudio
        
    # Database Tools
    - name: dbeaver
      description: "Universal database tool"
      category: "Database Tools"
      default: false
      linux:
        install_method: flatpak
        install_command: io.dbeaver.DBeaverCommunity
      macos:
        install_method: brew
        install_command: dbeaver-community
        cask: true
      windows:
        install_method: winget
        install_command: dbeaver.dbeaver
        
    - name: pgadmin
      description: "PostgreSQL administration tool"
      category: "Database Tools"
      default: false
      linux:
        install_method: flatpak
        install_command: org.pgadmin.pgAdmin4
        
    # Data Visualization Tools
    - name: tableau-public
      description: "Tableau Public for data visualization"
      category: "Visualization"
      default: false
      macos:
        install_method: brew
        install_command: tableau-public
        cask: true
      windows:
        install_method: winget
        install_command: Tableau.TableauPublic
        
    # Terminal and Shell Tools
    - name: zsh
      description: "Zsh shell with data science plugins"
      category: "Shell"
      default: true
      post_install:
        - shell: "sh -c '$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)'"
        
    - name: fzf
      description: "Fuzzy finder for files and data"
      category: "Utilities"
      default: true
      
    - name: ripgrep
      description: "Fast text search in datasets"
      category: "Utilities"
      default: true
      
    - name: jq
      description: "JSON processor for data manipulation"
      category: "Utilities"
      default: true
      
    - name: csvkit
      description: "CSV data manipulation tools"
      category: "Data Tools"
      default: true
      install_method: pip
      install_command: csvkit
      
    # Container and Virtualization
    - name: docker
      description: "Container platform for reproducible environments"
      category: "Containers"
      default: true
      post_install:
        - shell: "docker pull jupyter/scipy-notebook"
        - shell: "docker pull jupyter/datascience-notebook"
        
    - name: conda
      description: "Package and environment manager"
      category: "Package Managers"
      default: true
      linux:
        install_method: curlpipe
        install_command: "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
      macos:
        install_method: brew
        install_command: miniconda
        cask: true
      post_install:
        - shell: "conda config --set auto_activate_base false"
        - shell: "conda create -n datascience python=3.11 -y"

# Programming Languages
programming_languages:
  - name: Python
    description: "Python for data science and machine learning"
    category: "Programming Languages"
    install_method: mise
    install_command: python@3.11
    default: true
    priority: 20
    shell_updates:
      - "export PYTHONPATH=$PYTHONPATH:."
    post_install:
      # Core data science packages
      - shell: "pip install --upgrade pip setuptools wheel"
      - shell: "pip install numpy pandas matplotlib seaborn"
      - shell: "pip install scikit-learn scipy statsmodels"
      - shell: "pip install jupyter jupyterlab notebook"
      - shell: "pip install plotly bokeh altair"
      - shell: "pip install requests beautifulsoup4 scrapy"
      - shell: "pip install openpyxl xlrd SQLAlchemy"
      
      # Machine Learning and Deep Learning
      - shell: "pip install tensorflow torch torchvision"
      - shell: "pip install transformers datasets"
      - shell: "pip install lightgbm xgboost catboost"
      - shell: "pip install optuna hyperopt"
      
      # Development and Quality Tools
      - shell: "pip install black isort flake8 mypy"
      - shell: "pip install pytest pytest-cov"
      - shell: "pip install pre-commit"
      - shell: "pip install ipython ipdb"
      
      # Specialized Libraries
      - shell: "pip install geopandas folium"
      - shell: "pip install networkx igraph"
      - shell: "pip install nltk spacy textblob"
      - shell: "pip install opencv-python pillow"
      
  - name: R
    description: "R programming language for statistics"
    category: "Programming Languages"
    install_method: mise
    install_command: r@latest
    default: false
    post_install:
      - shell: "R -e \"install.packages(c('tidyverse', 'ggplot2', 'dplyr', 'shiny'), repos='https://cran.rstudio.com/')\""
      - shell: "R -e \"install.packages(c('caret', 'randomForest', 'e1071'), repos='https://cran.rstudio.com/')\""
      
  - name: Julia
    description: "Julia language for high-performance computing"
    category: "Programming Languages"
    install_method: mise
    install_command: julia@latest
    default: false
    post_install:
      - shell: "julia -e 'using Pkg; Pkg.add([\"DataFrames\", \"Plots\", \"MLJ\", \"IJulia\"])'"

# Databases and Data Storage
databases:
  - name: postgresql
    description: "PostgreSQL database for data storage"
    category: "SQL Databases"
    default: true
    service:
      name: postgresql
      enabled: true
      started: true
    post_install:
      - shell: "sudo -u postgres createuser --superuser $USER"
      - shell: "sudo -u postgres createdb $USER"
      - shell: "createdb datasets"
      
  - name: redis
    description: "Redis for caching and data structures"
    category: "NoSQL Databases"
    default: false
    service:
      name: redis-server
      enabled: true
      started: true
      
  - name: mongodb
    description: "MongoDB for document storage"
    category: "NoSQL Databases"
    default: false
    service:
      name: mongod
      enabled: true
      started: true
      
  - name: sqlite
    description: "SQLite for lightweight data storage"
    category: "SQL Databases"
    default: true

# Big Data and Analytics Tools
big_data:
  - name: spark
    description: "Apache Spark for big data processing"
    category: "Big Data"
    default: false
    install_method: curlpipe
    install_command: "https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz"
    
  - name: kafka
    description: "Apache Kafka for data streaming"
    category: "Streaming"
    default: false
    
  - name: elasticsearch
    description: "Elasticsearch for search and analytics"
    category: "Search"
    default: false

# Environment Configuration
environment:
  shell:
    default_shell: "zsh"
    oh_my_zsh: true
    plugins:
      - "git"
      - "python"
      - "docker"
      - "conda"
      - "jupyter"
    theme: "agnoster"
    
  fonts:
    - name: "JetBrains Mono"
      description: "Programming font for code and notebooks"
      method: "url"
      url: "https://download.jetbrains.com/fonts/JetBrainsMono-2.304.zip"
      destination: "~/.local/share/fonts/"
      post_install:
        - shell: "fc-cache -fv"
        
  python:
    global_packages:
      # Core data science stack
      - "numpy>=1.24.0"
      - "pandas>=2.0.0"
      - "matplotlib>=3.7.0"
      - "seaborn>=0.12.0"
      - "scikit-learn>=1.3.0"
      - "scipy>=1.10.0"
      - "statsmodels>=0.14.0"
      
      # Jupyter ecosystem
      - "jupyter>=1.0.0"
      - "jupyterlab>=4.0.0"
      - "notebook>=7.0.0"
      - "ipython>=8.0.0"
      
      # Visualization
      - "plotly>=5.0.0"
      - "bokeh>=3.0.0"
      - "altair>=5.0.0"
      
      # Machine Learning
      - "tensorflow>=2.13.0"
      - "torch>=2.0.0"
      - "transformers>=4.20.0"
      - "lightgbm>=4.0.0"
      - "xgboost>=1.7.0"
      
      # Data Tools
      - "requests>=2.31.0"
      - "beautifulsoup4>=4.12.0"
      - "openpyxl>=3.1.0"
      - "SQLAlchemy>=2.0.0"
      - "psycopg2-binary>=2.9.0"
      
      # Development Tools
      - "black>=23.0.0"
      - "isort>=5.12.0"
      - "flake8>=6.0.0"
      - "mypy>=1.5.0"
      - "pytest>=7.4.0"
      - "pre-commit>=3.3.0"

# System Configuration
system:
  git:
    settings:
      - key: "core.editor"
        value: "code --wait"
      - key: "init.defaultBranch"
        value: "main"
      - key: "pull.rebase"
        value: "false"
      - key: "lfs.track"
        value: "*.csv *.json *.parquet *.h5 *.pkl *.joblib"
        
  jupyter:
    config:
      notebook_dir: "~/notebooks"
      browser: "google-chrome"
      port: 8888
      
  environment_variables:
    JUPYTER_CONFIG_DIR: "~/.jupyter"
    PYTHONPATH: "$PYTHONPATH:."
    MPLBACKEND: "Agg"
    OMP_NUM_THREADS: "4"
    OPENBLAS_NUM_THREADS: "4"
    MKL_NUM_THREADS: "4"
    NUMEXPR_NUM_THREADS: "4"

Jupyter Environment Setup

JupyterLab Configuration

# ~/.jupyter/jupyter_lab_config.py
c = get_config()

# Server configuration
c.ServerApp.ip = 'localhost'
c.ServerApp.port = 8888
c.ServerApp.open_browser = True
c.ServerApp.notebook_dir = '~/notebooks'

# Security
c.ServerApp.token = ''
c.ServerApp.password = ''
c.ServerApp.disable_check_xsrf = False

# Extensions
c.LabApp.default_url = '/lab'

# Resource limits
c.ResourceUseDisplay.mem_limit = 8 * 1024**3  # 8GB
c.ResourceUseDisplay.track_cpu_percent = True

Jupyter Notebook Templates

# Data Analysis Template
# notebooks/templates/data_analysis_template.ipynb

# Cell 1: Imports and Setup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pathlib import Path
import warnings
warnings.filterwarnings('ignore')

# Set style
plt.style.use('default')
sns.set_palette("husl")

# Configuration
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 100)

print("📊 Data Analysis Environment Ready!")

# Cell 2: Data Loading
def load_data(file_path):
    """Load data with automatic format detection"""
    file_path = Path(file_path)
    
    if file_path.suffix == '.csv':
        return pd.read_csv(file_path)
    elif file_path.suffix == '.parquet':
        return pd.read_parquet(file_path)
    elif file_path.suffix in ['.xlsx', '.xls']:
        return pd.read_excel(file_path)
    else:
        raise ValueError(f"Unsupported file format: {file_path.suffix}")

# Load your data
# df = load_data('data/your_dataset.csv')

# Cell 3: Initial Data Exploration
def explore_data(df):
    """Generate comprehensive data exploration report"""
    print("📈 Dataset Overview")
    print(f"Shape: {df.shape}")
    print(f"Memory usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")
    print("\n📋 Column Information:")
    print(df.info())
    print("\n📊 Statistical Summary:")
    print(df.describe(include='all'))
    print("\n🔍 Missing Values:")
    missing = df.isnull().sum()
    print(missing[missing > 0])
    
# explore_data(df)

# Cell 4: Visualization Functions
def plot_distributions(df, numeric_cols=None, figsize=(15, 10)):
    """Plot distributions of numeric columns"""
    if numeric_cols is None:
        numeric_cols = df.select_dtypes(include=[np.number]).columns
    
    n_cols = min(3, len(numeric_cols))
    n_rows = (len(numeric_cols) + n_cols - 1) // n_cols
    
    fig, axes = plt.subplots(n_rows, n_cols, figsize=figsize)
    axes = axes.flatten() if n_rows > 1 else [axes]
    
    for i, col in enumerate(numeric_cols):
        if i < len(axes):
            df[col].hist(bins=30, ax=axes[i], alpha=0.7)
            axes[i].set_title(f'Distribution of {col}')
            axes[i].set_xlabel(col)
            axes[i].set_ylabel('Frequency')
    
    # Hide extra subplots
    for i in range(len(numeric_cols), len(axes)):
        axes[i].set_visible(False)
    
    plt.tight_layout()
    plt.show()

# plot_distributions(df)

VS Code Data Science Configuration

// ~/.devex/assets/vscode/data-science-settings.json
{
  "python.defaultInterpreterPath": "~/.local/share/mise/installs/python/3.11/bin/python",
  "python.terminal.activateEnvironment": true,
  "python.formatting.provider": "black",
  "python.linting.enabled": true,
  "python.linting.pylintEnabled": true,
  "python.linting.flake8Enabled": true,
  
  "jupyter.askForKernelRestart": false,
  "jupyter.interactiveWindowMode": "perFile",
  "jupyter.sendSelectionToInteractiveWindow": true,
  "jupyter.alwaysTrustNotebooks": true,
  "jupyter.widgetScriptSources": ["jsdelivr.com", "unpkg.com"],
  
  "files.associations": {
    "*.ipynb": "jupyter-notebook"
  },
  
  "editor.rulers": [79, 88],
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.organizeImports": true
  },
  
  "workbench.colorTheme": "Visual Studio Dark",
  "workbench.editorAssociations": {
    "*.ipynb": "jupyter.notebook.ipynb"
  },
  
  "[python]": {
    "editor.defaultFormatter": "ms-python.black-formatter",
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
      "source.organizeImports": true
    }
  },
  
  "[jupyter]": {
    "editor.defaultFormatter": "ms-python.black-formatter"
  }
}

Data Science Project Structure

Standard Project Template

# Create standardized data science project structure
mkdir -p ~/projects/data-science-template/{data/{raw,processed,external},notebooks/{exploratory,experiments,reports},src/{data,features,models,visualization},models,reports/{figures,papers},scripts,tests}

# Create project structure
cat > ~/projects/data-science-template/README.md << 'EOF'
# Data Science Project Template

## Project Structure

├── data/ │ ├── raw/ # Original, immutable data │ ├── processed/ # Cleaned, transformed data │ └── external/ # External data sources ├── notebooks/ │ ├── exploratory/ # EDA notebooks │ ├── experiments/ # ML experiments │ └── reports/ # Final analysis notebooks ├── src/ │ ├── data/ # Data loading and preprocessing │ ├── features/ # Feature engineering │ ├── models/ # Model training and prediction │ └── visualization/ # Plotting utilities ├── models/ # Trained models ├── reports/ │ ├── figures/ # Generated graphics │ └── papers/ # Research papers ├── scripts/ # Automation scripts └── tests/ # Unit tests


## Getting Started

1. Install dependencies: `pip install -r requirements.txt`
2. Set up pre-commit hooks: `pre-commit install`
3. Start with notebooks in `notebooks/exploratory/`
4. Move reusable code to `src/`
5. Save final models in `models/`
EOF

# Create requirements.txt
cat > ~/projects/data-science-template/requirements.txt << 'EOF'
# Core data science libraries
numpy>=1.24.0
pandas>=2.0.0
matplotlib>=3.7.0
seaborn>=0.12.0
scipy>=1.10.0
scikit-learn>=1.3.0
jupyter>=1.0.0
jupyterlab>=4.0.0

# Machine learning
tensorflow>=2.13.0
torch>=2.0.0
lightgbm>=4.0.0
xgboost>=1.7.0

# Visualization
plotly>=5.0.0
bokeh>=3.0.0

# Development tools
black>=23.0.0
isort>=5.12.0
flake8>=6.0.0
pytest>=7.4.0
pre-commit>=3.3.0
EOF

# Create environment setup script
cat > ~/projects/data-science-template/setup_environment.sh << 'EOF'
#!/bin/bash
# Setup data science environment

echo "🔬 Setting up data science environment..."

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install requirements
pip install --upgrade pip
pip install -r requirements.txt

# Set up pre-commit hooks
pre-commit install

# Create Jupyter kernel
python -m ipykernel install --user --name=datascience --display-name="Data Science"

echo "✅ Environment setup complete!"
echo "Activate with: source venv/bin/activate"
EOF

chmod +x ~/projects/data-science-template/setup_environment.sh

Specialized Setups

Computer Vision Environment

# Computer Vision with Deep Learning
pip install torch torchvision torchaudio
pip install tensorflow tensorflow-gpu
pip install opencv-python opencv-contrib-python
pip install albumentations
pip install timm  # PyTorch Image Models
pip install segmentation-models-pytorch
pip install detectron2
pip install ultralytics  # YOLOv8
pip install transformers  # Vision Transformers

# Visualization and annotation
pip install matplotlib seaborn
pip install plotly dash
pip install labelimg roboflow

Natural Language Processing Environment

# NLP Libraries
pip install nltk spacy textblob gensim
pip install transformers datasets
pip install sentence-transformers
pip install wordcloud
pip install flair
pip install stanza

# Download language models
python -m spacy download en_core_web_sm
python -m spacy download en_core_web_lg
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords')"

Time Series Analysis Environment

# Time Series Libraries
pip install statsmodels
pip install prophet
pip install pmdarima
pip install arch  # GARCH models
pip install pyflux
pip install tsfresh  # Feature extraction
pip install tslearn  # Time series clustering
pip install sktime   # Time series scikit-learn

# Financial data
pip install yfinance
pip install pandas-datareader
pip install quandl

Development Workflow

Automated Data Science Workflow

#!/bin/bash
# scripts/ds-workflow.sh
# Automated data science workflow

set -e

PROJECT_NAME="$1"
if [ -z "$PROJECT_NAME" ]; then
    echo "Usage: ds-workflow.sh <project-name>"
    exit 1
fi

echo "🔬 Creating data science project: $PROJECT_NAME"

# Create project structure
mkdir -p "$PROJECT_NAME"/{data/{raw,processed,external},notebooks/{exploratory,experiments,reports},src/{data,features,models,visualization},models,reports/{figures,papers},scripts,tests}

cd "$PROJECT_NAME"

# Initialize git repository
git init
echo "__pycache__/
*.py[cod]
*$py.class
.DS_Store
.ipynb_checkpoints/
.env
venv/
.venv/
data/raw/*
!data/raw/.gitkeep
models/*.pkl
models/*.h5
.pytest_cache/" > .gitignore

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install requirements
cat > requirements.txt << 'EOF'
numpy>=1.24.0
pandas>=2.0.0
matplotlib>=3.7.0
seaborn>=0.12.0
scikit-learn>=1.3.0
jupyter>=1.0.0
black>=23.0.0
isort>=5.12.0
pytest>=7.4.0
EOF

pip install -r requirements.txt

# Create initial notebook
cp ~/.devex/templates/data_analysis_template.ipynb notebooks/exploratory/01_initial_exploration.ipynb

# Set up pre-commit
cat > .pre-commit-config.yaml << 'EOF'
repos:
  - repo: https://github.com/psf/black
    rev: 23.7.0
    hooks:
      - id: black
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
EOF

pre-commit install

# Create Jupyter kernel
python -m ipykernel install --user --name="$PROJECT_NAME" --display-name="$PROJECT_NAME"

# Initial commit
git add .
git commit -m "Initial project setup"

echo "✅ Data science project '$PROJECT_NAME' created successfully!"
echo "📁 Location: $(pwd)"
echo "🔬 Activate environment: source venv/bin/activate"
echo "📓 Start Jupyter: jupyter lab"

Model Deployment Pipeline

# scripts/deploy_model.py
import joblib
import pandas as pd
from pathlib import Path
import yaml
import json
from datetime import datetime

def deploy_model(model_path, config_path, target_environment='staging'):
    """Deploy trained model to specified environment"""
    
    # Load model
    model = joblib.load(model_path)
    
    # Load deployment config
    with open(config_path, 'r') as f:
        config = yaml.safe_load(f)
    
    # Model metadata
    metadata = {
        'model_path': str(model_path),
        'model_type': type(model).__name__,
        'deployment_time': datetime.now().isoformat(),
        'environment': target_environment,
        'version': config.get('version', '1.0.0')
    }
    
    print(f"🚀 Deploying model to {target_environment}")
    print(f"📦 Model: {metadata['model_type']}")
    print(f"📅 Time: {metadata['deployment_time']}")
    
    # Create deployment package
    deployment_dir = Path(f"deployments/{target_environment}")
    deployment_dir.mkdir(parents=True, exist_ok=True)
    
    # Copy model
    joblib.dump(model, deployment_dir / "model.pkl")
    
    # Save metadata
    with open(deployment_dir / "metadata.json", 'w') as f:
        json.dump(metadata, f, indent=2)
    
    print(f"✅ Model deployed to {deployment_dir}")
    
    return deployment_dir

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 3:
        print("Usage: python deploy_model.py <model_path> <config_path>")
        sys.exit(1)
    
    model_path = sys.argv[1]
    config_path = sys.argv[2]
    
    deploy_model(model_path, config_path)

Monitoring and Performance

Resource Monitoring

# scripts/monitor_resources.py
import psutil
import GPUtil
import time
import pandas as pd
from datetime import datetime

class ResourceMonitor:
    def __init__(self):
        self.metrics = []
    
    def collect_metrics(self):
        """Collect system resource metrics"""
        # CPU and Memory
        cpu_percent = psutil.cpu_percent(interval=1)
        memory = psutil.virtual_memory()
        
        # GPU metrics (if available)
        gpu_metrics = {}
        try:
            gpus = GPUtil.getGPUs()
            if gpus:
                gpu = gpus[0]  # First GPU
                gpu_metrics = {
                    'gpu_load': gpu.load * 100,
                    'gpu_memory_used': gpu.memoryUsed,
                    'gpu_memory_total': gpu.memoryTotal,
                    'gpu_temperature': gpu.temperature
                }
        except:
            pass
        
        metrics = {
            'timestamp': datetime.now(),
            'cpu_percent': cpu_percent,
            'memory_percent': memory.percent,
            'memory_used_gb': memory.used / (1024**3),
            'memory_total_gb': memory.total / (1024**3),
            **gpu_metrics
        }
        
        self.metrics.append(metrics)
        return metrics
    
    def monitor_training(self, duration_minutes=60, interval_seconds=30):
        """Monitor resources during model training"""
        print(f"📊 Starting resource monitoring for {duration_minutes} minutes")
        
        end_time = time.time() + (duration_minutes * 60)
        
        while time.time() < end_time:
            metrics = self.collect_metrics()
            print(f"CPU: {metrics['cpu_percent']:.1f}% | "
                  f"Memory: {metrics['memory_percent']:.1f}% | "
                  f"GPU Load: {metrics.get('gpu_load', 'N/A')}")
            
            time.sleep(interval_seconds)
        
        # Save metrics
        df = pd.DataFrame(self.metrics)
        df.to_csv(f"monitoring/training_metrics_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv", index=False)
        
        print("✅ Monitoring complete. Metrics saved.")
        return df

# Usage
# monitor = ResourceMonitor()
# monitor.monitor_training(duration_minutes=30)

Data Science Best Practices:

Version Control: Use Git LFS for large datasets and models
Reproducibility: Pin package versions and use random seeds
Documentation: Document data sources, preprocessing steps, and model decisions
Testing: Write tests for data processing and model functions
Monitoring: Track model performance and data drift in production
Security: Don't commit API keys or sensitive data
Collaboration: Use shared data sources and standardized project structure
Environment Management: Use virtual environments and requirements.txt

This data science environment provides a comprehensive foundation for research, experimentation, and production machine learning workflows with proper tooling for reproducibility and collaboration.

Data Science Environment

Data Science Environment

Quick Start

Download Configuration

Apply Configuration

Verify Installation

Complete Configuration

Jupyter Environment Setup

JupyterLab Configuration

Jupyter Notebook Templates

VS Code Data Science Configuration

Data Science Project Structure

Standard Project Template

Specialized Setups

Computer Vision Environment

Natural Language Processing Environment

Time Series Analysis Environment

Development Workflow

Automated Data Science Workflow

Model Deployment Pipeline

Monitoring and Performance

Resource Monitoring

Web Development

Backend Development

Mobile Development

Customization Guide

On this page

Data Science Environment

Data Science EnvironmentCopy Anchor Link

Quick StartCopy Anchor Link

Download ConfigurationCopy Anchor Link

Apply ConfigurationCopy Anchor Link

Verify InstallationCopy Anchor Link

Complete ConfigurationCopy Anchor Link

Jupyter Environment SetupCopy Anchor Link

JupyterLab ConfigurationCopy Anchor Link

Jupyter Notebook TemplatesCopy Anchor Link

VS Code Data Science ConfigurationCopy Anchor Link

Data Science Project StructureCopy Anchor Link

Standard Project TemplateCopy Anchor Link

Specialized SetupsCopy Anchor Link

Computer Vision EnvironmentCopy Anchor Link

Natural Language Processing EnvironmentCopy Anchor Link

Time Series Analysis EnvironmentCopy Anchor Link

Development WorkflowCopy Anchor Link

Automated Data Science WorkflowCopy Anchor Link

Model Deployment PipelineCopy Anchor Link

Monitoring and PerformanceCopy Anchor Link

Resource MonitoringCopy Anchor Link

Web Development

Backend Development

Mobile Development

Customization Guide

On this page

Data Science Environment

Quick Start

Download Configuration

Apply Configuration

Verify Installation

Complete Configuration

Jupyter Environment Setup

JupyterLab Configuration

Jupyter Notebook Templates

VS Code Data Science Configuration

Data Science Project Structure

Standard Project Template

Specialized Setups

Computer Vision Environment

Natural Language Processing Environment

Time Series Analysis Environment

Development Workflow

Automated Data Science Workflow

Model Deployment Pipeline

Monitoring and Performance

Resource Monitoring