Development Workflow Guide¶

This guide explains the comprehensive development workflow for Cracktrader, including CI/CD automation, code quality tools, and best practices.

Table of Contents¶

Overview
Development Environment Setup
Code Quality Tools
Testing Strategy
CI/CD Pipeline
Pre-commit Hooks
Release Process
Performance Monitoring

Overview¶

Cracktrader uses a modern development workflow with automated quality checks, comprehensive testing, and continuous integration. The system is designed to catch issues early and maintain high code quality.

Key Technologies¶

Python 3.11+: Primary language
Ruff: Fast Python linter and formatter
Black: Code formatting
pytest: Testing framework
Bandit: Security scanning
GitHub Actions: CI/CD automation
Pre-commit: Git hooks for quality checks

Development Environment Setup¶

1. Initial Setup¶

# Clone repository
git clone <repository-url>
cd cracktrader

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# or
.venv\Scripts\activate  # Windows

# Install dependencies
pip install -e ".[dev,web]"

2. Install Pre-commit Hooks¶

# Install pre-commit hooks
pre-commit install

# Test hooks
pre-commit run --all-files

3. Verify Installation¶

# Run tests
pytest tests/unit/ -v

# Check code quality
ruff check src/ tests/
ruff format --check src/ tests/

# Run security scan
bandit -r src/ -f json -o bandit-results.json

Code Quality Tools¶

Ruff (Primary Linter and Formatter)¶

Ruff is our primary code quality tool, replacing flake8, isort, and other tools with a single fast implementation.

Configuration: pyproject.toml

[tool.ruff]
line-length = 100
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "W", "C90", "I", "N", "UP", "YTT", "S", "B", "A", "COM", "DTZ", "EM", "G", "INP", "PIE", "T20", "PT", "Q", "RSE", "RET", "SIM", "TID", "ARG", "ERA", "PD", "PGH", "PL", "TRY", "NPY", "PERF", "RUF"]
ignore = ["S101", "PLR0913", "PLR0912", "PLR0915"]

Commands:

# Check code
ruff check src/ tests/

# Auto-fix issues
ruff check src/ tests/ --fix

# Format code
ruff format src/ tests/

# Check formatting
ruff format --check src/ tests/

Black (Backup Formatter)¶

While Ruff handles most formatting, Black is included for compatibility.

# Format with Black
black src/ tests/

# Check formatting
black --check src/ tests/

Bandit (Security Scanner)¶

Bandit scans for common security issues in Python code.

# Run security scan
bandit -r src/ -f json -o bandit-results.json

# Run with configuration
bandit -c pyproject.toml -r src/

Testing Strategy¶

Test Structure¶

tests/
├── unit/           # Unit tests for individual components
├── integration/    # Integration tests for component interaction
└── helpers/        # Test utilities and fixtures

Test Categories¶

Unit Tests: Fast, isolated tests for individual functions/classes
Integration Tests: Test component interactions and system behavior
Performance Tests: Validate performance characteristics
Security Tests: Embedded in regular tests, plus Bandit scans

Running Tests¶

# Run all tests
pytest

# Run specific test categories
pytest tests/unit/ -v
pytest tests/integration/ -v

# Run with coverage
pytest tests/unit/ --cov=src/cracktrader --cov-report=html

# Run performance tests
pytest tests/unit/feed/test_sub_minute_timeframes.py -v -s

# Run specific test
pytest tests/integration/test_cerebro_compatibility.py::TestCerebroCompatibility::test_cerebro_run_with_preload_true -v

Test Configuration¶

Configuration: pyproject.toml

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = "-v --tb=short"
timeout = 30

CI/CD Pipeline¶

GitHub Actions Workflow¶

The CI/CD pipeline runs automatically on push and pull requests to main and develop branches.

File: .github/workflows/ci.yml

Pipeline Stages¶

1. Test Matrix¶

Python 3.11 and 3.12
Ubuntu latest
Parallel execution for speed

2. Dependency Caching¶

Caches pip dependencies
Speeds up subsequent runs
Cache key based on requirements files

3. Code Quality Checks¶

- name: Lint with ruff
  run: |
    ruff check src/ tests/ --output-format=github
    ruff format --check src/ tests/

4. Type Checking (Optional)¶

- name: Type check with mypy
  continue-on-error: true
  run: |
    pip install mypy || echo "mypy not available, skipping"
    mypy src/ || echo "Type checking completed with issues"

5. Unit Tests with Coverage¶

- name: Run unit tests with coverage
  run: |
    pytest tests/unit/ \
      --cov=src/cracktrader \
      --cov-report=xml \
      --cov-report=html \
      --cov-report=term-missing \
      --junitxml=test-results-unit.xml \
      -v

6. Integration Tests¶

- name: Run integration tests
  run: |
    pytest tests/integration/ \
      --junitxml=test-results-integration.xml \
      -v

7. Security Scanning¶

Separate job for security analysis
Uses Bandit for Python security issues
Uploads results as artifacts

8. Build Verification¶

Builds Python package
Verifies package integrity with twine
Uploads artifacts

9. Documentation Build¶

Builds documentation if MkDocs is configured
Uploads documentation artifacts

10. Coverage Badge Update¶

Updates README coverage badge automatically
Runs only on main branch
Uses green/yellow/red color coding

Workflow Triggers¶

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

Artifact Collection¶

The pipeline automatically collects: - Test results (JUnit XML) - Coverage reports (HTML, XML) - Security scan results (JSON) - Build artifacts (wheel, sdist) - Documentation

Pre-commit Hooks¶

Pre-commit hooks run automatically before each commit to ensure code quality.

Configuration: .pre-commit-config.yaml

Hooks Enabled¶

Basic Checks:
trailing-whitespace
end-of-file-fixer
check-yaml
check-added-large-files
check-merge-conflict
debug-statements
Code Formatting:
Black (code formatting)
Ruff (linting and formatting)
Security:
Bandit (security scanning)
Testing:
pytest-check (runs unit tests)

Hook Execution¶

# Manual execution
pre-commit run --all-files

# Skip hooks (emergency only)
git commit -m "message" --no-verify

# Update hooks
pre-commit autoupdate

Release Process¶

Version Management¶

Update Version: Increment version in src/cracktrader/_version.py
Update Changelog: Document changes
Create Git Tag: git tag v1.x.x
Push Tag: git push origin v1.x.x

Automated Release (Future Enhancement)¶

The pipeline can be extended to automatically: - Build and test release candidates - Publish to PyPI - Create GitHub releases - Generate release notes

Performance Monitoring¶

Continuous Performance Testing¶

Performance tests run in CI/CD
Detect performance regressions
Monitor memory usage and processing speed

Key Metrics Tracked¶

Data Processing Performance:
55,000+ candles/second throughput
<1MB memory usage for 1000 candles
<1s processing time for high-frequency data
Reordering Performance:
53,000+ candles/second with reordering
Maintains chronological order
Configurable buffer sizes
Memory Efficiency:
Queue-based architecture prevents memory leaks
Automatic garbage collection
Bounded buffer sizes

Performance Test Examples¶

# Run performance tests
pytest tests/unit/feed/test_sub_minute_timeframes.py::TestSubMinuteTimeframes::test_sub_minute_data_processing_performance -v -s

# Run with profiling
pytest tests/unit/feed/test_tick_reordering.py -v -s --profile

Development Best Practices¶

Code Quality Guidelines¶

Follow PEP 8: Enforced by ruff and black
Write Docstrings: Document all public functions/classes
Type Hints: Use type hints where appropriate
Test Coverage: Maintain >80% test coverage
Security First: No hardcoded secrets, use bandit scanning

Git Workflow¶

Branch Naming: feature/description, fix/description, docs/description
Commit Messages: Clear, descriptive messages
Pull Requests: Required for main branch
Code Review: All changes reviewed before merge

Testing Guidelines¶

Test-Driven Development: Write tests first when possible
Comprehensive Coverage: Unit + integration + performance tests
Mock External Dependencies: Use mocks for exchanges, networks
Performance Regression: Include performance tests for critical paths

Troubleshooting¶

Common Issues¶

1. Pre-commit Hook Failures

# Fix formatting issues
ruff format src/ tests/
ruff check src/ tests/ --fix

# Re-run hooks
pre-commit run --all-files

2. Test Failures

# Run specific failing test
pytest tests/path/to/test.py::TestClass::test_method -v -s

# Debug with pdb
pytest --pdb tests/path/to/test.py::TestClass::test_method

3. CI/CD Pipeline Issues - Check GitHub Actions logs - Verify dependency versions - Ensure all tests pass locally first

4. Performance Issues - Run performance tests locally - Check memory usage patterns - Review queue sizes and buffer configurations

Getting Help¶

Documentation: Check existing docs in docs/
Tests: Look at test examples for usage patterns
Code Comments: Comprehensive docstrings throughout codebase
Issues: Create GitHub issues for bugs or feature requests

Summary¶

The Cracktrader development workflow provides:

✅ Automated Quality Assurance: Pre-commit hooks + CI/CD pipeline
✅ Comprehensive Testing: Unit, integration, and performance tests
✅ Security First: Automated security scanning with Bandit
✅ Performance Monitoring: Continuous performance regression detection
✅ Developer Experience: Fast feedback loops and helpful error messages
✅ Production Ready: 19,200+ lines of tests vs 7,400 lines of source code

This workflow ensures high code quality, catches issues early, and maintains the production-ready status of the Cracktrader cryptocurrency trading framework.