Skip to content

Code Quality Tools & Standards

This document describes the code quality tools configured for the MicroDC Worker project.

Overview

The project uses modern Python code quality tools to maintain high standards:

  • Black (v24.10.0) - Code formatting
  • Ruff (v0.9.1) - Fast linting and auto-fixing
  • isort (v5.13.2) - Import sorting
  • MyPy (v1.14.0) - Type checking
  • Bandit (v1.8.2) - Security vulnerability scanning
  • pytest (v8.4.1) - Testing framework with coverage

Quick Start

1. Install Development Dependencies

# Install all dev tools
pip install -e ".[dev]"

# Or install pre-commit separately
pip install pre-commit

# Install git hooks
pre-commit install

# Run on all files (optional - good first test)
pre-commit run --all-files

After installation, the hooks will run automatically on git commit.

Manual Tool Usage

Code Formatting with Black

# Format all Python files
python -m black src/ tests/ --line-length 100

# Check what would be changed (dry-run)
python -m black src/ tests/ --line-length 100 --check

# Format specific file
python -m black src/core/client.py

Linting with Ruff

# Lint and auto-fix issues
python -m ruff check src/ tests/ --fix

# Check without fixing
python -m ruff check src/ tests/

# Format code (alternative to black)
python -m ruff format src/ tests/

Import Sorting with isort

# Sort imports
python -m isort src/ tests/ --profile black --line-length 100

# Check without changes
python -m isort src/ tests/ --profile black --check-only

Type Checking with MyPy

# Type check source code (tests excluded by default)
python -m mypy src/ --ignore-missing-imports --no-strict-optional

# Check specific module
python -m mypy src/core/ --ignore-missing-imports

Security Scanning with Bandit

# Scan for security issues
python -m bandit -r src/ -c pyproject.toml

# Scan with detailed output
python -m bandit -r src/ -c pyproject.toml -v

Running Tests

Basic Test Execution

# Run all tests
python -m pytest tests/ -v

# Run specific test file
python -m pytest tests/test_config.py -v

# Run specific test class
python -m pytest tests/test_api_models.py::TestEnums -v

# Run with specific test name pattern
python -m pytest tests/ -k "test_config" -v

Test Coverage

# Run with coverage report
python -m pytest tests/ --cov=src --cov-report=term-missing

# Generate HTML coverage report
python -m pytest tests/ --cov=src --cov-report=html
# Open htmlcov/index.html in browser

# Generate JSON coverage report (for CI/CD)
python -m pytest tests/ --cov=src --cov-report=json

Current Coverage Stats (as of 2025-11-23)

  • Overall Coverage: 37% (4547 statements, 2880 uncovered)
  • High Coverage Modules:
  • api/models.py: 98%
  • core/exceptions.py: 100%
  • jobs/monitor.py: 87%
  • core/config.py: 78%
  • Focus Areas for Improvement:
  • jobs/executor.py: 47%
  • processors/document_processor.py: 74%
  • engines/ollama.py: 45%
  • Integration modules (client.py, server_client.py) - tested via integration tests

Pre-Commit Hook Configuration

The project includes a comprehensive .pre-commit-config.yaml with:

General Checks

  • Trailing whitespace removal
  • End-of-file fixer
  • YAML/TOML/JSON validation
  • Large file detection (>1MB)
  • Merge conflict detection
  • Debug statement detection

Python-Specific Hooks

  • Black: Code formatting (line-length=100)
  • isort: Import sorting (black profile)
  • Ruff: Linting with auto-fix
  • MyPy: Type checking (with types-requests)
  • Bandit: Security scanning

Other Hooks

  • markdownlint: Markdown file linting
  • pretty-format-yaml: YAML formatting
  • python-safety-dependencies-check: Dependency vulnerability scanning

Configuration Files

pyproject.toml

Contains configuration for:

  • Black formatting settings
  • Ruff linting rules
  • isort import sorting
  • pytest settings
  • Bandit security rules

pytest.ini

Contains pytest-specific configuration:

  • Test discovery patterns
  • Asyncio settings
  • Coverage exclusions

Continuous Integration

Pre-Commit CI

The project is configured for pre-commit.ci which will:

  • Run all hooks on every PR
  • Auto-fix issues when possible
  • Update hooks weekly
  • Comment on PRs with results

To skip CI on a commit:

git commit -m "message [skip ci]"

Best Practices

Before Committing

  1. Run tests: python -m pytest tests/ -v
  2. Check coverage: python -m pytest tests/ --cov=src
  3. Format code: python -m black src/ tests/
  4. Lint code: python -m ruff check src/ tests/ --fix
  5. Type check: python -m mypy src/

Or simply commit and let pre-commit hooks handle it automatically!

Writing New Code

  1. Add type hints to all function signatures
  2. Write tests for new functionality (aim for >80% coverage)
  3. Document public functions with docstrings
  4. Follow Black formatting (line-length=100)
  5. Avoid security issues - Bandit will catch common vulnerabilities

Updating Dependencies

After updating requirements.txt:

# Check for known vulnerabilities
python -m safety check -r requirements.txt

# Or use pre-commit hook
pre-commit run python-safety-dependencies-check --all-files

Troubleshooting

Pre-Commit Hooks Failing

# Skip hooks temporarily (not recommended)
git commit --no-verify -m "message"

# Update hooks to latest versions
pre-commit autoupdate

# Clear cache and reinstall
pre-commit clean
pre-commit install --install-hooks

MyPy Type Errors

# Ignore missing imports (common for third-party packages)
python -m mypy src/ --ignore-missing-imports

# Generate type stubs for package
stubgen -p package_name -o stubs/

Ruff vs Black Conflicts

Both tools should be compatible with --profile black for isort. If issues arise:

# Ruff can replace Black for formatting
python -m ruff format src/ tests/

Additional Resources