Add pod_executor README with usage examples
This commit is contained in:
parent
db75b822f4
commit
63d9b55a00
1 changed files with 217 additions and 0 deletions
217
src/pod_executor/README.md
Normal file
217
src/pod_executor/README.md
Normal file
|
|
@ -0,0 +1,217 @@
|
||||||
|
# pod_executor
|
||||||
|
|
||||||
|
Standalone Python code execution in Podman containers with security isolation.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
`pod_executor` provides both stateless and stateful (Jupyter) code execution backends that run Python code in isolated Podman containers. It's designed to be usable standalone or as part of larger systems like MCP-Forge.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- ✅ **Stateless execution**: Each code snippet runs in a fresh container
|
||||||
|
- ✅ **Stateful execution**: Jupyter kernels maintain namespace across executions
|
||||||
|
- ✅ **Security isolation**: Containers with configurable resource limits
|
||||||
|
- ✅ **Protocol-based design**: Pluggable audit logging and validation
|
||||||
|
- ✅ **No external configuration**: All parameters explicit
|
||||||
|
- ✅ **Rootless Podman support**: Works with user-level Podman
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Requires Python 3.11+ and Podman
|
||||||
|
pip install podman jupyter-client pyzmq
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Simple Stateless Execution
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
from pod_executor import (
|
||||||
|
CodeExecutor,
|
||||||
|
ResourceLimits,
|
||||||
|
SecureContainerManager,
|
||||||
|
PodmanClient,
|
||||||
|
NoOpValidator,
|
||||||
|
NullAuditLogger
|
||||||
|
)
|
||||||
|
|
||||||
|
# Setup Podman client (user socket)
|
||||||
|
client = PodmanClient(
|
||||||
|
socket_path=Path("/run/user/1000/podman/podman.sock"),
|
||||||
|
validator=NoOpValidator(),
|
||||||
|
audit_logger=NullAuditLogger()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Setup container manager
|
||||||
|
container_manager = SecureContainerManager(
|
||||||
|
podman_client=client,
|
||||||
|
validator=NoOpValidator(),
|
||||||
|
audit_logger=NullAuditLogger()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Setup executor with resource limits
|
||||||
|
limits = ResourceLimits(
|
||||||
|
memory="512m",
|
||||||
|
cpu_quota=100000, # 100% of 1 CPU
|
||||||
|
storage="1g",
|
||||||
|
timeout=30
|
||||||
|
)
|
||||||
|
|
||||||
|
executor = CodeExecutor(
|
||||||
|
container_manager=container_manager,
|
||||||
|
image="python:3.12",
|
||||||
|
resource_limits=limits
|
||||||
|
)
|
||||||
|
|
||||||
|
# Execute code
|
||||||
|
result = executor.execute("print('Hello from container!')")
|
||||||
|
print(result.stdout) # "Hello from container!\n"
|
||||||
|
print(result.exit_code) # 0
|
||||||
|
print(result.execution_time) # e.g., 0.523
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stateful Jupyter Execution
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pod_executor import JupyterBackend
|
||||||
|
|
||||||
|
# Setup backend
|
||||||
|
backend = JupyterBackend(
|
||||||
|
container_manager=container_manager,
|
||||||
|
image="mcp-forge/jupyter:latest",
|
||||||
|
default_timeout=300,
|
||||||
|
default_memory="512m",
|
||||||
|
max_sessions=10,
|
||||||
|
idle_timeout=3600
|
||||||
|
)
|
||||||
|
|
||||||
|
# Execute in session - variables persist
|
||||||
|
result1 = backend.execute("x = 42", session_id="my-session")
|
||||||
|
result2 = backend.execute("print(x * 2)", session_id="my-session")
|
||||||
|
print(result2.stdout) # "84\n"
|
||||||
|
|
||||||
|
# List active sessions
|
||||||
|
sessions = backend.list_sessions()
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
backend.destroy_session("my-session")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
pod_executor/
|
||||||
|
├── security/ # Security components
|
||||||
|
│ ├── resource_limits.py # Memory, CPU, storage limits
|
||||||
|
│ ├── audit.py # Audit logging protocols
|
||||||
|
│ └── validation.py # Security validation protocols
|
||||||
|
├── containers/ # Container management
|
||||||
|
│ ├── client.py # Podman client wrapper
|
||||||
|
│ └── manager.py # Container lifecycle
|
||||||
|
├── simple/ # Stateless execution
|
||||||
|
│ └── executor.py # CodeExecutor
|
||||||
|
└── jupyter/ # Stateful execution
|
||||||
|
├── backend.py # JupyterBackend
|
||||||
|
├── kernel.py # Kernel management
|
||||||
|
└── sessions.py # Session management
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
### Validators
|
||||||
|
|
||||||
|
Three validator implementations:
|
||||||
|
|
||||||
|
1. **NoOpValidator**: No validation (testing only!)
|
||||||
|
2. **BasicValidator**: Minimal checks (image allowlist, forbidden params)
|
||||||
|
3. **Custom**: Implement `OperationValidatorProtocol`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pod_executor.security import BasicValidator
|
||||||
|
|
||||||
|
validator = BasicValidator(allowed_images=["python:3.12*", "jupyter/*"])
|
||||||
|
```
|
||||||
|
|
||||||
|
### Audit Logging
|
||||||
|
|
||||||
|
Three audit logger implementations:
|
||||||
|
|
||||||
|
1. **NullAuditLogger**: No logging
|
||||||
|
2. **SimpleFileAuditLogger**: JSON Lines file logging
|
||||||
|
3. **Custom**: Implement `AuditLoggerProtocol`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pod_executor.security import SimpleFileAuditLogger
|
||||||
|
|
||||||
|
logger = SimpleFileAuditLogger(Path("/var/log/executor/audit.log"))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Resource Limits
|
||||||
|
|
||||||
|
Control container resources:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pod_executor import ResourceLimits
|
||||||
|
|
||||||
|
limits = ResourceLimits(
|
||||||
|
memory="2g", # Memory limit
|
||||||
|
cpu_quota=200000, # CPU quota (200% = 2 CPUs)
|
||||||
|
storage="5g", # Storage limit (tracked, not enforced)
|
||||||
|
timeout=600 # Max execution time in seconds
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Container Images
|
||||||
|
|
||||||
|
Requires Python-capable container images:
|
||||||
|
|
||||||
|
- **Simple executor**: Any Python image (`python:3.12`, `python:3.11-slim`, etc.)
|
||||||
|
- **Jupyter backend**: Image with `ipykernel` installed
|
||||||
|
|
||||||
|
Build Jupyter image:
|
||||||
|
```dockerfile
|
||||||
|
FROM python:3.12-slim
|
||||||
|
RUN pip install ipykernel==6.29.0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pod_executor import SecurityError, KernelError, SessionError
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = executor.execute("import os; os.system('bad')")
|
||||||
|
except SecurityError as e:
|
||||||
|
print(f"Security violation: {e}")
|
||||||
|
except KernelError as e:
|
||||||
|
print(f"Kernel error: {e}")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- **podman** (Python library) - Podman API client
|
||||||
|
- **jupyter-client** - Jupyter kernel protocol (for stateful execution)
|
||||||
|
- **pyzmq** - ZMQ messaging (for Jupyter communication)
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- Requires Podman (not Docker)
|
||||||
|
- Resource limits may not work in all rootless configurations
|
||||||
|
- Jupyter backend needs host networking for ZMQ communication
|
||||||
|
- No automatic image pulling (images must exist)
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run syntax checks
|
||||||
|
python3 -m py_compile src/pod_executor/**/*.py
|
||||||
|
|
||||||
|
# Test simple execution
|
||||||
|
python3 -c "from pod_executor import CodeExecutor; print('Import OK')"
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Part of MCP-Forge project.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue