Implement real Jupyter backend with jupyter-client

- Replaced mock exec() implementation with real Jupyter protocol
- Uses jupyter-client (host) to connect to ipykernel (container) via ZMQ
- 1:1 mapping: one container per session, one kernel per container
- Proper Jupyter message protocol for code execution
- Kernel lifecycle management (start, execute, shutdown, restart)
- Namespace inspection via introspection code
- Idle kernel cleanup
- Connection file management
- Backed up old implementation as kernel_old.py

TODO:
- Container image needs ipykernel installed
- Need to implement proper port mapping for ZMQ
- Need to mount connection file into container
- Add better kernel readiness check
- Implement restart_kernel properly with KernelManager
- Write tests
This commit is contained in:
Hans Aschauer 2026-02-07 08:06:44 +01:00
parent 372af75b90
commit 244a3e5574
6 changed files with 940 additions and 253 deletions

View file

@ -291,17 +291,59 @@ execute_python(
**Purpose:** Stateful, multi-step workflows
**Implementation:**
- Spawn Podman container with IPython kernel
- Spawn Podman container with IPython kernel (one per session)
- jupyter-client runs in MCP-Forge server (host), not in container
- ipykernel runs inside container as kernel process
- Keep kernel running for session lifetime
- Execute code cells via Jupyter protocol (ZMQ)
- Maintain namespace between executions
- Support rich output formats
**Architecture Flow:**
```
MCP-Forge Server (Host)
├─ jupyter-client (ZMQ client library)
│ │
│ ├─ Manages connections to kernel containers
│ └─ Communicates via ZMQ sockets (shell, iopub, stdin, control, heartbeat)
├─ Session "session-123" ──→ Container A ──→ ipykernel Process A
├─ Session "session-456" ──→ Container B ──→ ipykernel Process B
└─ Session "session-789" ──→ Container C ──→ ipykernel Process C
```
**Session-to-Kernel Mapping (1:1):**
- **One container per session** - complete isolation
- **One kernel process per container** - dedicated resources
- **Separate Python namespaces** - no variable sharing between sessions
- **Independent resource limits** - each session has own CPU/memory quota
- **Strong security boundary** - container escape affects only one session
**File Sharing Between Sessions:**
Sessions can share files (not variables) via shared volumes:
```python
# Session 1: Write data
execute_python(
code="df.to_parquet('/shared/data.parquet')",
session_id="session-123",
volumes={"/shared": {"bind": "/mcp-forge/projects/abc", "mode": "rw"}}
)
# Session 2: Read data (different container, different namespace)
execute_python(
code="df = pd.read_parquet('/shared/data.parquet')",
session_id="session-456",
volumes={"/shared": {"bind": "/mcp-forge/projects/abc", "mode": "ro"}}
)
```
**Characteristics:**
- State persists between calls
- Variable persistence
- State persists between calls within same session
- Variable persistence in session namespace
- Interactive workflow support
- Higher resource usage
- Higher resource usage per session
- Full isolation between sessions
**Use cases:**
- Multi-step data analysis
@ -311,9 +353,11 @@ execute_python(
**Session Management:**
- Sessions identified by unique ID
- Each session gets dedicated container and kernel
- Automatic timeout after inactivity (configurable, default: 1 hour)
- Manual cleanup via session deletion
- Resource limits per session
- Resource limits enforced per container/session
- Clean lifecycle: destroy container = destroy session
### 3. Custom Environment Building