Implement real Jupyter backend with jupyter-client
- Replaced mock exec() implementation with real Jupyter protocol - Uses jupyter-client (host) to connect to ipykernel (container) via ZMQ - 1:1 mapping: one container per session, one kernel per container - Proper Jupyter message protocol for code execution - Kernel lifecycle management (start, execute, shutdown, restart) - Namespace inspection via introspection code - Idle kernel cleanup - Connection file management - Backed up old implementation as kernel_old.py TODO: - Container image needs ipykernel installed - Need to implement proper port mapping for ZMQ - Need to mount connection file into container - Add better kernel readiness check - Implement restart_kernel properly with KernelManager - Write tests
This commit is contained in:
parent
372af75b90
commit
244a3e5574
6 changed files with 940 additions and 253 deletions
|
|
@ -291,17 +291,59 @@ execute_python(
|
|||
**Purpose:** Stateful, multi-step workflows
|
||||
|
||||
**Implementation:**
|
||||
- Spawn Podman container with IPython kernel
|
||||
- Spawn Podman container with IPython kernel (one per session)
|
||||
- jupyter-client runs in MCP-Forge server (host), not in container
|
||||
- ipykernel runs inside container as kernel process
|
||||
- Keep kernel running for session lifetime
|
||||
- Execute code cells via Jupyter protocol (ZMQ)
|
||||
- Maintain namespace between executions
|
||||
- Support rich output formats
|
||||
|
||||
**Architecture Flow:**
|
||||
```
|
||||
MCP-Forge Server (Host)
|
||||
│
|
||||
├─ jupyter-client (ZMQ client library)
|
||||
│ │
|
||||
│ ├─ Manages connections to kernel containers
|
||||
│ └─ Communicates via ZMQ sockets (shell, iopub, stdin, control, heartbeat)
|
||||
│
|
||||
├─ Session "session-123" ──→ Container A ──→ ipykernel Process A
|
||||
├─ Session "session-456" ──→ Container B ──→ ipykernel Process B
|
||||
└─ Session "session-789" ──→ Container C ──→ ipykernel Process C
|
||||
```
|
||||
|
||||
**Session-to-Kernel Mapping (1:1):**
|
||||
- **One container per session** - complete isolation
|
||||
- **One kernel process per container** - dedicated resources
|
||||
- **Separate Python namespaces** - no variable sharing between sessions
|
||||
- **Independent resource limits** - each session has own CPU/memory quota
|
||||
- **Strong security boundary** - container escape affects only one session
|
||||
|
||||
**File Sharing Between Sessions:**
|
||||
Sessions can share files (not variables) via shared volumes:
|
||||
```python
|
||||
# Session 1: Write data
|
||||
execute_python(
|
||||
code="df.to_parquet('/shared/data.parquet')",
|
||||
session_id="session-123",
|
||||
volumes={"/shared": {"bind": "/mcp-forge/projects/abc", "mode": "rw"}}
|
||||
)
|
||||
|
||||
# Session 2: Read data (different container, different namespace)
|
||||
execute_python(
|
||||
code="df = pd.read_parquet('/shared/data.parquet')",
|
||||
session_id="session-456",
|
||||
volumes={"/shared": {"bind": "/mcp-forge/projects/abc", "mode": "ro"}}
|
||||
)
|
||||
```
|
||||
|
||||
**Characteristics:**
|
||||
- State persists between calls
|
||||
- Variable persistence
|
||||
- State persists between calls within same session
|
||||
- Variable persistence in session namespace
|
||||
- Interactive workflow support
|
||||
- Higher resource usage
|
||||
- Higher resource usage per session
|
||||
- Full isolation between sessions
|
||||
|
||||
**Use cases:**
|
||||
- Multi-step data analysis
|
||||
|
|
@ -311,9 +353,11 @@ execute_python(
|
|||
|
||||
**Session Management:**
|
||||
- Sessions identified by unique ID
|
||||
- Each session gets dedicated container and kernel
|
||||
- Automatic timeout after inactivity (configurable, default: 1 hour)
|
||||
- Manual cleanup via session deletion
|
||||
- Resource limits per session
|
||||
- Resource limits enforced per container/session
|
||||
- Clean lifecycle: destroy container = destroy session
|
||||
|
||||
### 3. Custom Environment Building
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue