Implement real Jupyter backend with jupyter-client

- Replaced mock exec() implementation with real Jupyter protocol
- Uses jupyter-client (host) to connect to ipykernel (container) via ZMQ
- 1:1 mapping: one container per session, one kernel per container
- Proper Jupyter message protocol for code execution
- Kernel lifecycle management (start, execute, shutdown, restart)
- Namespace inspection via introspection code
- Idle kernel cleanup
- Connection file management
- Backed up old implementation as kernel_old.py

TODO:
- Container image needs ipykernel installed
- Need to implement proper port mapping for ZMQ
- Need to mount connection file into container
- Add better kernel readiness check
- Implement restart_kernel properly with KernelManager
- Write tests

docs/todo.md

@@ -1079,22 +1079,42 @@ class SimpleBackend:
   - All tests use mocked ZMQ and containers
 
 **Implementation requirements:**
+
+**Architecture:**
+- `jupyter-client` runs in MCP-Forge server (host) - manages ZMQ connections
+- `ipykernel` runs inside Podman container - actual kernel process
+- **1:1 mapping**: One container per session, one kernel per container
+- **No shared variables** between sessions (separate namespaces)
+- **Optional shared volumes** for file-based data exchange
+
 ```python
 from jupyter_client import KernelManager, BlockingKernelClient
 from typing import Any, Dict, List, Optional
 import zmq
+import json
+import tempfile
 
 @dataclass
 class KernelInfo:
     """Information about running kernel."""
     kernel_id: str
     container_id: str
-    connection_file: Path
+    session_id: str
+    connection_info: Dict[str, Any]  # ZMQ ports and keys
     started_at: datetime
     last_activity: datetime
+    client: Optional[BlockingKernelClient] = None
 
 class JupyterKernelManager:
-    """Manages Jupyter kernel in container."""
+    """
+    Manages IPython kernels in containers via jupyter-client.
+    
+    Architecture:
+    - This class runs on host (MCP-Forge server process)
+    - Creates one container per session with ipykernel running inside
+    - Connects to kernel via ZMQ protocol (jupyter-client)
+    - Communicates with kernel using Jupyter message protocol
+    """
     
     def __init__(
         self,
@@ -1103,7 +1123,7 @@ class JupyterKernelManager:
         resource_limits: ResourceLimits
     ):
         self.container_manager = container_manager
-        self.image = image
+        self.image = image  # Image with ipykernel installed
         self.resource_limits = resource_limits
         self.kernels: Dict[str, KernelInfo] = {}
     
@@ -1113,17 +1133,18 @@ class JupyterKernelManager:
         volumes: Optional[Dict[str, dict]] = None
     ) -> str:
         """
-        Start IPython kernel in container.
+        Start IPython kernel in dedicated container.
         
         Process:
-        1. Create container with IPython kernel
-        2. Start container
-        3. Wait for kernel to be ready
-        4. Connect to kernel via ZMQ
-        5. Verify kernel is responsive
+        1. Generate ZMQ connection info (ports, keys)
+        2. Create container with ipykernel
+        3. Start ipykernel process with connection file
+        4. Wait for kernel to be ready
+        5. Connect jupyter-client to kernel via ZMQ
+        6. Verify kernel is responsive
         
         Returns:
-            kernel_id
+            kernel_id: Unique identifier for this kernel
         """
         pass
     
@@ -1134,23 +1155,34 @@ class JupyterKernelManager:
         timeout: int = 300
     ) -> ExecutionResult:
         """
-        Execute code in kernel.
+        Execute code in kernel via ZMQ.
         
-        Uses ZMQ to send execute request and receive result.
-        Captures stdout, stderr, display data, and result.
+        Uses jupyter-client to:
+        1. Send execute_request message
+        2. Receive stream (stdout/stderr) messages
+        3. Receive execute_result/display_data messages
+        4. Collect and parse all output
+        
+        Returns ExecutionResult with stdout, stderr, result
         """
         pass
     
     def shutdown_kernel(self, kernel_id: str) -> None:
-        """Shutdown kernel and cleanup container."""
+        """
+        Shutdown kernel and cleanup container.
+        
+        1. Send shutdown_request via ZMQ
+        2. Wait for kernel shutdown
+        3. Stop and remove container
+        """
         pass
     
     def inspect_namespace(self, kernel_id: str) -> List[str]:
         """
         Get list of variables in kernel namespace.
         
-        Executes: dir() to get variable names
-        Filters out private variables and builtins
+        Executes introspection code:
+        [var for var in dir() if not var.startswith('_')]
         """
         pass
     
@@ -1160,20 +1192,24 @@ class JupyterKernelManager:
         variable_name: str
     ) -> Dict[str, Any]:
         """
-        Get information about a variable.
+        Get detailed information about a variable.
         
-        Returns:
-            {
-                "type": str,
-                "size_bytes": int (if applicable),
-                "shape": tuple (if array-like),
-                "repr": str (shortened)
-            }
+        Executes introspection code to get:
+        - type(var).__name__
+        - sys.getsizeof(var) if available
+        - var.shape if hasattr(var, 'shape')
+        - repr(var)[:100]
+        
+        Returns dict with type, size, shape, repr
         """
         pass
     
     def restart_kernel(self, kernel_id: str) -> None:
-        """Restart kernel (keeps container, resets namespace)."""
+        """
+        Restart kernel (namespace reset, container kept).
+        
+        Sends restart_request via ZMQ.
+        """
         pass
     
     def cleanup_idle_kernels(
@@ -1190,14 +1226,17 @@ class JupyterKernelManager:
 ```
 
 **Acceptance criteria:**
-- [ ] Kernel starts successfully in container
-- [ ] ZMQ connection established correctly
-- [ ] Code execution works via ZMQ protocol
-- [ ] Namespace persists between executions
+- [ ] jupyter-client dependency in server (host), ipykernel in container image
+- [ ] Kernel starts successfully in dedicated container (1 per session)
+- [ ] ZMQ connection established correctly (ports exposed from container)
+- [ ] Code execution works via Jupyter message protocol
+- [ ] Namespace persists between executions within same session
+- [ ] Each session has completely isolated namespace
 - [ ] Variable introspection works
 - [ ] Variable info includes type, size, shape
 - [ ] Kernel shutdown cleans up container
 - [ ] Idle kernel cleanup works
+- [ ] Kernel restart clears namespace but keeps container
 - [ ] Kernel restart works
 - [ ] Handles kernel crashes gracefully
 - [ ] All tests use mocked ZMQ and containers