Add subinterpreter parallelism with OWN_GIL support

.github/workflows/ci.yml

@@ -186,6 +186,82 @@ jobs:
           '
     continue-on-error: true  # Free-threading is experimental
 
+  # Sanitizer builds for detecting memory issues and race conditions
+  test-sanitizers:
+    name: ${{ matrix.sanitizer }} / Python ${{ matrix.python }}
+    runs-on: ubuntu-24.04
+
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # ASan + UBSan with Python 3.12
+          - sanitizer: "ASan+UBSan"
+            python: "3.12"
+            cmake_flags: "-DENABLE_ASAN=ON -DENABLE_UBSAN=ON"
+            env_vars: "ASAN_OPTIONS=detect_leaks=1:abort_on_error=1"
+          # ASan + UBSan with Python 3.13
+          - sanitizer: "ASan+UBSan"
+            python: "3.13"
+            cmake_flags: "-DENABLE_ASAN=ON -DENABLE_UBSAN=ON"
+            env_vars: "ASAN_OPTIONS=detect_leaks=1:abort_on_error=1"
+          # TSan with Python 3.12 (separate because incompatible with ASan)
+          - sanitizer: "TSan"
+            python: "3.12"
+            cmake_flags: "-DENABLE_TSAN=ON"
+            env_vars: "TSAN_OPTIONS=second_deadlock_stack=1"
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Set up Erlang
+        uses: erlef/setup-beam@v1
+        with:
+          otp-version: "27.0"
+          rebar3-version: "3.24"
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y cmake
+
+      - name: Set Python library path
+        run: |
+          PYTHON_LIB=$(python3 -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
+          echo "LD_LIBRARY_PATH=${PYTHON_LIB}:${LD_LIBRARY_PATH}" >> $GITHUB_ENV
+
+      - name: Clean and compile with sanitizers
+        run: |
+          rm -rf _build/cmake
+          mkdir -p _build/cmake
+          cd _build/cmake
+          cmake ../../c_src ${{ matrix.cmake_flags }}
+          cmake --build . -- -j $(nproc)
+          cd ../..
+          rebar3 compile
+
+      - name: Run tests with sanitizers
+        env:
+          ASAN_OPTIONS: ${{ contains(matrix.env_vars, 'ASAN_OPTIONS') && 'detect_leaks=1:abort_on_error=1' || '' }}
+          TSAN_OPTIONS: ${{ contains(matrix.env_vars, 'TSAN_OPTIONS') && 'second_deadlock_stack=1' || '' }}
+        run: |
+          rebar3 ct --readable=compact
+
+      - name: Check debug counters
+        run: |
+          erl -pa _build/default/lib/erlang_python/ebin -noshell -eval '
+            application:ensure_all_started(erlang_python),
+            Counters = py_nif:get_debug_counters(),
+            io:format("Debug counters: ~p~n", [Counters]),
+            halt().
+          '
+
   lint:
     name: Lint
     runs-on: ubuntu-24.04

CHANGELOG.md

@@ -4,6 +4,18 @@
 
 ### Added
 
+- **`erlang.reactor` module** - FD-based protocol handling for building custom servers
+  - `reactor.Protocol` - Base class for implementing protocols
+  - `reactor.serve(sock, protocol_factory)` - Serve connections using a protocol
+  - `reactor.run_fd(fd, protocol_factory)` - Handle a single FD with a protocol
+  - Integrates with Erlang's `enif_select` for efficient I/O multiplexing
+  - Zero-copy buffer management for high-throughput scenarios
+
+- **ETF encoding for PIDs and References** - Full Erlang term format support
+  - Erlang PIDs encode/decode properly in ETF binary format
+  - Erlang References encode/decode properly in ETF binary format
+  - Enables proper serialization for distributed Erlang communication
+
 - **PID serialization** - Erlang PIDs now convert to `erlang.Pid` objects in Python
   and back to real PIDs when returned to Erlang. Previously, PIDs fell through to
   `None` (Erlang→Python) or string representation (Python→Erlang).
@@ -16,12 +28,106 @@
   Subclass of `Exception`, so it's catchable with `except Exception` or
   `except erlang.ProcessError`.
 
+- **Audit hook sandbox** - Block dangerous operations when running inside Erlang VM
+  - Uses Python's `sys.addaudithook()` (PEP 578) for low-level blocking
+  - Blocks: `os.fork`, `os.system`, `os.popen`, `os.exec*`, `os.spawn*`, `subprocess.Popen`
+  - Raises `RuntimeError` with clear message about using Erlang ports instead
+  - Automatically installed when `py_event_loop` NIF is available
+
+- **Process-per-context architecture** - Each Python context runs in dedicated process
+  - `py_context_process` - Gen_server managing a single Python context
+  - `py_context_sup` - Supervisor for context processes
+  - `py_context_router` - Routes calls to appropriate context process
+  - Improved isolation between contexts
+  - Better crash recovery and resource management
+
+- **Worker thread pool** - High-throughput Python operations
+  - Configurable pool size for parallel execution
+  - Efficient work distribution across threads
+
+- **`py:contexts_started/0`** - Helper to check if contexts are ready
+
 ### Changed
 
+- **`py:call_async` renamed to `py:cast`** - Follows gen_server convention where
+  `call` is synchronous and `cast` is asynchronous. The semantics are identical,
+  only the name changed.
+
+- **Unified `erlang` Python module** - Consolidated callback and event loop APIs
+  - `erlang.run(coro)` - Run coroutine with ErlangEventLoop (like uvloop.run)
+  - `erlang.new_event_loop()` - Create new ErlangEventLoop instance
+  - `erlang.install()` - Install ErlangEventLoopPolicy (deprecated in 3.12+)
+  - `erlang.EventLoopPolicy` - Alias for ErlangEventLoopPolicy
+  - Removed separate `erlang_asyncio` module - all functionality now in `erlang`
+
+- **Async worker backend replaced with event loop model** - The pthread+usleep
+  polling async workers have been replaced with an event-driven model using
+  `py_event_loop` and `enif_select`:
+  - Removed `py_async_worker.erl` and `py_async_worker_sup.erl`
+  - Removed `py_async_worker_t` and `async_pending_t` structs from C code
+  - Deprecated `async_worker_new`, `async_call`, `async_gather`, `async_stream` NIFs
+  - Added `py_event_loop_pool.erl` for managing event loop-based async execution
+  - Added `py_event_loop:run_async/2` for submitting coroutines to event loops
+  - Added `nif_event_loop_run_async` NIF for direct coroutine submission
+  - Added `_run_and_send` wrapper in Python for result delivery via `erlang.send()`
+  - **Internal change**: `py:async_call/3,4` and `py:await/1,2` API unchanged
+
 - **`SuspensionRequired` base class** - Now inherits from `BaseException` instead
   of `Exception`. This prevents ASGI/WSGI middleware `except Exception` handlers
   from intercepting the suspension control flow used by `erlang.call()`.
 
+- **Per-interpreter isolation in py_event_loop.c** - Removed global state for
+  proper subinterpreter support. Each interpreter now has isolated event loop state.
+
+- **ErlangEventLoopPolicy always returns ErlangEventLoop** - Previously only
+  returned ErlangEventLoop for main thread; now consistent across all threads.
+
+### Removed
+
+- **Context affinity functions** - Removed `py:bind`, `py:unbind`, `py:is_bound`,
+  `py:with_context`, and `py:ctx_*` functions. The new `py_context_router` provides
+  automatic scheduler-affinity routing. For explicit context control, use
+  `py_context_router:bind_context/1` and `py_context:call/5`.
+
+- **Signal handling support** - Removed `add_signal_handler`/`remove_signal_handler`
+  from ErlangEventLoop. Signal handling should be done at the Erlang VM level.
+  Methods now raise `NotImplementedError` with guidance.
+
+- **Subprocess support** - ErlangEventLoop raises `NotImplementedError` for
+  `subprocess_shell` and `subprocess_exec`. Use Erlang ports (`open_port/2`)
+  for subprocess management instead.
+
+### Fixed
+
+- **FD stealing and UDP connected socket issues** - Fixed file descriptor handling
+  for UDP sockets in connected mode
+
+- **Context test expectations** - Updated tests for Python contextvars behavior
+
+- **Unawaited coroutine warnings** - Fixed warnings in test suite
+
+- **Timer scheduling for standalone ErlangEventLoop** - Fixed timer callbacks not
+  firing for loops created outside the main event loop infrastructure
+
+- **Subinterpreter cleanup and thread worker re-registration** - Fixed cleanup
+  issues when subinterpreters are destroyed and recreated
+
+- **Thread worker handlers not re-registering after app restart** - Workers now
+  properly re-register when application restarts
+
+- **Timeout handling** - Improved timeout handling across the codebase
+
+- **Eval locals_term initialization** - Fixed uninitialized variable in eval
+
+- **Two race conditions in worker pool** - Fixed concurrent access issues
+
+### Performance
+
+- **Async coroutine latency reduced from ~10-20ms to <1ms** - The event loop model
+  eliminates pthread polling overhead
+- **Zero CPU usage when idle** - Event-driven instead of usleep-based polling
+- **No extra threads** - Coroutines run on the existing event loop infrastructure
+
 ## 1.8.1 (2026-02-25)
 
 ### Fixed
@@ -102,16 +208,15 @@
 ### Added
 
 - **Shared Router Architecture for Event Loops**
-  - Single `py_event_router` process handles all event loops (both shared and isolated)
+  - Single `py_event_router` process handles all event loops
   - Timer and FD messages include loop identity for correct dispatch
   - Eliminates need for per-loop router processes
   - Handle-based Python C API using PyCapsule for loop references
 
-- **Isolated Event Loops** - Create isolated event loops with `ErlangEventLoop(isolated=True)`
-  - Default (`isolated=False`): uses the shared global loop managed by Erlang
-  - Isolated (`isolated=True`): creates a dedicated loop with its own pending queue
-  - Full asyncio support (timers, FD operations) for both modes
-  - Useful for multi-threaded Python applications where each thread needs its own loop
+- **Per-Loop Capsule Architecture** - Each `ErlangEventLoop` instance has its own isolated capsule
+  - Dedicated pending queue per loop for proper event routing
+  - Full asyncio support (timers, FD operations) with correct loop isolation
+  - Safe for multi-threaded Python applications where each thread needs its own loop
   - See `docs/asyncio.md` for usage and architecture details
 
 ## 1.6.1 (2026-02-22)

README.md

@@ -32,6 +32,7 @@ Key features:
 - **AI/ML ready** - Examples for embeddings, semantic search, RAG, and LLMs
 - **Logging integration** - Python logging forwarded to Erlang logger
 - **Distributed tracing** - Span-based tracing from Python code
+- **Security sandbox** - Blocks fork/exec operations that would corrupt the VM
 
 ## Requirements
 
@@ -66,7 +67,7 @@ application:ensure_all_started(erlang_python).
 {ok, 25} = py:eval(<<"x * y">>, #{x => 5, y => 5}).
 
 %% Async calls
-Ref = py:call_async(math, factorial, [100]),
+Ref = py:cast(math, factorial, [100]),
 {ok, Result} = py:await(Ref).
 
 %% Streaming from generators
@@ -443,7 +444,7 @@ escript examples/logging_example.erl
 {ok, Result} = py:call(Module, Function, Args, KwArgs, Timeout).
 
 %% Async
-Ref = py:call_async(Module, Function, Args).
+Ref = py:cast(Module, Function, Args).
 {ok, Result} = py:await(Ref).
 {ok, Result} = py:await(Ref, Timeout).
 ```
@@ -573,6 +574,8 @@ py:execution_mode().  %% => free_threaded | subinterp | multi_executor
 - [Threading](docs/threading.md)
 - [Logging and Tracing](docs/logging.md)
 - [Asyncio Event Loop](docs/asyncio.md) - Erlang-native asyncio with TCP/UDP support
+- [Reactor](docs/reactor.md) - FD-based protocol handling
+- [Security](docs/security.md) - Sandbox and blocked operations
 - [Web Frameworks](docs/web-frameworks.md) - ASGI/WSGI integration
 - [Changelog](https://github.com/benoitc/erlang-python/releases)
 

benchmark_results/baseline_20260224_133948.txt.log

@@ -0,0 +1,10 @@
+Error! Failed to eval: 
+            application:ensure_all_started(erlang_python),
+            Results = py_scalable_io_bench:run_all(),
+            py_scalable_io_bench:save_results(Results, "/Users/benoitc/Projects/erlang-python/benchmark_results/baseline_20260224_133948.txt"),
+            init:stop()
+
+
+Runtime terminating during boot ({undef,[{py_scalable_io_bench,run_all,[],[]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,920}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,668}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,276}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
+
+Crash dump is being written to: erl_crash.dump...done

benchmark_results/current_20260224_133950.txt.log

@@ -0,0 +1,10 @@
+Error! Failed to eval: 
+            application:ensure_all_started(erlang_python),
+            Results = py_scalable_io_bench:run_all(),
+            py_scalable_io_bench:save_results(Results, "/Users/benoitc/Projects/erlang-python/benchmark_results/current_20260224_133950.txt"),
+            init:stop()
+
+
+Runtime terminating during boot ({undef,[{py_scalable_io_bench,run_all,[],[]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,920}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,668}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,276}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
+
+Crash dump is being written to: erl_crash.dump...done