Parallel testing for faster local testing

[shaypal5]

This pull request introduces parallel test execution support to the testing infrastructure, improving test speed and reliability. It adds new pytest fixtures and configuration to enable safe parallelization, including worker-specific isolation for SQL and cache-based tests. The test runner script (test-local.sh) is enhanced with options for parallel execution, and the test dependencies are updated accordingly.

Key changes include:

Parallel Test Execution Support:

• Added pytest-xdist and related plugins to tests/requirements.txt to enable parallel test execution and automatic retries for flaky tests.
• Implemented new command-line options (-p/--parallel, -w/--workers) in scripts/test-local.sh to run tests in parallel and specify the number of workers. The script now installs pytest-xdist if needed and handles serial vs. parallel test execution, including special handling for pickle and memory tests. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Test Isolation and Fixtures:

• Added tests/conftest.py with fixtures for:
  • Worker-specific PostgreSQL schema isolation for SQL tests, allowing safe parallelization without table conflicts.
  • Isolated cache directories for pickle and maxage tests, preventing cross-worker interference.
  • Cleanup of worker-specific schemas after tests complete.
  • Custom pytest options for parallel testing.

Configuration Updates:

• Updated pyproject.toml:
  • Added tool.black configuration for consistent formatting.
  • Extended ruff ignores for additional security warnings.
  • Added new pytest markers for flaky and serial-local tests.
  • Documented and enabled parallel coverage collection.
  • Enhanced pytest options and documentation for parallel execution. [1] [2] [3] [4] [5]

These changes collectively make the test suite faster, more robust, and ready for parallel execution, especially important for large or slow-running test suites.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (87e79a4) to head (c08e089).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #360   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines         1701      1701           
  Branches       213       213           
=========================================
  Hits          1701      1701           

Flag	Coverage Δ
local	58.55% <ø> (ø)
mongodb	40.91% <ø> (ø)
postgres	43.85% <ø> (ø)
redis	47.14% <ø> (ø)
s3	42.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87e79a4...c08e089. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds opt-in parallel execution support for the Cachier pytest suite (via pytest-xdist) and updates local tooling/docs to make faster, safer local test runs possible across multiple backends.

Changes:

• Add parallel-test dependencies (pytest-xdist, pytest-rerunfailures, pytest-asyncio) to test requirements.
• Introduce new shared pytest fixtures for worker isolation (SQL schema and cache directory).
• Enhance scripts/test-local.sh with --parallel/--workers support and update pyproject.toml pytest/coverage configuration.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tests/requirements.txt	Adds plugins needed for parallel execution and flaky retries.
tests/conftest.py	New autouse fixtures to isolate SQL schemas and cache dirs across xdist workers.
tests/README.md	Adds comprehensive testing/parallelization documentation.
scripts/test-local.sh	Adds CLI flags and logic to run pytest with xdist and manage serial-vs-parallel runs.
pyproject.toml	Adds Black config, new pytest markers, and enables coverage parallel mode.