QDQTransformerTests.MatMul_U8U8S8_FastMath test fails on linux-aarch64 #27510
Description
Describe the issue
The QDQTransformerTests.MatMul_U8U8S8_FastMath ctest fails on linux-aarch64 on the main and rel-1.24.3 branches as demonstrated on CI here:
The output of the failure is as follows:
1: [ RUN ] QDQTransformerTests.MatMul_U8U8S8_FastMath
1: /home/runner/work/onnxruntime-qa/onnxruntime-qa/onnxruntime-src/onnxruntime/test/unittest_util/graph_transform_test_builder.cc:218: Failure
1: Expected equality of these values:
1: ret.first
1: Which is: 4-byte object <01-00 00-00>
1: COMPARE_RESULT::SUCCESS
1: Which is: 4-byte object <00-00 00-00>
1: results differed, cosine similarity factor is 0.969572.
1: Google Test trace:
1: /home/runner/work/onnxruntime-qa/onnxruntime-qa/onnxruntime-src/onnxruntime/test/common/random_generator.h:52: ORT test random seed: 2345
1: Stack trace:
1: 0xab20cca8b36c: onnxruntime::test::TransformerTester()
1: 0xab20cb4f1e60: onnxruntime::test::QDQTransformerMatMulTests<>()::{lambda()#1}::operator()()
1: 0xab20cb4f28c0: onnxruntime::test::QDQTransformerMatMulTests<>()
1: 0xab20ccae5fc8: testing::internal::HandleExceptionsInMethodIfSupported<>()
1: 0xab20ccae64c8: testing::Test::Run()
1: 0xab20ccae676c: testing::TestInfo::Run()
1: ... Google Test internal frames ...
1:
1: /home/runner/work/onnxruntime-qa/onnxruntime-qa/onnxruntime-src/onnxruntime/test/unittest_util/graph_transform_test_builder.cc:218: Failure
1: Expected equality of these values:
1: ret.first
1: Which is: 4-byte object <01-00 00-00>
1: COMPARE_RESULT::SUCCESS
1: Which is: 4-byte object <00-00 00-00>
1: results differed, cosine similarity factor is 0.969572.
1: Google Test trace:
1: /home/runner/work/onnxruntime-qa/onnxruntime-qa/onnxruntime-src/onnxruntime/test/common/random_generator.h:52: ORT test random seed: 2345
1: Stack trace:
1: 0xab20cca8b36c: onnxruntime::test::TransformerTester()
1: 0xab20cb4f2170: onnxruntime::test::QDQTransformerMatMulTests<>()::{lambda()#1}::operator()()
1: 0xab20cb4f28c0: onnxruntime::test::QDQTransformerMatMulTests<>()
1: 0xab20ccae5fc8: testing::internal::HandleExceptionsInMethodIfSupported<>()
1: 0xab20ccae64c8: testing::Test::Run()
1: 0xab20ccae676c: testing::TestInfo::Run()
1: ... Google Test internal frames ...
1:
1: [ FAILED ] QDQTransformerTests.MatMul_U8U8S8_FastMath (51 ms)
To reproduce
The onnxruntime-qa repository provides pixi lock files for reproducible build environments. The build steps taken are:
python tools/ci_build/build.py --update --build_wheel --enable_pybind --build_dir build --config Release --build --parallel --compile_no_warning_as_error --skip_pip_install
ctest -V -C Release --test-dir build/Release/
Urgency
These test failures will likely block us from publishing the upcoming 1.24.3 version on conda-forge.
Platform
Linux
OS Version
ubuntu 24.04
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
rel-1.24.3 and main
ONNX Runtime API
Python
Architecture
ARM64
Execution Provider
Default CPU
Execution Provider Library Version
No response