Jan Hutař's blog: Different Numpy results on different systems

Recently mine wife in her compute intensive project noticed strange issue when same input date and same code produced different output on different hosts in the cloud she is using. She tracked it down to this simple python & numpy test case:

#!/usr/bin/env python

import numpy

a = [[0.67115835, -0.74131401], [0.74131401, 0.67115835]]
b = [[-4.95494273, -1.77170756, ...], [1.87737564, 4.99951546, ...]]
c = numpy.matmul(a, b)
print(c)

On one host it was returning (correct result):

[[-4.71727605 -4.89530718 -4.71727605 -4.89530718 -4.71727605 -4.89530718
...

On different host it was returning (wrong result):

[[ 0.34761728  0.12429531  0.34761728  0.12429531 -0.13170853 -0.35074431
...

We have been googling a bit and found some tips:

Same Python code, same data, different results on different machines: There is a suggestion to use export MKL_CBWR=AUTO - this did not helped
Different results on different computers when using Scipy stack compiled with MKL: Some more tips are here: export MKL_CBWR=AVX; export OMP_NUM_THREADS=1 or with export MKL_CBWR=SSE4_2 - this did not helped again (anyway, our Numpy does not seem to be build with this Intel Math Kernel Library - see ldd output below)
How can I make sure svd having same result on different machine: Here this is mentioned: export NUMEXPR_NUM_THREADS=1; export OPENBLAS_NUM_THREADS=1 - no luck, but finally: export OPENBLAS_CORETYPE=prescott - bingo!

According to OpenBLAS ussage instructions (OpenBLAS is "an optimized BLAS (Basic Linear Algebra Subprograms) library" if you have same knowleadge about it as I do), OPENBLAS_CORETYPE is environment variable which control the kernel selection. Looking at Prescott CPU description, it was launched in 2000, so is probably a safe default. Some more details about our setup:

Numpy in our setup is linked with these libraries:

ldd $( rpm -ql python3-numpy | grep '\.so$' ) | grep -v '\.so:$' | sed 's/([0-9a-zx]\+)/(...)/' | sort -u
	/lib64/ld-linux-x86-64.so.2 (...)
	libc.so.6 => /lib64/libc.so.6 (...)
	libdl.so.2 => /lib64/libdl.so.2 (...)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (...)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (...)
	libm.so.6 => /lib64/libm.so.6 (...)
	libopenblasp.so.0 => /lib64/libopenblasp.so.0 (...)
	libpthread.so.0 => /lib64/libpthread.so.0 (...)
	libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (...)
	libutil.so.1 => /lib64/libutil.so.1 (...)
	linux-vdso.so.1 (...)

The code is packaged in Singularity containers and is running on Metacentrum cloud. Two machines we have hit were - the one with correct result:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)
stepping        : 9
microcode       : 0x1
cpu MHz         : 2199.998
cache size      : 16384 KB
physical id     : 15
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt
+tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4399.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 x86_64 x86_64 GNU/Linux

The other host - the one with wrong results:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
stepping        : 4
microcode       : 0x200004d
cpu MHz         : 2399.392
cache size      : 22528 KB
physical id     : 1
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 63
initial apicid  : 63
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
+nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
+epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
+clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4201.71
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 x86_64 x86_64 GNU/Linux

Packages in the container are:

python3-3.7.4-1.fc30.x86_64
python3-numpy-1.16.4-2.fc30.x86_64

If you want to try, full test case is here:

import numpy

a = [[0.67115835,-0.74131401],[0.74131401,0.67115835]]
b = [[-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936],[1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674]]
c = numpy.matmul(a,b)
print(c)

Jan Hutař's blog

2020-06-03

Different Numpy results on different systems

No comments:

Post a Comment