picnoir/glibc
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Vector sincos for x86_64 and tests.

Browse source 
Here is implementation of vectorized sincos containing SSE, AVX,
AVX2 and AVX512 versions according to Vector ABI
<https://groups.google.com/forum/#!topic/x86-64-abi/LmppCfN1rZ4>.

    * NEWS: Mention addition of x86_64 vector sincos.
    * bits/libm-simd-decl-stubs.h: Added stubs for sincos.
    * math/math.h (__MATHDECL_VEC): New macro.
    * math/bits/mathcalls.h: Added sincos declaration with __MATHDECL_VEC.
    * math/gen-libm-have-vector-test.sh: Added generation of sincos wrapper
    declaration under condition.
    * math/test-vec-loop.h (TEST_VEC_LOOP): Refactored.
    * math/test-double-vlen2.h: Added wrapper for sincos tests, reflected
    TEST_VEC_LOOP change.
    * math/test-double-vlen4.h: Likewise.
    * math/test-double-vlen8.h: Likewise.
    * math/test-float-vlen16.h: Reflected TEST_VEC_LOOP change.
    * math/test-float-vlen4.h: Likewise.
    * math/test-float-vlen8.h: Likewise.
    * sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New symbols added.
    * sysdeps/x86/fpu/bits/math-vector.h: Added sincos SIMD declaration.
    * sysdeps/x86_64/fpu/Makefile (libmvec-support): Added new files.
    * sysdeps/x86_64/fpu/Versions: New versions added.
    * sysdeps/x86_64/fpu/libm-test-ulps: Regenerated.
    * sysdeps/x86_64/fpu/multiarch/Makefile (libmvec-sysdep_routines):
    Added build of SSE, AVX2 and AVX512 IFUNC versions.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core_sse4.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core_avx2.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core_avx512.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos2_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos4_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos4_core_avx.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos8_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos_data.S: New file.
    * sysdeps/x86_64/fpu/svml_d_sincos_data.h: New file.
    * sysdeps/x86_64/fpu/svml_d_wrapper_impl.h: Added wrappers for sincos.
    * sysdeps/x86_64/fpu/test-double-vlen2-wrappers.c: Vector sincos tests.
    * sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-avx2-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen8-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise.
This commit is contained in:
Andrew Senkevich 2015-06-18 17:55:55 +03:00
parent 8aa92022e2
commit c9a8c526ac
40 changed files with 1917 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
ChangeLog
View file

@ -1,5 +1,48 @@
2015-06-18 Andrew Senkevich <andrew.senkevich@intel.com>
* NEWS: Mention addition of x86_64 vector sincos.
* bits/libm-simd-decl-stubs.h: Added stubs for sincos.
* math/math.h (__MATHDECL_VEC): New macro.
* math/bits/mathcalls.h: Added sincos declaration with __MATHDECL_VEC.
* math/gen-libm-have-vector-test.sh: Added generation of sincos wrapper
declaration under condition.
* math/test-vec-loop.h (TEST_VEC_LOOP): Refactored.
* math/test-double-vlen2.h: Added wrapper for sincos tests, reflected
TEST_VEC_LOOP change.
* math/test-double-vlen4.h: Likewise.
* math/test-double-vlen8.h: Likewise.
* math/test-float-vlen16.h: Reflected TEST_VEC_LOOP change.
* math/test-float-vlen4.h: Likewise.
* math/test-float-vlen8.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New symbols added.
* sysdeps/x86/fpu/bits/math-vector.h: Added sincos SIMD declaration.
* sysdeps/x86_64/fpu/Makefile (libmvec-support): Added new files.
* sysdeps/x86_64/fpu/Versions: New versions added.
* sysdeps/x86_64/fpu/libm-test-ulps: Regenerated.
* sysdeps/x86_64/fpu/multiarch/Makefile (libmvec-sysdep_routines):
Added build of SSE, AVX2 and AVX512 IFUNC versions.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core_sse4.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core_avx2.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core_avx512.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos2_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos4_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos4_core_avx.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos8_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos_data.S: New file.
* sysdeps/x86_64/fpu/svml_d_sincos_data.h: New file.
* sysdeps/x86_64/fpu/svml_d_wrapper_impl.h: Added wrappers for sincos.
* sysdeps/x86_64/fpu/test-double-vlen2-wrappers.c: Vector sincos tests.
* sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-avx2-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen8-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise.
* sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New symbols added.
* sysdeps/x86/fpu/bits/math-vector.h: Added SIMD declaration and asm
redirections for powf.

3
NEWS
View file

@ -55,7 +55,8 @@ Version 2.22
condition in some applications.
* Added vector math library named libmvec with the following vectorized x86_64
implementations: cos, cosf, sin, sinf, log, logf, exp, expf, pow, powf.
implementations: cos, cosf, sin, sinf, sincos, log, logf, exp, expf, pow,
powf.
The library can be disabled with --disable-mathvec. Use of the functions is
enabled with -fopenmp -ffast-math starting from -O1 for GCC version >= 4.9.0.
The library is linked in as needed when using -lm (no need to specify -lmvec

4
bits/libm-simd-decl-stubs.h
View file

@ -41,6 +41,10 @@
#define __DECL_SIMD_sinf
#define __DECL_SIMD_sinl
#define __DECL_SIMD_sincos
#define __DECL_SIMD_sincosf
#define __DECL_SIMD_sincosl
#define __DECL_SIMD_log
#define __DECL_SIMD_logf
#define __DECL_SIMD_logl

4
math/bits/mathcalls.h
View file

@ -78,8 +78,8 @@ _Mdouble_END_NAMESPACE
#ifdef __USE_GNU
/* Cosine and sine of X. */
__MATHDECL (void,sincos,,
(_Mdouble_ __x, _Mdouble_ *__sinx, _Mdouble_ *__cosx));
__MATHDECL_VEC (void,sincos,,
(_Mdouble_ __x, _Mdouble_ *__sinx, _Mdouble_ *__cosx));
#endif
#if defined __USE_XOPEN_EXTENDED || defined __USE_ISOC99

8
math/gen-libm-have-vector-test.sh
View file

@ -33,7 +33,7 @@ print_defs()
echo
}
for func in $(cat libm-test.inc | grep ALL_RM_TEST | grep -v define | grep -v RUN_TEST_LOOP_ff_f | sed -r "s/.*\(//; s/,.*//" ); do
for func in $(cat libm-test.inc | grep ALL_RM_TEST | grep -v define | grep -v RUN_TEST_LOOP_ff_f | grep -v RUN_TEST_LOOP_fFF_11 | sed -r "s/.*\(//; s/,.*//" ); do
print_defs ${func}
print_defs ${func}f
print_defs ${func}l
@ -45,6 +45,12 @@ for func in $(cat libm-test.inc | grep ALL_RM_TEST | grep RUN_TEST_LOOP_ff_f | s
print_defs ${func}l "_ff"
done
for func in $(cat libm-test.inc | grep ALL_RM_TEST | grep RUN_TEST_LOOP_fFF_11 | sed -r "s/.*\(//; s/,.*//" ); do
print_defs ${func} "_fFF"
print_defs ${func}f "_fFF"
print_defs ${func}l "_fFF"
done
# When all functions will use ALL_RM_TEST instead of using START directly,
# this code can be removed.
for func in $(grep 'START.*;$' libm-test.inc | sed -r "s/.*\(//; s/,.*//"); do

4
math/math.h
View file

@ -58,6 +58,10 @@ __BEGIN_DECLS
__SIMD_DECL (__MATH_PRECNAME (function, suffix)) \
__MATHCALL (function, suffix, args)
#define __MATHDECL_VEC(type, function,suffix, args) \
__SIMD_DECL (__MATH_PRECNAME (function, suffix)) \
__MATHDECL(type, function,suffix, args)
#define __MATHCALL(function,suffix, args) \
__MATHDECL (_Mdouble_,function,suffix, args)
#define __MATHDECL(type, function,suffix, args) \

23
math/test-double-vlen2.h
View file

@ -44,6 +44,7 @@
#define WRAPPER_DECL(function) extern FLOAT function (FLOAT);
#define WRAPPER_DECL_ff(function) extern FLOAT function (FLOAT, FLOAT);
#define WRAPPER_DECL_fFF(function) extern void function (FLOAT, FLOAT *, FLOAT *);
// Wrapper from scalar to vector function with vector length 2.
#define VECTOR_WRAPPER(scalar_func, vector_func) \
@ -54,7 +55,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 2); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (2); \
TEST_VEC_LOOP (mr, 2); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +69,22 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 2); \
INIT_VEC_LOOP (my, y, 2); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (2); \
TEST_VEC_LOOP (mr, 2); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 3 argument function to vector one.
#define VECTOR_WRAPPER_fFF(scalar_func, vector_func) \
extern void vector_func (VEC_TYPE, VEC_TYPE *, VEC_TYPE *); \
void scalar_func (FLOAT x, FLOAT * r, FLOAT * r1) \
{ \
int i; \
VEC_TYPE mx, mr, mr1; \
INIT_VEC_LOOP (mx, x, 2); \
vector_func (mx, &mr, &mr1); \
TEST_VEC_LOOP (mr, 2); \
TEST_VEC_LOOP (mr1, 2); \
*r = (FLOAT) mr[0]; \
*r1 = (FLOAT) mr1[0]; \
return; \
}

23
math/test-double-vlen4.h
View file

@ -44,6 +44,7 @@
#define WRAPPER_DECL(function) extern FLOAT function (FLOAT);
#define WRAPPER_DECL_ff(function) extern FLOAT function (FLOAT, FLOAT);
#define WRAPPER_DECL_fFF(function) extern void function (FLOAT, FLOAT *, FLOAT *);
// Wrapper from scalar to vector function with vector length 4.
#define VECTOR_WRAPPER(scalar_func, vector_func) \
@ -54,7 +55,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 4); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (4); \
TEST_VEC_LOOP (mr, 4); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +69,22 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 4); \
INIT_VEC_LOOP (my, y, 4); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (4); \
TEST_VEC_LOOP (mr, 4); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 3 argument function to vector one.
#define VECTOR_WRAPPER_fFF(scalar_func, vector_func) \
extern void vector_func (VEC_TYPE, VEC_TYPE *, VEC_TYPE *); \
void scalar_func (FLOAT x, FLOAT * r, FLOAT * r1) \
{ \
int i; \
VEC_TYPE mx, mr, mr1; \
INIT_VEC_LOOP (mx, x, 4); \
vector_func (mx, &mr, &mr1); \
TEST_VEC_LOOP (mr, 4); \
TEST_VEC_LOOP (mr1, 4); \
*r = (FLOAT) mr[0]; \
*r1 = (FLOAT) mr1[0]; \
return; \
}

23
math/test-double-vlen8.h
View file

@ -44,6 +44,7 @@
#define WRAPPER_DECL(function) extern FLOAT function (FLOAT);
#define WRAPPER_DECL_ff(function) extern FLOAT function (FLOAT, FLOAT);
#define WRAPPER_DECL_fFF(function) extern void function (FLOAT, FLOAT *, FLOAT *);
// Wrapper from scalar to vector function with vector length 8.
#define VECTOR_WRAPPER(scalar_func, vector_func) \
@ -54,7 +55,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 8); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (8); \
TEST_VEC_LOOP (mr, 8); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +69,22 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 8); \
INIT_VEC_LOOP (my, y, 8); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (8); \
TEST_VEC_LOOP (mr, 8); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 3 argument function to vector one.
#define VECTOR_WRAPPER_fFF(scalar_func, vector_func) \
extern void vector_func (VEC_TYPE, VEC_TYPE *, VEC_TYPE *); \
void scalar_func (FLOAT x, FLOAT * r, FLOAT * r1) \
{ \
int i; \
VEC_TYPE mx, mr, mr1; \
INIT_VEC_LOOP (mx, x, 8); \
vector_func (mx, &mr, &mr1); \
TEST_VEC_LOOP (mr, 8); \
TEST_VEC_LOOP (mr1, 8); \
*r = (FLOAT) mr[0]; \
*r1 = (FLOAT) mr1[0]; \
return; \
}

6
math/test-float-vlen16.h
View file

@ -54,7 +54,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 16); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (16); \
TEST_VEC_LOOP (mr, 16); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +68,6 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 16); \
INIT_VEC_LOOP (my, y, 16); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (16); \
TEST_VEC_LOOP (mr, 16); \
return ((FLOAT) mr[0]); \
}

6
math/test-float-vlen4.h
View file

@ -54,7 +54,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 4); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (4); \
TEST_VEC_LOOP (mr, 4); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +68,6 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 4); \
INIT_VEC_LOOP (my, y, 4); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (4); \
TEST_VEC_LOOP (mr, 4); \
return ((FLOAT) mr[0]); \
}

6
math/test-float-vlen8.h
View file

@ -54,7 +54,8 @@ FLOAT scalar_func (FLOAT x) \
VEC_TYPE mx; \
INIT_VEC_LOOP (mx, x, 8); \
VEC_TYPE mr = vector_func (mx); \
TEST_VEC_LOOP (8); \
TEST_VEC_LOOP (mr, 8); \
return ((FLOAT) mr[0]); \
}
// Wrapper from scalar 2 argument function to vector one.
@ -67,5 +68,6 @@ FLOAT scalar_func (FLOAT x, FLOAT y) \
INIT_VEC_LOOP (mx, x, 8); \
INIT_VEC_LOOP (my, y, 8); \
VEC_TYPE mr = vector_func (mx, my); \
TEST_VEC_LOOP (8); \
TEST_VEC_LOOP (mr, 8); \
return ((FLOAT) mr[0]); \
}

8
math/test-vec-loop.h
View file

@ -17,17 +17,17 @@
<http://www.gnu.org/licenses/>. */
/* This macro is used in VECTOR_WRAPPER macros for vector tests. */
#define TEST_VEC_LOOP(len) \
#define TEST_VEC_LOOP(vec, len) \
do \
{ \
for (i = 1; i < len; i++) \
{ \
if ((FLOAT) mr[0] != (FLOAT) mr[i]) \
if ((FLOAT) vec[0] != (FLOAT) vec[i]) \
{ \
return ((FLOAT) mr[0] + 0.1); \
vec[0] = (FLOAT) vec[0] + 0.1; \
break; \
} \
} \
return ((FLOAT) mr[0]); \
} \
while (0)

4
sysdeps/unix/sysv/linux/x86_64/libmvec.abilist
View file

@ -5,6 +5,7 @@ GLIBC_2.22
_ZGVbN2v_log F
_ZGVbN2v_sin F
_ZGVbN2vv_pow F
_ZGVbN2vvv_sincos F
_ZGVbN4v_cosf F
_ZGVbN4v_expf F
_ZGVbN4v_logf F
@ -15,6 +16,7 @@ GLIBC_2.22
_ZGVcN4v_log F
_ZGVcN4v_sin F
_ZGVcN4vv_pow F
_ZGVcN4vvv_sincos F
_ZGVcN8v_cosf F
_ZGVcN8v_expf F
_ZGVcN8v_logf F
@ -25,6 +27,7 @@ GLIBC_2.22
_ZGVdN4v_log F
_ZGVdN4v_sin F
_ZGVdN4vv_pow F
_ZGVdN4vvv_sincos F
_ZGVdN8v_cosf F
_ZGVdN8v_expf F
_ZGVdN8v_logf F
@ -40,3 +43,4 @@ GLIBC_2.22
_ZGVeN8v_log F
_ZGVeN8v_sin F
_ZGVeN8vv_pow F
_ZGVeN8vvv_sincos F

2
sysdeps/x86/fpu/bits/math-vector.h
View file

@ -36,6 +36,8 @@
# define __DECL_SIMD_sin __DECL_SIMD_x86_64
# undef __DECL_SIMD_sinf
# define __DECL_SIMD_sinf __DECL_SIMD_x86_64
# undef __DECL_SIMD_sincos
# define __DECL_SIMD_sincos __DECL_SIMD_x86_64
# undef __DECL_SIMD_log
# define __DECL_SIMD_log __DECL_SIMD_x86_64
# undef __DECL_SIMD_logf

2
sysdeps/x86_64/fpu/Makefile
View file

@ -7,6 +7,8 @@ libmvec-support += svml_d_cos2_core svml_d_cos4_core_avx \
svml_s_cosf8_core svml_s_cosf16_core svml_s_cosf_data \
svml_s_sinf4_core svml_s_sinf8_core_avx \
svml_s_sinf8_core svml_s_sinf16_core svml_s_sinf_data \
svml_d_sincos2_core svml_d_sincos4_core_avx \
svml_d_sincos4_core svml_d_sincos8_core svml_d_sincos_data \
svml_d_log2_core svml_d_log4_core_avx svml_d_log4_core \
svml_d_log8_core svml_d_log_data svml_s_logf4_core \
svml_s_logf8_core_avx svml_s_logf8_core svml_s_logf16_core \

1
sysdeps/x86_64/fpu/Versions
View file

@ -2,6 +2,7 @@ libmvec {
GLIBC_2.22 {
_ZGVbN2v_cos; _ZGVcN4v_cos; _ZGVdN4v_cos; _ZGVeN8v_cos;
_ZGVbN2v_sin; _ZGVcN4v_sin; _ZGVdN4v_sin; _ZGVeN8v_sin;
_ZGVbN2vvv_sincos; _ZGVcN4vvv_sincos; _ZGVdN4vvv_sincos; _ZGVeN8vvv_sincos;
_ZGVbN2v_log; _ZGVcN4v_log; _ZGVdN4v_log; _ZGVeN8v_log;
_ZGVbN2v_exp; _ZGVcN4v_exp; _ZGVdN4v_exp; _ZGVeN8v_exp;
_ZGVbN2vv_pow; _ZGVcN4vv_pow; _ZGVdN4vv_pow; _ZGVeN8vv_pow;

12
sysdeps/x86_64/fpu/libm-test-ulps
View file

@ -2031,6 +2031,18 @@ idouble: 1
ildouble: 3
ldouble: 3
Function: "sincos_vlen2":
double: 1
Function: "sincos_vlen4":
double: 1
Function: "sincos_vlen4_avx2":
double: 1
Function: "sincos_vlen8":
double: 1
Function: "sinh":
double: 2
float: 2

3
sysdeps/x86_64/fpu/multiarch/Makefile
View file

@ -57,7 +57,8 @@ libmvec-sysdep_routines += svml_d_cos2_core_sse4 svml_d_cos4_core_avx2 \
svml_d_cos8_core_avx512 svml_d_sin2_core_sse4 \
svml_d_sin4_core_avx2 svml_d_sin8_core_avx512 \
svml_d_log2_core_sse4 svml_d_log4_core_avx2 \
svml_d_log8_core_avx512 \
svml_d_log8_core_avx512 svml_d_sincos2_core_sse4 \
svml_d_sincos4_core_avx2 svml_d_sincos8_core_avx512 \
svml_s_cosf4_core_sse4 svml_s_cosf8_core_avx2 \
svml_s_cosf16_core_avx512 svml_s_sinf4_core_sse4 \
svml_s_sinf8_core_avx2 svml_s_sinf16_core_avx512 \

38
sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core.S Normal file
View file

@ -0,0 +1,38 @@
/* Multiple versions of vectorized sincos.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include <init-arch.h>
.text
ENTRY (_ZGVbN2vvv_sincos)
.type _ZGVbN2vvv_sincos, @gnu_indirect_function
cmpl $0, KIND_OFFSET+__cpu_features(%rip)
jne 1f
call __init_cpu_features
1: leaq _ZGVbN2vvv_sincos_sse4(%rip), %rax
testl $bit_SSE4_1, __cpu_features+CPUID_OFFSET+index_SSE4_1(%rip)
jz 2f
ret
2: leaq _ZGVbN2vvv_sincos_sse2(%rip), %rax
ret
END (_ZGVbN2vvv_sincos)
libmvec_hidden_def (_ZGVbN2vvv_sincos)
#define _ZGVbN2vvv_sincos _ZGVbN2vvv_sincos_sse2
#include "../svml_d_sincos2_core.S"

314
sysdeps/x86_64/fpu/multiarch/svml_d_sincos2_core_sse4.S Normal file
View file

@ -0,0 +1,314 @@
/* Function sincos vectorized with SSE4.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_sincos_data.h"
.text
ENTRY (_ZGVbN2vvv_sincos_sse4)
/*
ALGORITHM DESCRIPTION:
( low accuracy ( < 4ulp ) or enhanced performance
( half of correct mantissa ) implementation )
Argument representation:
arg = N*Pi + R
Result calculation:
sin(arg) = sin(N*Pi + R) = (-1)^N * sin(R)
arg + Pi/2 = (N'*Pi + R')
cos(arg) = sin(arg+Pi/2) = sin(N'*Pi + R') = (-1)^N' * sin(R')
sin(R), sin(R') are approximated by corresponding polynomial. */
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-64, %rsp
subq $320, %rsp
movq __svml_dsincos_data@GOTPCREL(%rip), %rax
movups %xmm11, 160(%rsp)
movups %xmm12, 144(%rsp)
movups __dSignMask(%rax), %xmm11
/* ARGUMENT RANGE REDUCTION:
Absolute argument: X' = |X| */
movaps %xmm11, %xmm4
/* Grab sign bit from argument */
movaps %xmm11, %xmm7
movups __dInvPI(%rax), %xmm5
andnps %xmm0, %xmm4
/* SinY = X'*InvPi + RS : right shifter add */
mulpd %xmm4, %xmm5
addpd __dRShifter(%rax), %xmm5
/* SinSignRes = Y<<63 : shift LSB to MSB place for result sign */
movaps %xmm5, %xmm12
andps %xmm0, %xmm7
/* SinN = Y - RS : right shifter sub */
subpd __dRShifter(%rax), %xmm5
movups %xmm10, 176(%rsp)
psllq $63, %xmm12
movups __dPI1(%rax), %xmm10
/* SinR = X' - SinN*Pi1 */
movaps %xmm10, %xmm1
mulpd %xmm5, %xmm1
movups __dPI2(%rax), %xmm6
/* SinR = SinR - SinN*Pi1 */
movaps %xmm6, %xmm2
mulpd %xmm5, %xmm2
movups %xmm13, 112(%rsp)
movaps %xmm4, %xmm13
subpd %xmm1, %xmm13
subpd %xmm2, %xmm13
/* Sine result sign: SinRSign = SignMask & SinR */
movaps %xmm11, %xmm2
/* CosR = SinX - CosN*Pi1 */
movaps %xmm4, %xmm1
movups __dOneHalf(%rax), %xmm3
andps %xmm13, %xmm2
/* Set SinRSign to 0.5 */
orps %xmm2, %xmm3
/* Update CosRSign and CosSignRes signs */
xorps %xmm11, %xmm2
/* CosN = SinN +(-)0.5 */
addpd %xmm5, %xmm3
cmpnlepd __dRangeVal(%rax), %xmm4
mulpd %xmm3, %xmm10
/* CosR = CosR - CosN*Pi2 */
mulpd %xmm3, %xmm6
subpd %xmm10, %xmm1
movmskpd %xmm4, %ecx
movups __dPI3(%rax), %xmm10
xorps %xmm12, %xmm2
subpd %xmm6, %xmm1
/* SinR = SinR - SinN*Pi3 */
movaps %xmm10, %xmm6
/* Final reconstruction.
Combine Sin result's sign */
xorps %xmm7, %xmm12
mulpd %xmm5, %xmm6
/* CosR = CosR - CosN*Pi3 */
mulpd %xmm3, %xmm10
subpd %xmm6, %xmm13
subpd %xmm10, %xmm1
movups __dPI4(%rax), %xmm6
/* SinR = SinR - SinN*Pi4 */
mulpd %xmm6, %xmm5
/* CosR = CosR - CosN*Pi4 */
mulpd %xmm6, %xmm3
subpd %xmm5, %xmm13
subpd %xmm3, %xmm1
/* SinR2 = SinR^2 */
movaps %xmm13, %xmm6
/* CosR2 = CosR^2 */
movaps %xmm1, %xmm10
mulpd %xmm13, %xmm6
mulpd %xmm1, %xmm10
/* Polynomial approximation */
movups __dC7(%rax), %xmm5
movaps %xmm5, %xmm3
mulpd %xmm6, %xmm3
mulpd %xmm10, %xmm5
addpd __dC6(%rax), %xmm3
addpd __dC6(%rax), %xmm5
mulpd %xmm6, %xmm3
mulpd %xmm10, %xmm5
addpd __dC5(%rax), %xmm3
addpd __dC5(%rax), %xmm5
mulpd %xmm6, %xmm3
mulpd %xmm10, %xmm5
addpd __dC4(%rax), %xmm3
addpd __dC4(%rax), %xmm5
/* SinPoly = C3 + SinR2*(C4 + SinR2*(C5 + SinR2*(C6 + SinR2*C7))) */
mulpd %xmm6, %xmm3
/* CosPoly = C3 + CosR2*(C4 + CosR2*(C5 + CosR2*(C6 + CosR2*C7))) */
mulpd %xmm10, %xmm5
addpd __dC3(%rax), %xmm3
addpd __dC3(%rax), %xmm5
/* SinPoly = C2 + SinR2*SinPoly */
mulpd %xmm6, %xmm3
/* CosPoly = C2 + CosR2*CosPoly */
mulpd %xmm10, %xmm5
addpd __dC2(%rax), %xmm3
addpd __dC2(%rax), %xmm5
/* SinPoly = C1 + SinR2*SinPoly */
mulpd %xmm6, %xmm3
/* CosPoly = C1 + CosR2*CosPoly */
mulpd %xmm10, %xmm5
addpd __dC1(%rax), %xmm3
addpd __dC1(%rax), %xmm5
/* SinPoly = SinR2*SinPoly */
mulpd %xmm3, %xmm6
/* CosPoly = CosR2*CosPoly */
mulpd %xmm5, %xmm10
/* SinPoly = SinR*SinPoly */
mulpd %xmm13, %xmm6
/* CosPoly = CosR*CosPoly */
mulpd %xmm1, %xmm10
addpd %xmm6, %xmm13
addpd %xmm10, %xmm1
/* Update Sin result's sign */
xorps %xmm12, %xmm13
/* Update Cos result's sign */
xorps %xmm2, %xmm1
testl %ecx, %ecx
jne .LBL_1_3
.LBL_1_2:
cfi_remember_state
movups 176(%rsp), %xmm10
movaps %xmm13, (%rdi)
movups 160(%rsp), %xmm11
movups 144(%rsp), %xmm12
movups 112(%rsp), %xmm13
movups %xmm1, (%rsi)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.LBL_1_3:
cfi_restore_state
movups %xmm0, 128(%rsp)
movups %xmm13, 192(%rsp)
movups %xmm1, 256(%rsp)
je .LBL_1_2
xorb %dl, %dl
xorl %eax, %eax
movups %xmm8, 48(%rsp)
movups %xmm9, 32(%rsp)
movups %xmm14, 16(%rsp)
movups %xmm15, (%rsp)
movq %rsi, 64(%rsp)
movq %r12, 104(%rsp)
cfi_offset_rel_rsp (12, 104)
movb %dl, %r12b
movq %r13, 96(%rsp)
cfi_offset_rel_rsp (13, 96)
movl %eax, %r13d
movq %r14, 88(%rsp)
cfi_offset_rel_rsp (14, 88)
movl %ecx, %r14d
movq %r15, 80(%rsp)
cfi_offset_rel_rsp (15, 80)
movq %rbx, 72(%rsp)
movq %rdi, %rbx
cfi_remember_state
.LBL_1_6:
btl %r13d, %r14d
jc .LBL_1_13
.LBL_1_7:
lea 1(%r13), %esi
btl %esi, %r14d
jc .LBL_1_10
.LBL_1_8:
incb %r12b
addl $2, %r13d
cmpb $16, %r12b
jb .LBL_1_6
movups 48(%rsp), %xmm8
movq %rbx, %rdi
movups 32(%rsp), %xmm9
movups 16(%rsp), %xmm14
movups (%rsp), %xmm15
movq 64(%rsp), %rsi
movq 104(%rsp), %r12
cfi_restore (%r12)
movq 96(%rsp), %r13
cfi_restore (%r13)
movq 88(%rsp), %r14
cfi_restore (%r14)
movq 80(%rsp), %r15
cfi_restore (%r15)
movq 72(%rsp), %rbx
movups 192(%rsp), %xmm13
movups 256(%rsp), %xmm1
jmp .LBL_1_2
.LBL_1_10:
cfi_restore_state
movzbl %r12b, %r15d
shlq $4, %r15
movsd 136(%rsp,%r15), %xmm0
call sin@PLT
movsd %xmm0, 200(%rsp,%r15)
movsd 136(%rsp,%r15), %xmm0
call cos@PLT
movsd %xmm0, 264(%rsp,%r15)
jmp .LBL_1_8
.LBL_1_13:
movzbl %r12b, %r15d
shlq $4, %r15
movsd 128(%rsp,%r15), %xmm0
call sin@PLT
movsd %xmm0, 192(%rsp,%r15)
movsd 128(%rsp,%r15), %xmm0
call cos@PLT
movsd %xmm0, 256(%rsp,%r15)
jmp .LBL_1_7
END (_ZGVbN2vvv_sincos_sse4)

38
sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core.S Normal file
View file

@ -0,0 +1,38 @@
/* Multiple versions of vectorized sincos.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include <init-arch.h>
.text
ENTRY (_ZGVdN4vvv_sincos)
.type _ZGVdN4vvv_sincos, @gnu_indirect_function
cmpl $0, KIND_OFFSET+__cpu_features(%rip)
jne 1f
call __init_cpu_features
1: leaq _ZGVdN4vvv_sincos_avx2(%rip), %rax
testl $bit_AVX2_Usable, __cpu_features+FEATURE_OFFSET+index_AVX2_Usable(%rip)
jz 2f
ret
2: leaq _ZGVdN4vvv_sincos_sse_wrapper(%rip), %rax
ret
END (_ZGVdN4vvv_sincos)
libmvec_hidden_def (_ZGVdN4vvv_sincos)
#define _ZGVdN4vvv_sincos _ZGVdN4vvv_sincos_sse_wrapper
#include "../svml_d_sincos4_core.S"

277
sysdeps/x86_64/fpu/multiarch/svml_d_sincos4_core_avx2.S Normal file
View file

@ -0,0 +1,277 @@
/* Function sincos vectorized with AVX2.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_sincos_data.h"
.text
ENTRY (_ZGVdN4vvv_sincos_avx2)
/*
ALGORITHM DESCRIPTION:
( low accuracy ( < 4ulp ) or enhanced performance
( half of correct mantissa ) implementation )
Argument representation:
arg = N*Pi + R
Result calculation:
sin(arg) = sin(N*Pi + R) = (-1)^N * sin(R)
arg + Pi/2 = (N'*Pi + R')
cos(arg) = sin(arg+Pi/2) = sin(N'*Pi + R') = (-1)^N' * sin(R')
sin(R), sin(R') are approximated by corresponding polynomial. */
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-64, %rsp
subq $448, %rsp
movq __svml_dsincos_data@GOTPCREL(%rip), %rax
vmovups %ymm14, 288(%rsp)
vmovups %ymm8, 352(%rsp)
vmovupd __dSignMask(%rax), %ymm6
vmovupd __dInvPI(%rax), %ymm2
vmovupd __dPI1_FMA(%rax), %ymm5
vmovups %ymm9, 224(%rsp)
/* ARGUMENT RANGE REDUCTION:
Absolute argument: X' = |X| */
vandnpd %ymm0, %ymm6, %ymm1
/* SinY = X'*InvPi + RS : right shifter add */
vfmadd213pd __dRShifter(%rax), %ymm1, %ymm2
/* SinSignRes = Y<<63 : shift LSB to MSB place for result sign */
vpsllq $63, %ymm2, %ymm4
/* SinN = Y - RS : right shifter sub */
vsubpd __dRShifter(%rax), %ymm2, %ymm2
/* SinR = X' - SinN*Pi1 */
vmovdqa %ymm1, %ymm14
vfnmadd231pd %ymm2, %ymm5, %ymm14
/* SinR = SinR - SinN*Pi1 */
vfnmadd231pd __dPI2_FMA(%rax), %ymm2, %ymm14
/* Sine result sign: SinRSign = SignMask & SinR */
vandpd %ymm14, %ymm6, %ymm7
/* Set SinRSign to 0.5 */
vorpd __dOneHalf(%rax), %ymm7, %ymm3
/* CosN = SinN +(-)0.5 */
vaddpd %ymm3, %ymm2, %ymm3
/* CosR = SinX - CosN*Pi1 */
vmovdqa %ymm1, %ymm8
vfnmadd231pd %ymm3, %ymm5, %ymm8
vmovupd __dPI3_FMA(%rax), %ymm5
vcmpnle_uqpd __dRangeVal(%rax), %ymm1, %ymm1
/* CosR = CosR - CosN*Pi2 */
vfnmadd231pd __dPI2_FMA(%rax), %ymm3, %ymm8
/* SinR = SinR - SinN*Pi3 */
vfnmadd213pd %ymm14, %ymm5, %ymm2
/* CosR = CosR - CosN*Pi3 */
vfnmadd213pd %ymm8, %ymm5, %ymm3
vmovupd __dC6(%rax), %ymm8
/* SinR2 = SinR^2 */
vmulpd %ymm2, %ymm2, %ymm14
/* CosR2 = CosR^2 */
vmulpd %ymm3, %ymm3, %ymm5
/* Grab SignX */
vandpd %ymm0, %ymm6, %ymm9
/* Update CosRSign and CosSignRes signs */
vxorpd %ymm6, %ymm7, %ymm6
vxorpd %ymm6, %ymm4, %ymm7
/* Update sign SinSignRes */
vxorpd %ymm9, %ymm4, %ymm6
/* Polynomial approximation */
vmovupd __dC7(%rax), %ymm4
vmovdqa %ymm8, %ymm9
vfmadd231pd __dC7(%rax), %ymm14, %ymm9
vfmadd213pd %ymm8, %ymm5, %ymm4
vfmadd213pd __dC5(%rax), %ymm14, %ymm9
vfmadd213pd __dC5(%rax), %ymm5, %ymm4
vfmadd213pd __dC4(%rax), %ymm14, %ymm9
vfmadd213pd __dC4(%rax), %ymm5, %ymm4
/* SinPoly = C3 + SinR2*(C4 + SinR2*(C5 + SinR2*(C6 + SinR2*C7))) */
vfmadd213pd __dC3(%rax), %ymm14, %ymm9
/* CosPoly = C3 + CosR2*(C4 + CosR2*(C5 + CosR2*(C6 + CosR2*C7))) */
vfmadd213pd __dC3(%rax), %ymm5, %ymm4
/* SinPoly = C2 + SinR2*SinPoly */
vfmadd213pd __dC2(%rax), %ymm14, %ymm9
/* CosPoly = C2 + CosR2*CosPoly */
vfmadd213pd __dC2(%rax), %ymm5, %ymm4
/* SinPoly = C1 + SinR2*SinPoly */
vfmadd213pd __dC1(%rax), %ymm14, %ymm9
/* CosPoly = C1 + CosR2*CosPoly */
vfmadd213pd __dC1(%rax), %ymm5, %ymm4
/* SinPoly = SinR2*SinPoly */
vmulpd %ymm14, %ymm9, %ymm8
/* CosPoly = CosR2*CosPoly */
vmulpd %ymm5, %ymm4, %ymm4
/* SinPoly = SinR*SinPoly */
vfmadd213pd %ymm2, %ymm2, %ymm8
/* CosPoly = CosR*CosPoly */
vfmadd213pd %ymm3, %ymm3, %ymm4
vmovmskpd %ymm1, %ecx
/* Final reconstruction
Update Sin result's sign */
vxorpd %ymm6, %ymm8, %ymm3
/* Update Cos result's sign */
vxorpd %ymm7, %ymm4, %ymm2
testl %ecx, %ecx
jne .LBL_1_3
.LBL_1_2:
cfi_remember_state
vmovups 352(%rsp), %ymm8
vmovups 224(%rsp), %ymm9
vmovups 288(%rsp), %ymm14
vmovupd %ymm2, (%rsi)
vmovdqa %ymm3, (%rdi)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.LBL_1_3:
cfi_restore_state
vmovupd %ymm0, 256(%rsp)
vmovupd %ymm3, 320(%rsp)
vmovupd %ymm2, 384(%rsp)
je .LBL_1_2
xorb %dl, %dl
xorl %eax, %eax
vmovups %ymm10, 128(%rsp)
vmovups %ymm11, 96(%rsp)
vmovups %ymm12, 64(%rsp)
vmovups %ymm13, 32(%rsp)
vmovups %ymm15, (%rsp)
movq %rsi, 160(%rsp)
movq %r12, 200(%rsp)
cfi_offset_rel_rsp (12, 200)
movb %dl, %r12b
movq %r13, 192(%rsp)
cfi_offset_rel_rsp (13, 192)
movl %eax, %r13d
movq %r14, 184(%rsp)
cfi_offset_rel_rsp (14, 184)
movl %ecx, %r14d
movq %r15, 176(%rsp)
cfi_offset_rel_rsp (15, 176)
movq %rbx, 168(%rsp)
movq %rdi, %rbx
cfi_remember_state
.LBL_1_6:
btl %r13d, %r14d
jc .LBL_1_13
.LBL_1_7:
lea 1(%r13), %esi
btl %esi, %r14d
jc .LBL_1_10
.LBL_1_8:
incb %r12b
addl $2, %r13d
cmpb $16, %r12b
jb .LBL_1_6
vmovups 128(%rsp), %ymm10
movq %rbx, %rdi
vmovups 96(%rsp), %ymm11
vmovups 64(%rsp), %ymm12
vmovups 32(%rsp), %ymm13
vmovups (%rsp), %ymm15
vmovupd 320(%rsp), %ymm3
vmovupd 384(%rsp), %ymm2
movq 160(%rsp), %rsi
movq 200(%rsp), %r12
cfi_restore (%r12)
movq 192(%rsp), %r13
cfi_restore (%r13)
movq 184(%rsp), %r14
cfi_restore (%r14)
movq 176(%rsp), %r15
cfi_restore (%r15)
movq 168(%rsp), %rbx
jmp .LBL_1_2
.LBL_1_10:
cfi_restore_state
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 264(%rsp,%r15), %xmm0
vzeroupper
call sin@PLT
vmovsd %xmm0, 328(%rsp,%r15)
vmovsd 264(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 392(%rsp,%r15)
jmp .LBL_1_8
.LBL_1_13:
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 256(%rsp,%r15), %xmm0
vzeroupper
call sin@PLT
vmovsd %xmm0, 320(%rsp,%r15)
vmovsd 256(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 384(%rsp,%r15)
jmp .LBL_1_7
END (_ZGVdN4vvv_sincos_avx2)

39
sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core.S Normal file
View file

@ -0,0 +1,39 @@
/* Multiple versions of vectorized sincos.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include <init-arch.h>
.text
ENTRY (_ZGVeN8vvv_sincos)
.type _ZGVeN8vvv_sincos, @gnu_indirect_function
cmpl $0, KIND_OFFSET+__cpu_features(%rip)
jne 1
call __init_cpu_features
1: leaq _ZGVeN8vvv_sincos_skx(%rip), %rax
testl $bit_AVX512DQ_Usable, __cpu_features+FEATURE_OFFSET+index_AVX512DQ_Usable(%rip)
jnz 3
2: leaq _ZGVeN8vvv_sincos_knl(%rip), %rax
testl $bit_AVX512F_Usable, __cpu_features+FEATURE_OFFSET+index_AVX512F_Usable(%rip)
jnz 3
leaq _ZGVeN8vvv_sincos_avx2_wrapper(%rip), %rax
3: ret
END (_ZGVeN8vvv_sincos)
#define _ZGVeN8vvv_sincos _ZGVeN8vvv_sincos_avx2_wrapper
#include "../svml_d_sincos8_core.S"

593
sysdeps/x86_64/fpu/multiarch/svml_d_sincos8_core_avx512.S Normal file
View file

@ -0,0 +1,593 @@
/* Function sincos vectorized with AVX-512. KNL and SKX versions.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_sincos_data.h"
#include "svml_d_wrapper_impl.h"
/*
ALGORITHM DESCRIPTION:
( low accuracy ( < 4ulp ) or enhanced performance
( half of correct mantissa ) implementation )
Argument representation:
arg = N*Pi + R
Result calculation:
sin(arg) = sin(N*Pi + R) = (-1)^N * sin(R)
arg + Pi/2 = (N'*Pi + R')
cos(arg) = sin(arg+Pi/2) = sin(N'*Pi + R') = (-1)^N' * sin(R')
sin(R), sin(R') are approximated by corresponding polynomial. */
.text
ENTRY (_ZGVeN8vvv_sincos_knl)
#ifndef HAVE_AVX512_ASM_SUPPORT
WRAPPER_IMPL_AVX512_fFF _ZGVdN4vvv_sincos
#else
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-64, %rsp
subq $1344, %rsp
movq __svml_dsincos_data@GOTPCREL(%rip), %rax
vmovaps %zmm0, %zmm4
movq $-1, %rdx
vmovups __dSignMask(%rax), %zmm12
vmovups __dInvPI(%rax), %zmm5
/* ARGUMENT RANGE REDUCTION:
Absolute argument: X' = |X| */
vpandnq %zmm4, %zmm12, %zmm3
vmovups __dPI1_FMA(%rax), %zmm7
vmovups __dPI3_FMA(%rax), %zmm9
/* SinR = X' - SinN*Pi1 */
vmovaps %zmm3, %zmm8
/* CosR = SinX - CosN*Pi1 */
vmovaps %zmm3, %zmm10
/* SinY = X'*InvPi + RS : right shifter add */
vfmadd213pd __dRShifter(%rax), %zmm3, %zmm5
vmovups __dC6(%rax), %zmm13
/* SinN = Y - RS : right shifter sub */
vsubpd __dRShifter(%rax), %zmm5, %zmm1
vmovaps %zmm13, %zmm14
/* SinSignRes = Y<<63 : shift LSB to MSB place for result sign */
vpsllq $63, %zmm5, %zmm2
vcmppd $22, __dRangeVal(%rax), %zmm3, %k1
/* Update CosRSign and CosSignRes signs */
vmovaps %zmm12, %zmm5
vfnmadd231pd %zmm1, %zmm7, %zmm8
/* SinR = SinR - SinN*Pi1 */
vfnmadd231pd __dPI2_FMA(%rax), %zmm1, %zmm8
/* Sine result sign: SinRSign = SignMask & SinR */
vpandq %zmm8, %zmm12, %zmm11
/* Set SinRSign to 0.5 */
vporq __dOneHalf(%rax), %zmm11, %zmm6
vpternlogq $150, %zmm2, %zmm11, %zmm5
/* Update sign SinSignRes */
vpternlogq $120, %zmm4, %zmm12, %zmm2
/* Polynomial approximation */
vmovups __dC7(%rax), %zmm11
/* CosN = SinN +(-)0.5 */
vaddpd %zmm6, %zmm1, %zmm0
/* SinR = SinR - SinN*Pi3 */
vfnmadd213pd %zmm8, %zmm9, %zmm1
vfnmadd231pd %zmm0, %zmm7, %zmm10
/* SinR2 = SinR^2 */
vmulpd %zmm1, %zmm1, %zmm15
/* Grab SignX
CosR = CosR - CosN*Pi2 */
vfnmadd231pd __dPI2_FMA(%rax), %zmm0, %zmm10
vfmadd231pd __dC7(%rax), %zmm15, %zmm14
/* CosR = CosR - CosN*Pi3 */
vfnmadd213pd %zmm10, %zmm9, %zmm0
vfmadd213pd __dC5(%rax), %zmm15, %zmm14
/* CosR2 = CosR^2 */
vmulpd %zmm0, %zmm0, %zmm12
vfmadd213pd __dC4(%rax), %zmm15, %zmm14
vfmadd213pd %zmm13, %zmm12, %zmm11
/* SinPoly = C3 + SinR2*(C4 + SinR2*(C5 + SinR2*(C6 + SinR2*C7))) */
vfmadd213pd __dC3(%rax), %zmm15, %zmm14
vfmadd213pd __dC5(%rax), %zmm12, %zmm11
/* SinPoly = C2 + SinR2*SinPoly */
vfmadd213pd __dC2(%rax), %zmm15, %zmm14
vfmadd213pd __dC4(%rax), %zmm12, %zmm11
/* SinPoly = C1 + SinR2*SinPoly */
vfmadd213pd __dC1(%rax), %zmm15, %zmm14
/* CosPoly = C3 + CosR2*(C4 + CosR2*(C5 + CosR2*(C6 + CosR2*C7))) */
vfmadd213pd __dC3(%rax), %zmm12, %zmm11
/* SinPoly = SinR2*SinPoly */
vmulpd %zmm15, %zmm14, %zmm13
/* CosPoly = C2 + CosR2*CosPoly */
vfmadd213pd __dC2(%rax), %zmm12, %zmm11
/* SinPoly = SinR*SinPoly */
vfmadd213pd %zmm1, %zmm1, %zmm13
vpbroadcastq %rdx, %zmm1{%k1}{z}
/* CosPoly = C1 + CosR2*CosPoly */
vfmadd213pd __dC1(%rax), %zmm12, %zmm11
vptestmq %zmm1, %zmm1, %k0
kmovw %k0, %ecx
/* CosPoly = CosR2*CosPoly */
vmulpd %zmm12, %zmm11, %zmm14
movzbl %cl, %ecx
/* CosPoly = CosR*CosPoly */
vfmadd213pd %zmm0, %zmm0, %zmm14
/* Final reconstruction.
Update Sin result's sign */
vpxorq %zmm2, %zmm13, %zmm0
/* Update Cos result's sign */
vpxorq %zmm5, %zmm14, %zmm2
testl %ecx, %ecx
jne .LBL_1_3
.LBL_1_2:
cfi_remember_state
vmovups %zmm0, (%rdi)
vmovups %zmm2, (%rsi)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.LBL_1_3:
cfi_restore_state
vmovups %zmm4, 1152(%rsp)
vmovups %zmm0, 1216(%rsp)
vmovups %zmm2, 1280(%rsp)
je .LBL_1_2
xorb %dl, %dl
kmovw %k4, 1048(%rsp)
xorl %eax, %eax
kmovw %k5, 1040(%rsp)
kmovw %k6, 1032(%rsp)
kmovw %k7, 1024(%rsp)
vmovups %zmm16, 960(%rsp)
vmovups %zmm17, 896(%rsp)
vmovups %zmm18, 832(%rsp)
vmovups %zmm19, 768(%rsp)
vmovups %zmm20, 704(%rsp)
vmovups %zmm21, 640(%rsp)
vmovups %zmm22, 576(%rsp)
vmovups %zmm23, 512(%rsp)
vmovups %zmm24, 448(%rsp)
vmovups %zmm25, 384(%rsp)
vmovups %zmm26, 320(%rsp)
vmovups %zmm27, 256(%rsp)
vmovups %zmm28, 192(%rsp)
vmovups %zmm29, 128(%rsp)
vmovups %zmm30, 64(%rsp)
vmovups %zmm31, (%rsp)
movq %rsi, 1056(%rsp)
movq %r12, 1096(%rsp)
cfi_offset_rel_rsp (12, 1096)
movb %dl, %r12b
movq %r13, 1088(%rsp)
cfi_offset_rel_rsp (13, 1088)
movl %eax, %r13d
movq %r14, 1080(%rsp)
cfi_offset_rel_rsp (14, 1080)
movl %ecx, %r14d
movq %r15, 1072(%rsp)
cfi_offset_rel_rsp (15, 1072)
movq %rbx, 1064(%rsp)
movq %rdi, %rbx
cfi_remember_state
.LBL_1_6:
btl %r13d, %r14d
jc .LBL_1_13
.LBL_1_7:
lea 1(%r13), %esi
btl %esi, %r14d
jc .LBL_1_10
.LBL_1_8:
addb $1, %r12b
addl $2, %r13d
cmpb $16, %r12b
jb .LBL_1_6
movq %rbx, %rdi
kmovw 1048(%rsp), %k4
movq 1056(%rsp), %rsi
kmovw 1040(%rsp), %k5
movq 1096(%rsp), %r12
cfi_restore (%r12)
kmovw 1032(%rsp), %k6
movq 1088(%rsp), %r13
cfi_restore (%r13)
kmovw 1024(%rsp), %k7
vmovups 960(%rsp), %zmm16
vmovups 896(%rsp), %zmm17
vmovups 832(%rsp), %zmm18
vmovups 768(%rsp), %zmm19
vmovups 704(%rsp), %zmm20
vmovups 640(%rsp), %zmm21
vmovups 576(%rsp), %zmm22
vmovups 512(%rsp), %zmm23
vmovups 448(%rsp), %zmm24
vmovups 384(%rsp), %zmm25
vmovups 320(%rsp), %zmm26
vmovups 256(%rsp), %zmm27
vmovups 192(%rsp), %zmm28
vmovups 128(%rsp), %zmm29
vmovups 64(%rsp), %zmm30
vmovups (%rsp), %zmm31
movq 1080(%rsp), %r14
cfi_restore (%r14)
movq 1072(%rsp), %r15
cfi_restore (%r15)
movq 1064(%rsp), %rbx
vmovups 1216(%rsp), %zmm0
vmovups 1280(%rsp), %zmm2
jmp .LBL_1_2
.LBL_1_10:
cfi_restore_state
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 1160(%rsp,%r15), %xmm0
call sin@PLT
vmovsd %xmm0, 1224(%rsp,%r15)
vmovsd 1160(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 1288(%rsp,%r15)
jmp .LBL_1_8
.LBL_1_13:
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 1152(%rsp,%r15), %xmm0
call sin@PLT
vmovsd %xmm0, 1216(%rsp,%r15)
vmovsd 1152(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 1280(%rsp,%r15)
jmp .LBL_1_7
#endif
END (_ZGVeN8vvv_sincos_knl)
ENTRY (_ZGVeN8vvv_sincos_skx)
#ifndef HAVE_AVX512_ASM_SUPPORT
WRAPPER_IMPL_AVX512_fFF _ZGVdN4vvv_sincos
#else
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-64, %rsp
subq $1344, %rsp
movq __svml_dsincos_data@GOTPCREL(%rip), %rax
vmovaps %zmm0, %zmm8
vmovups __dSignMask(%rax), %zmm4
vmovups __dInvPI(%rax), %zmm9
vmovups __dRShifter(%rax), %zmm10
vmovups __dPI1_FMA(%rax), %zmm13
vmovups __dPI2_FMA(%rax), %zmm14
vmovups __dOneHalf(%rax), %zmm11
vmovups __dPI3_FMA(%rax), %zmm2
/* ARGUMENT RANGE REDUCTION:
Absolute argument: X' = |X| */
vandnpd %zmm8, %zmm4, %zmm7
/* SinY = X'*InvPi + RS : right shifter add */
vfmadd213pd %zmm10, %zmm7, %zmm9
vcmppd $18, __dRangeVal(%rax), %zmm7, %k1
/* SinSignRes = Y<<63 : shift LSB to MSB place for result sign */
vpsllq $63, %zmm9, %zmm6
/* SinN = Y - RS : right shifter sub */
vsubpd %zmm10, %zmm9, %zmm5
vmovups __dC5(%rax), %zmm9
vmovups __dC4(%rax), %zmm10
/* SinR = X' - SinN*Pi1 */
vmovaps %zmm7, %zmm15
vfnmadd231pd %zmm5, %zmm13, %zmm15
/* SinR = SinR - SinN*Pi1 */
vfnmadd231pd %zmm5, %zmm14, %zmm15
/* Sine result sign: SinRSign = SignMask & SinR */
vandpd %zmm15, %zmm4, %zmm1
/* Set SinRSign to 0.5 */
vorpd %zmm1, %zmm11, %zmm12
vmovups __dC3(%rax), %zmm11
/* CosN = SinN +(-)0.5 */
vaddpd %zmm12, %zmm5, %zmm3
/* SinR = SinR - SinN*Pi3 */
vfnmadd213pd %zmm15, %zmm2, %zmm5
vmovups __dC2(%rax), %zmm12
/* SinR2 = SinR^2 */
vmulpd %zmm5, %zmm5, %zmm15
/* CosR = SinX - CosN*Pi1 */
vmovaps %zmm7, %zmm0
vfnmadd231pd %zmm3, %zmm13, %zmm0
vmovups __dC1(%rax), %zmm13
/* Grab SignX
CosR = CosR - CosN*Pi2 */
vfnmadd231pd %zmm3, %zmm14, %zmm0
/* CosR = CosR - CosN*Pi3 */
vfnmadd213pd %zmm0, %zmm2, %zmm3
/* Polynomial approximation */
vmovups __dC7(%rax), %zmm0
/* Update CosRSign and CosSignRes signs */
vmovaps %zmm4, %zmm2
vpternlogq $150, %zmm6, %zmm1, %zmm2
/* Update sign SinSignRes */
vpternlogq $120, %zmm8, %zmm4, %zmm6
/* CosR2 = CosR^2 */
vmulpd %zmm3, %zmm3, %zmm1
vmovups __dC6(%rax), %zmm4
vmovaps %zmm0, %zmm14
vfmadd213pd %zmm4, %zmm1, %zmm0
vfmadd213pd %zmm4, %zmm15, %zmm14
vfmadd213pd %zmm9, %zmm1, %zmm0
vfmadd213pd %zmm9, %zmm15, %zmm14
vfmadd213pd %zmm10, %zmm1, %zmm0
vfmadd213pd %zmm10, %zmm15, %zmm14
/* CosPoly = C3 + CosR2*(C4 + CosR2*(C5 + CosR2*(C6 + CosR2*C7))) */
vfmadd213pd %zmm11, %zmm1, %zmm0
/* SinPoly = C3 + SinR2*(C4 + SinR2*(C5 + SinR2*(C6 + SinR2*C7))) */
vfmadd213pd %zmm11, %zmm15, %zmm14
/* CosPoly = C2 + CosR2*CosPoly */
vfmadd213pd %zmm12, %zmm1, %zmm0
/* SinPoly = C2 + SinR2*SinPoly */
vfmadd213pd %zmm12, %zmm15, %zmm14
/* CosPoly = C1 + CosR2*CosPoly */
vfmadd213pd %zmm13, %zmm1, %zmm0
/* SinPoly = C1 + SinR2*SinPoly */
vfmadd213pd %zmm13, %zmm15, %zmm14
/* CosPoly = CosR2*CosPoly */
vmulpd %zmm1, %zmm0, %zmm1
/* SinPoly = SinR2*SinPoly */
vmulpd %zmm15, %zmm14, %zmm4
/* CosPoly = CosR*CosPoly */
vfmadd213pd %zmm3, %zmm3, %zmm1
/* SinPoly = SinR*SinPoly */
vfmadd213pd %zmm5, %zmm5, %zmm4
vpbroadcastq .L_2il0floatpacket.15(%rip), %zmm3
/* Update Cos result's sign */
vxorpd %zmm2, %zmm1, %zmm1
/* Final reconstruction.
Update Sin result's sign */
vxorpd %zmm6, %zmm4, %zmm0
vpandnq %zmm7, %zmm7, %zmm3{%k1}
vcmppd $3, %zmm3, %zmm3, %k0
kmovw %k0, %ecx
testl %ecx, %ecx
jne .LBL_2_3
.LBL_2_2:
cfi_remember_state
vmovups %zmm0, (%rdi)
vmovups %zmm1, (%rsi)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.LBL_2_3:
cfi_restore_state
vmovups %zmm8, 1152(%rsp)
vmovups %zmm0, 1216(%rsp)
vmovups %zmm1, 1280(%rsp)
je .LBL_2_2
xorb %dl, %dl
xorl %eax, %eax
kmovw %k4, 1048(%rsp)
kmovw %k5, 1040(%rsp)
kmovw %k6, 1032(%rsp)
kmovw %k7, 1024(%rsp)
vmovups %zmm16, 960(%rsp)
vmovups %zmm17, 896(%rsp)
vmovups %zmm18, 832(%rsp)
vmovups %zmm19, 768(%rsp)
vmovups %zmm20, 704(%rsp)
vmovups %zmm21, 640(%rsp)
vmovups %zmm22, 576(%rsp)
vmovups %zmm23, 512(%rsp)
vmovups %zmm24, 448(%rsp)
vmovups %zmm25, 384(%rsp)
vmovups %zmm26, 320(%rsp)
vmovups %zmm27, 256(%rsp)
vmovups %zmm28, 192(%rsp)
vmovups %zmm29, 128(%rsp)
vmovups %zmm30, 64(%rsp)
vmovups %zmm31, (%rsp)
movq %rsi, 1056(%rsp)
movq %r12, 1096(%rsp)
cfi_offset_rel_rsp (12, 1096)
movb %dl, %r12b
movq %r13, 1088(%rsp)
cfi_offset_rel_rsp (13, 1088)
movl %eax, %r13d
movq %r14, 1080(%rsp)
cfi_offset_rel_rsp (14, 1080)
movl %ecx, %r14d
movq %r15, 1072(%rsp)
cfi_offset_rel_rsp (15, 1072)
movq %rbx, 1064(%rsp)
movq %rdi, %rbx
cfi_remember_state
.LBL_2_6:
btl %r13d, %r14d
jc .LBL_2_13
.LBL_2_7:
lea 1(%r13), %esi
btl %esi, %r14d
jc .LBL_2_10
.LBL_2_8:
incb %r12b
addl $2, %r13d
cmpb $16, %r12b
jb .LBL_2_6
kmovw 1048(%rsp), %k4
movq %rbx, %rdi
kmovw 1040(%rsp), %k5
kmovw 1032(%rsp), %k6
kmovw 1024(%rsp), %k7
vmovups 960(%rsp), %zmm16
vmovups 896(%rsp), %zmm17
vmovups 832(%rsp), %zmm18
vmovups 768(%rsp), %zmm19
vmovups 704(%rsp), %zmm20
vmovups 640(%rsp), %zmm21
vmovups 576(%rsp), %zmm22
vmovups 512(%rsp), %zmm23
vmovups 448(%rsp), %zmm24
vmovups 384(%rsp), %zmm25
vmovups 320(%rsp), %zmm26
vmovups 256(%rsp), %zmm27
vmovups 192(%rsp), %zmm28
vmovups 128(%rsp), %zmm29
vmovups 64(%rsp), %zmm30
vmovups (%rsp), %zmm31
vmovups 1216(%rsp), %zmm0
vmovups 1280(%rsp), %zmm1
movq 1056(%rsp), %rsi
movq 1096(%rsp), %r12
cfi_restore (%r12)
movq 1088(%rsp), %r13
cfi_restore (%r13)
movq 1080(%rsp), %r14
cfi_restore (%r14)
movq 1072(%rsp), %r15
cfi_restore (%r15)
movq 1064(%rsp), %rbx
jmp .LBL_2_2
.LBL_2_10:
cfi_restore_state
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 1160(%rsp,%r15), %xmm0
vzeroupper
vmovsd 1160(%rsp,%r15), %xmm0
call sin@PLT
vmovsd %xmm0, 1224(%rsp,%r15)
vmovsd 1160(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 1288(%rsp,%r15)
jmp .LBL_2_8
.LBL_2_13:
movzbl %r12b, %r15d
shlq $4, %r15
vmovsd 1152(%rsp,%r15), %xmm0
vzeroupper
vmovsd 1152(%rsp,%r15), %xmm0
call sin@PLT
vmovsd %xmm0, 1216(%rsp,%r15)
vmovsd 1152(%rsp,%r15), %xmm0
call cos@PLT
vmovsd %xmm0, 1280(%rsp,%r15)
jmp .LBL_2_7
#endif
END (_ZGVeN8vvv_sincos_skx)
.section .rodata, "a"
.L_2il0floatpacket.15:
.long 0xffffffff,0xffffffff
.type .L_2il0floatpacket.15,@object

29
sysdeps/x86_64/fpu/svml_d_sincos2_core.S Normal file
View file

@ -0,0 +1,29 @@
/* Function sincos vectorized with SSE2.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_wrapper_impl.h"
.text
ENTRY (_ZGVbN2vvv_sincos)
WRAPPER_IMPL_SSE2_fFF sincos
END (_ZGVbN2vvv_sincos)
#ifndef USE_MULTIARCH
libmvec_hidden_def (_ZGVbN2vvv_sincos)
#endif

29
sysdeps/x86_64/fpu/svml_d_sincos4_core.S Normal file
View file

@ -0,0 +1,29 @@
/* Function sincos vectorized with AVX2, wrapper version.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_wrapper_impl.h"
.text
ENTRY (_ZGVdN4vvv_sincos)
WRAPPER_IMPL_AVX_fFF _ZGVbN2vvv_sincos
END (_ZGVdN4vvv_sincos)
#ifndef USE_MULTIARCH
libmvec_hidden_def (_ZGVdN4vvv_sincos)
#endif

25
sysdeps/x86_64/fpu/svml_d_sincos4_core_avx.S Normal file
View file

@ -0,0 +1,25 @@
/* Function sincos vectorized in AVX ISA as wrapper to SSE4 ISA version.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_wrapper_impl.h"
.text
ENTRY (_ZGVcN4vvv_sincos)
WRAPPER_IMPL_AVX_fFF _ZGVbN2vvv_sincos
END (_ZGVcN4vvv_sincos)

25
sysdeps/x86_64/fpu/svml_d_sincos8_core.S Normal file
View file

@ -0,0 +1,25 @@
/* Function sincos vectorized with AVX-512. Wrapper to AVX2 version.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include <sysdep.h>
#include "svml_d_wrapper_impl.h"
.text
ENTRY (_ZGVeN8vvv_sincos)
WRAPPER_IMPL_AVX512_fFF _ZGVdN4vvv_sincos
END (_ZGVeN8vvv_sincos)

111
sysdeps/x86_64/fpu/svml_d_sincos_data.S Normal file
View file

@ -0,0 +1,111 @@
/* Data for function sincos.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#include "svml_d_sincos_data.h"
.section .rodata, "a"
.align 64
/* Data table for vector implementations of function sincos.
The table may contain polynomial, reduction, lookup coefficients and
other coefficients obtained through different methods of research and
experimental work. */
.globl __svml_dsincos_data
__svml_dsincos_data:
/* General purpose constants:
dSignMask */
double_vector __dSignMask 0x8000000000000000
/* dAbsMask */
double_vector __dAbsMask 0x7fffffffffffffff
/* lRangeVal */
double_vector __dRangeVal 0x4160000000000000
/* HalfPI */
double_vector __dHalfPI 0x3ff921fb54442d18
/* InvPI */
double_vector __dInvPI 0x3fd45f306dc9c883
/* dRShifter */
double_vector __dRShifter 0x4338000000000000
/* dOneHalf */
double_vector __dOneHalf 0x3fe0000000000000
/* Range reduction PI-based constants:
PI1 */
double_vector __dPI1 0x400921fb40000000
/* PI2 */
double_vector __dPI2 0x3e84442d00000000
/* PI3 */
double_vector __dPI3 0x3d08469880000000
/* PI4 */
double_vector __dPI4 0x3b88cc51701b839a
/* Range reduction PI-based constants if FMA available:
PI1_FMA */
double_vector __dPI1_FMA 0x400921fb54442d18
/* PI2_FMA */
double_vector __dPI2_FMA 0x3ca1a62633145c06
/* PI3_FMA */
double_vector __dPI3_FMA 0x395c1cd129024e09
/* HalfPI1 */
double_vector __dHalfPI1 0x3ff921fc00000000
/* HalfPI2 */
double_vector __dHalfPI2 0xbea5777a00000000
/* HalfPI3 */
double_vector __dHalfPI3 0xbd473dcc00000000
/* HalfPI4 */
double_vector __dHalfPI4 0x3bf898cc51701b84
/* Polynomial coefficients (relative error 2^(-52.115)):
C1 */
double_vector __dC1 0xbfc55555555554a7
/* C2 */
double_vector __dC2 0x3f8111111110a4a8
/* C3 */
double_vector __dC3 0xbf2a01a019a5b86d
/* C4 */
double_vector __dC4 0x3ec71de38030fea0
/* C5 */
double_vector __dC5 0xbe5ae63546002231
/* C6 */
double_vector __dC6 0x3de60e6857a2f220
/* C7 */
double_vector __dC7 0xbd69f0d60811aac8
.type __svml_dsincos_data,@object
.size __svml_dsincos_data,.-__svml_dsincos_data

57
sysdeps/x86_64/fpu/svml_d_sincos_data.h Normal file
View file

@ -0,0 +1,57 @@
/* Offsets for data table for function sincos.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
#ifndef D_SINCOS_DATA_H
#define D_SINCOS_DATA_H
#define __dSignMask 0
#define __dAbsMask 64
#define __dRangeVal 128
#define __dHalfPI 192
#define __dInvPI 256
#define __dRShifter 320
#define __dOneHalf 384
#define __dPI1 448
#define __dPI2 512
#define __dPI3 576
#define __dPI4 640
#define __dPI1_FMA 704
#define __dPI2_FMA 768
#define __dPI3_FMA 832
#define __dHalfPI1 896
#define __dHalfPI2 960
#define __dHalfPI3 1024
#define __dHalfPI4 1088
#define __dC1 1152
#define __dC2 1216
#define __dC3 1280
#define __dC4 1344
#define __dC5 1408
#define __dC6 1472
#define __dC7 1536
.macro double_vector offset value
.if .-__svml_dsincos_data != \offset
.err
.endif
.rept 8
.quad \value
.endr
.endm
#endif

170
sysdeps/x86_64/fpu/svml_d_wrapper_impl.h
View file

@ -54,6 +54,47 @@
ret
.endm
/* 3 argument SSE2 ISA version as wrapper to scalar. */
.macro WRAPPER_IMPL_SSE2_fFF callee
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
pushq %rbx
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbx, 0)
movq %rdi, %rbp
movq %rsi, %rbx
subq $40, %rsp
cfi_adjust_cfa_offset(40)
leaq 16(%rsp), %rsi
leaq 24(%rsp), %rdi
movaps %xmm0, (%rsp)
call \callee@PLT
leaq 16(%rsp), %rsi
leaq 24(%rsp), %rdi
movsd 24(%rsp), %xmm0
movapd (%rsp), %xmm1
movsd %xmm0, 0(%rbp)
unpckhpd %xmm1, %xmm1
movsd 16(%rsp), %xmm0
movsd %xmm0, (%rbx)
movapd %xmm1, %xmm0
call \callee@PLT
movsd 24(%rsp), %xmm0
movsd %xmm0, 8(%rbp)
movsd 16(%rsp), %xmm0
movsd %xmm0, 8(%rbx)
addq $40, %rsp
cfi_adjust_cfa_offset(-40)
popq %rbx
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbx)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.endm
/* AVX/AVX2 ISA version as wrapper to SSE ISA version. */
.macro WRAPPER_IMPL_AVX callee
pushq %rbp
@ -108,6 +149,49 @@
ret
.endm
/* 3 argument AVX/AVX2 ISA version as wrapper to SSE ISA version. */
.macro WRAPPER_IMPL_AVX_fFF callee
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-32, %rsp
pushq %r13
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%r13, 0)
pushq %r14
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%r14, 0)
subq $48, %rsp
movq %rsi, %r14
movq %rdi, %r13
vextractf128 $1, %ymm0, 32(%rsp)
vzeroupper
call HIDDEN_JUMPTARGET(\callee)
vmovaps 32(%rsp), %xmm0
lea (%rsp), %rdi
lea 16(%rsp), %rsi
call HIDDEN_JUMPTARGET(\callee)
vmovapd (%rsp), %xmm0
vmovapd 16(%rsp), %xmm1
vmovapd %xmm0, 16(%r13)
vmovapd %xmm1, 16(%r14)
addq $48, %rsp
popq %r14
cfi_adjust_cfa_offset (-8)
cfi_restore (%r14)
popq %r13
cfi_adjust_cfa_offset (-8)
cfi_restore (%r13)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.endm
/* AVX512 ISA version as wrapper to AVX2 ISA version. */
.macro WRAPPER_IMPL_AVX512 callee
pushq %rbp
@ -209,3 +293,89 @@
cfi_restore (%rbp)
ret
.endm
/* 3 argument AVX512 ISA version as wrapper to AVX2 ISA version. */
.macro WRAPPER_IMPL_AVX512_fFF callee
pushq %rbp
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%rbp, 0)
movq %rsp, %rbp
cfi_def_cfa_register (%rbp)
andq $-64, %rsp
pushq %r12
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%r12, 0)
pushq %r13
cfi_adjust_cfa_offset (8)
cfi_rel_offset (%r13, 0)
subq $176, %rsp
movq %rsi, %r13
/* Below is encoding for vmovaps %zmm0, (%rsp). */
.byte 0x62
.byte 0xf1
.byte 0x7c
.byte 0x48
.byte 0x29
.byte 0x04
.byte 0x24
movq %rdi, %r12
/* Below is encoding for vmovapd (%rsp), %ymm0. */
.byte 0xc5
.byte 0xfd
.byte 0x28
.byte 0x04
.byte 0x24
call HIDDEN_JUMPTARGET(\callee)
/* Below is encoding for vmovapd 32(%rsp), %ymm0. */
.byte 0xc5
.byte 0xfd
.byte 0x28
.byte 0x44
.byte 0x24
.byte 0x20
lea 64(%rsp), %rdi
lea 96(%rsp), %rsi
call HIDDEN_JUMPTARGET(\callee)
/* Below is encoding for vmovapd 64(%rsp), %ymm0. */
.byte 0xc5
.byte 0xfd
.byte 0x28
.byte 0x44
.byte 0x24
.byte 0x40
/* Below is encoding for vmovapd 96(%rsp), %ymm1. */
.byte 0xc5
.byte 0xfd
.byte 0x28
.byte 0x4c
.byte 0x24
.byte 0x60
/* Below is encoding for vmovapd %ymm0, 32(%r12). */
.byte 0xc4
.byte 0xc1
.byte 0x7d
.byte 0x29
.byte 0x44
.byte 0x24
.byte 0x20
/* Below is encoding for vmovapd %ymm1, 32(%r13). */
.byte 0xc4
.byte 0xc1
.byte 0x7d
.byte 0x29
.byte 0x4d
.byte 0x20
addq $176, %rsp
popq %r13
cfi_adjust_cfa_offset (-8)
cfi_restore (%r13)
popq %r12
cfi_adjust_cfa_offset (-8)
cfi_restore (%r12)
movq %rbp, %rsp
cfi_def_cfa_register (%rsp)
popq %rbp
cfi_adjust_cfa_offset (-8)
cfi_restore (%rbp)
ret
.endm

1
sysdeps/x86_64/fpu/test-double-vlen2-wrappers.c
View file

@ -24,6 +24,7 @@
VECTOR_WRAPPER (WRAPPER_NAME (cos), _ZGVbN2v_cos)
VECTOR_WRAPPER (WRAPPER_NAME (sin), _ZGVbN2v_sin)
VECTOR_WRAPPER_fFF (WRAPPER_NAME (sincos), _ZGVbN2vvv_sincos)
VECTOR_WRAPPER (WRAPPER_NAME (log), _ZGVbN2v_log)
VECTOR_WRAPPER (WRAPPER_NAME (exp), _ZGVbN2v_exp)
VECTOR_WRAPPER_ff (WRAPPER_NAME (pow), _ZGVbN2vv_pow)

1
sysdeps/x86_64/fpu/test-double-vlen2.c
View file

@ -20,6 +20,7 @@
#define TEST_VECTOR_cos 1
#define TEST_VECTOR_sin 1
#define TEST_VECTOR_sincos 1
#define TEST_VECTOR_log 1
#define TEST_VECTOR_exp 1
#define TEST_VECTOR_pow 1

1
sysdeps/x86_64/fpu/test-double-vlen4-avx2-wrappers.c
View file

@ -27,6 +27,7 @@
VECTOR_WRAPPER (WRAPPER_NAME (cos), _ZGVdN4v_cos)
VECTOR_WRAPPER (WRAPPER_NAME (sin), _ZGVdN4v_sin)
VECTOR_WRAPPER_fFF (WRAPPER_NAME (sincos), _ZGVdN4vvv_sincos)
VECTOR_WRAPPER (WRAPPER_NAME (log), _ZGVdN4v_log)
VECTOR_WRAPPER (WRAPPER_NAME (exp), _ZGVdN4v_exp)
VECTOR_WRAPPER_ff (WRAPPER_NAME (pow), _ZGVdN4vv_pow)

1
sysdeps/x86_64/fpu/test-double-vlen4-avx2.c
View file

@ -23,6 +23,7 @@
#define TEST_VECTOR_cos 1
#define TEST_VECTOR_sin 1
#define TEST_VECTOR_sincos 1
#define TEST_VECTOR_log 1
#define TEST_VECTOR_exp 1
#define TEST_VECTOR_pow 1

1
sysdeps/x86_64/fpu/test-double-vlen4-wrappers.c
View file

@ -24,6 +24,7 @@
VECTOR_WRAPPER (WRAPPER_NAME (cos), _ZGVcN4v_cos)
VECTOR_WRAPPER (WRAPPER_NAME (sin), _ZGVcN4v_sin)
VECTOR_WRAPPER_fFF (WRAPPER_NAME (sincos), _ZGVcN4vvv_sincos)
VECTOR_WRAPPER (WRAPPER_NAME (log), _ZGVcN4v_log)
VECTOR_WRAPPER (WRAPPER_NAME (exp), _ZGVcN4v_exp)
VECTOR_WRAPPER_ff (WRAPPER_NAME (pow), _ZGVcN4vv_pow)

1
sysdeps/x86_64/fpu/test-double-vlen4.c
View file

@ -20,6 +20,7 @@
#define TEST_VECTOR_cos 1
#define TEST_VECTOR_sin 1
#define TEST_VECTOR_sincos 1
#define TEST_VECTOR_log 1
#define TEST_VECTOR_exp 1
#define TEST_VECTOR_pow 1

1
sysdeps/x86_64/fpu/test-double-vlen8-wrappers.c
View file

@ -24,6 +24,7 @@
VECTOR_WRAPPER (WRAPPER_NAME (cos), _ZGVeN8v_cos)
VECTOR_WRAPPER (WRAPPER_NAME (sin), _ZGVeN8v_sin)
VECTOR_WRAPPER_fFF (WRAPPER_NAME (sincos), _ZGVeN8vvv_sincos)
VECTOR_WRAPPER (WRAPPER_NAME (log), _ZGVeN8v_log)
VECTOR_WRAPPER (WRAPPER_NAME (exp), _ZGVeN8v_exp)
VECTOR_WRAPPER_ff (WRAPPER_NAME (pow), _ZGVeN8vv_pow)

1
sysdeps/x86_64/fpu/test-double-vlen8.c
View file

@ -20,6 +20,7 @@
#define TEST_VECTOR_cos 1
#define TEST_VECTOR_sin 1
#define TEST_VECTOR_sincos 1
#define TEST_VECTOR_log 1
#define TEST_VECTOR_exp 1
#define TEST_VECTOR_pow 1