diff options
author | Arnold Schwaighofer <aschwaighofer@apple.com> | 2014-01-28 01:01:53 +0000 |
---|---|---|
committer | Arnold Schwaighofer <aschwaighofer@apple.com> | 2014-01-28 01:01:53 +0000 |
commit | a47aa4b4efb2b6c5944c2edb9a4346c0f3ffcaf1 (patch) | |
tree | f055ceb22bddd71944654b889fb90cf9944c563c /test | |
parent | 1c2827cd6a5456126b9a8041642148279456f71a (diff) | |
download | llvm-a47aa4b4efb2b6c5944c2edb9a4346c0f3ffcaf1.tar.gz llvm-a47aa4b4efb2b6c5944c2edb9a4346c0f3ffcaf1.tar.bz2 llvm-a47aa4b4efb2b6c5944c2edb9a4346c0f3ffcaf1.tar.xz |
LoopVectorize: Support conditional stores by scalarizing
The vectorizer takes a loop like this and widens all instructions except for the
store. The stores are scalarized/unrolled and hidden behind an "if" block.
for (i = 0; i < 128; ++i) {
if (a[i] < 10)
a[i] += val;
}
for (i = 0; i < 128; i+=2) {
v = a[i:i+1];
v0 = (extract v, 0) + 10;
v1 = (extract v, 1) + 10;
if (v0 < 10)
a[i] = v0;
if (v1 < 10)
a[i] = v1;
}
The vectorizer relies on subsequent optimizations to sink instructions into the
conditional block where they are anticipated.
The flag "vectorize-num-stores-pred" controls whether and how many stores to
handle this way. Vectorization of conditional stores is disabled per default for
now.
This patch also adds a change to the heuristic when the flag
"enable-loadstore-runtime-unroll" is enabled (off by default). It unrolls small
loops until load/store ports are saturated. This heuristic uses TTI's
getMaxUnrollFactor as a measure for load/store ports.
I also added a second flag -enable-cond-stores-vec. It will enable vectorization
of conditional stores. But there is no cost model for vectorization of
conditional stores in place yet so this will not do good at the moment.
rdar://15892953
Results for x86-64 -O3 -mavx +/- -mllvm -enable-loadstore-runtime-unroll
-vectorize-num-stores-pred=1 (before the BFI change):
Performance Regressions:
Benchmarks/Ptrdist/yacr2/yacr2 7.35% (maze3() is identical but 10% slower)
Applications/siod/siod 2.18%
Performance improvements:
mesa -4.42%
libquantum -4.15%
With a patch that slightly changes the register heuristics (by subtracting the
induction variable on both sides of the register pressure equation, as the
induction variable is probably not really unrolled):
Performance Regressions:
Benchmarks/Ptrdist/yacr2/yacr2 7.73%
Applications/siod/siod 1.97%
Performance Improvements:
libquantum -13.05% (we now also unroll quantum_toffoli)
mesa -4.27%
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@200270 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'test')
-rw-r--r-- | test/Transforms/LoopVectorize/if-pred-stores.ll | 86 |
1 files changed, 86 insertions, 0 deletions
diff --git a/test/Transforms/LoopVectorize/if-pred-stores.ll b/test/Transforms/LoopVectorize/if-pred-stores.ll new file mode 100644 index 0000000000..909a1cb17d --- /dev/null +++ b/test/Transforms/LoopVectorize/if-pred-stores.ll @@ -0,0 +1,86 @@ +; RUN: opt -S -vectorize-num-stores-pred=1 -force-vector-width=1 -force-vector-unroll=2 -loop-vectorize < %s | FileCheck %s --check-prefix=UNROLL +; RUN: opt -S -vectorize-num-stores-pred=1 -force-vector-width=2 -force-vector-unroll=1 -loop-vectorize -enable-cond-stores-vec < %s | FileCheck %s --check-prefix=VEC +target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128" +target triple = "x86_64-apple-macosx10.9.0" + +; Test predication of stores. +define i32 @test(i32* nocapture %f) #0 { +entry: + br label %for.body + +; VEC-LABEL: test +; VEC: %[[v8:.+]] = icmp sgt <2 x i32> %{{.*}}, <i32 100, i32 100> +; VEC: %[[v9:.+]] = add nsw <2 x i32> %{{.*}}, <i32 20, i32 20> +; VEC: %[[v10:.+]] = and <2 x i1> %[[v8]], <i1 true, i1 true> +; VEC: %[[v11:.+]] = extractelement <2 x i1> %[[v10]], i32 0 +; VEC: %[[v12:.+]] = icmp eq i1 %[[v11]], true +; VEC: br i1 %[[v12]], label %[[cond:.+]], label %[[else:.+]] +; +; VEC: [[cond]]: +; VEC: %[[v13:.+]] = extractelement <2 x i32> %[[v9]], i32 0 +; VEC: %[[v14:.+]] = extractelement <2 x i32*> %{{.*}}, i32 0 +; VEC: store i32 %[[v13]], i32* %[[v14]], align 4 +; VEC: br label %[[else:.+]] +; +; VEC: [[else]]: +; VEC: %[[v15:.+]] = extractelement <2 x i1> %[[v10]], i32 1 +; VEC: %[[v16:.+]] = icmp eq i1 %[[v15]], true +; VEC: br i1 %[[v16]], label %[[cond2:.+]], label %[[else2:.+]] +; +; VEC: [[cond2]]: +; VEC: %[[v17:.+]] = extractelement <2 x i32> %[[v9]], i32 1 +; VEC: %[[v18:.+]] = extractelement <2 x i32*> %{{.+}} i32 1 +; VEC: store i32 %[[v17]], i32* %[[v18]], align 4 +; VEC: br label %[[else2:.+]] +; +; VEC: [[else2]]: + +; UNROLL-LABEL: test +; UNROLL: vector.body: +; UNROLL: %[[IND:[a-zA-Z0-9]+]] = add i64 %{{.*}}, 0 +; UNROLL: %[[IND1:[a-zA-Z0-9]+]] = add i64 %{{.*}}, 1 +; UNROLL: %[[v0:[a-zA-Z0-9]+]] = getelementptr inbounds i32* %f, i64 %[[IND]] +; UNROLL: %[[v1:[a-zA-Z0-9]+]] = getelementptr inbounds i32* %f, i64 %[[IND1]] +; UNROLL: %[[v2:[a-zA-Z0-9]+]] = load i32* %[[v0]], align 4 +; UNROLL: %[[v3:[a-zA-Z0-9]+]] = load i32* %[[v1]], align 4 +; UNROLL: %[[v4:[a-zA-Z0-9]+]] = icmp sgt i32 %[[v2]], 100 +; UNROLL: %[[v5:[a-zA-Z0-9]+]] = icmp sgt i32 %[[v3]], 100 +; UNROLL: %[[v6:[a-zA-Z0-9]+]] = add nsw i32 %[[v2]], 20 +; UNROLL: %[[v7:[a-zA-Z0-9]+]] = add nsw i32 %[[v3]], 20 +; UNROLL: %[[v8:[a-zA-Z0-9]+]] = icmp eq i1 %[[v4]], true +; UNROLL: br i1 %[[v8]], label %[[cond:[a-zA-Z0-9.]+]], label %[[else:[a-zA-Z0-9.]+]] +; +; UNROLL: [[cond]]: +; UNROLL: store i32 %[[v6]], i32* %[[v0]], align 4 +; UNROLL: br label %[[else]] +; +; UNROLL: [[else]]: +; UNROLL: %[[v9:[a-zA-Z0-9]+]] = icmp eq i1 %[[v5]], true +; UNROLL: br i1 %[[v9]], label %[[cond2:[a-zA-Z0-9.]+]], label %[[else2:[a-zA-Z0-9.]+]] +; +; UNROLL: [[cond2]]: +; UNROLL: store i32 %[[v7]], i32* %[[v1]], align 4 +; UNROLL: br label %[[else2]] +; +; UNROLL: [[else2]]: + +for.body: + %indvars.iv = phi i64 [ 0, %entry ], [ %indvars.iv.next, %for.inc ] + %arrayidx = getelementptr inbounds i32* %f, i64 %indvars.iv + %0 = load i32* %arrayidx, align 4 + %cmp1 = icmp sgt i32 %0, 100 + br i1 %cmp1, label %if.then, label %for.inc + +if.then: + %add = add nsw i32 %0, 20 + store i32 %add, i32* %arrayidx, align 4 + br label %for.inc + +for.inc: + %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1 + %exitcond = icmp eq i64 %indvars.iv.next, 128 + br i1 %exitcond, label %for.end, label %for.body + +for.end: + ret i32 0 +} |