summaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAge
...
* SLPVectorize: Put horizontal reductions feeding a store under separate flagArnold Schwaighofer2013-09-25
| | | | | | | Put them under a separate flag for experimentation. They are more likely to interfere with loop vectorization which happens later in the pass pipeline. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191371 91177308-0d34-0410-b5e6-96231b3b80d8
* set the cost of tiny trees to INT_MAX in SLP vectorizer to disable ↵Yi Jiang2013-09-24
| | | | | | vectorization on them git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191314 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "LoopVectorizer: Only allow vectorization of intrinsics."Arnold Schwaighofer2013-09-23
| | | | | | | | | | | | | | Revert 191122 - with extra checks we are allowed to vectorize math library function calls. Standard library indentifiers are reserved names so functions with external linkage must not overrided them. However, functions with internal linkage can. Therefore, we can vectorize calls to math library functions with a check for external linkage and matching signature. This matches what we do during SelectionDAG building. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191206 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix multiline comment warningArnold Schwaighofer2013-09-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191135 91177308-0d34-0410-b5e6-96231b3b80d8
* Reapply "SLPVectorizer: Handle more horizontal reductions (disabled)""Arnold Schwaighofer2013-09-21
| | | | | | | | | | | | | | Reapply r191108 with a fix for a memory corruption error I introduced. Of course, we can't reference the scalars that we replace by vectorizing and then call their eraseFromParent method. I only 'needed' the scalars to get the DebugLoc. Just store the DebugLoc before actually vectorizing instead. As a nice side effect, this also simplifies the interface between BoUpSLP and the HorizontalReduction class to returning a value pointer (the vectorized tree root). radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191123 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Only allow vectorization of intrinsics. We can't know for ↵Nadav Rotem2013-09-21
| | | | | | | | | | sure that the functions 'abs' or 'round' are the functions from libm. rdar://15012650 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191122 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "SLPVectorizer: Handle more horizontal reductions (disabled)"Arnold Schwaighofer2013-09-21
| | | | | | | | | This reverts commit r191108. The horizontal.ll test case fails under libgmalloc. Thanks Shuxin for pointing this out to me. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191121 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Handle more horizontal reductions (disabled)Arnold Schwaighofer2013-09-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Match reductions starting at binary operation feeding into a phi. The code handles trees like r += v1 + v2 + v3 ... and r += v1 r += v2 ... and r *= v1 + v2 + ... We currently only handle associative operations (add, fadd fast). The code can now also handle reductions feeding into stores. a[i] = v1 + v2 + v3 + ... The code is currently disabled behind the flag "-slp-vectorize-hor". The cost model for most architectures is not there yet. I found one opportunity of a horizontal reduction feeding a phi in TSVC (LoopRerolling-flt) and there are several opportunities where reductions feed into stores. radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191108 91177308-0d34-0410-b5e6-96231b3b80d8
* Prevent LoopVectorizer and SLPVectorizer running if the target has no vector ↵Robert Lytton2013-09-18
| | | | | | | | | | registers. XCore target: Add XCoreTargetTransformInfo This is where getNumberOfRegisters() resides, which in turn returns the number of vector registers (=0). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190936 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert accidental commit I had to make to get the test case in PR17268 to ↵Craig Topper2013-09-18
| | | | | | still work correctly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190917 91177308-0d34-0410-b5e6-96231b3b80d8
* Lift alignment restrictions for load/store folding on ↵Craig Topper2013-09-18
| | | | | | VINSERTF128/VEXTRACTF128. Fixes PR17268. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190916 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Don't vectorize phi nodes that use invoke valuesArnold Schwaighofer2013-09-17
| | | | | | | | | We can't insert an insertelement after an invoke. We would have to split a critical edge. So when we see a phi node that uses an invoke we just give up. radar://14990770 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190871 91177308-0d34-0410-b5e6-96231b3b80d8
* Don't vectorize if there are outside loop users of the induction variable.Arnold Schwaighofer2013-09-16
| | | | | | | | | | | | We would have to compute the pre increment value, either by computing it on every loop iteration or by splitting the edge out of the loop and inserting a computation for it there. For now, just give up vectorizing such loops. Fixes PR17179. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190790 91177308-0d34-0410-b5e6-96231b3b80d8
* Don't assert on invalid loop vectorization hint.Eli Friedman2013-09-10
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190450 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: PHI nodes are always at the beginning of a block, no need to ↵Benjamin Kramer2013-09-10
| | | | | | scan the whole block. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190422 91177308-0d34-0410-b5e6-96231b3b80d8
* In this patch we are trying to do two things:Yi Jiang2013-09-03
| | | | | | | | | 1) If the width of vectorization list candidate is bigger than vector reg width, we will break it down to fit the vector reg. 2) We do not vectorize the width which is not power of two. The performance result shows it will help some spec benchmarks. mesa improved 6.97% and ammp improved 1.54%. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189830 91177308-0d34-0410-b5e6-96231b3b80d8
* Disable unrolling in the loop vectorizer when disabled in the pass managerHal Finkel2013-08-28
| | | | | | | | | | | | | | | | | When unrolling is disabled in the pass manager, the loop vectorizer should also not unroll loops. This will allow the -fno-unroll-loops option in Clang to behave as expected (even for vectorizable loops). The loop vectorizer's -force-vector-unroll option will (continue to) override the pass-manager setting (including -force-vector-unroll=0 to force use of the internal auto-selection logic). In order to test this, I added a flag to opt (-disable-loop-unrolling) to force disable unrolling through opt (the analog of -fno-unroll-loops in Clang). Also, this fixes a small bug in opt where the loop vectorizer was enabled only after the pass manager populated the queue of passes (the global_alias.ll test needed a slight update to the RUN line as a result of this fix). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189499 91177308-0d34-0410-b5e6-96231b3b80d8
* Refactor 'vectorizeLoop' no functionality change.Nadav Rotem2013-08-27
| | | | | | | | This patch merges LoopVectorize of InnerLoopVectorizer and InnerLoopUnroller by adding checks for VF=1. This helps in erasing the Unroller code that is almost identical to the InnerLoopVectorizer code. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189391 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix inserting instructions before last in bundle.Matt Arsenault2013-08-26
| | | | | | | | | | | The builder inserts from before the insert point, not after, so this would insert before the last instruction in the bundle instead of after it. I'm not sure if this can actually be a problem with any of the current insertions. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189285 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Implement partial loop unrolling when vectorization is not ↵Nadav Rotem2013-08-26
| | | | | | | | | | | | | | | | | | profitable. This patch enables unrolling of loops when vectorization is legal but not profitable. We add a new class InnerLoopUnroller, that extends InnerLoopVectorizer and replaces some of the vector-specific logic with scalars. This patch does not introduce any runtime regressions and improves the following workloads: SingleSource/Benchmarks/Shootout/matrix -22.64% SingleSource/Benchmarks/Shootout-C++/matrix -13.06% External/SPEC/CINT2006/464_h264ref/464_h264ref -3.99% SingleSource/Benchmarks/Adobe-C++/simple_types_constant_folding -1.95% git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189281 91177308-0d34-0410-b5e6-96231b3b80d8
* test commit. Remove blank lineYi Jiang2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189265 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix unused variable in release buildMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189264 91177308-0d34-0410-b5e6-96231b3b80d8
* Constify functionsMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189234 91177308-0d34-0410-b5e6-96231b3b80d8
* Vectorize starting from insertelements building a vectorMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189233 91177308-0d34-0410-b5e6-96231b3b80d8
* Check if in set on insertion instead of separatelyMatt Arsenault2013-08-24
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189179 91177308-0d34-0410-b5e6-96231b3b80d8
* Teach the SLP vectorizer the correct way to check for consecutive accessChandler Carruth2013-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | using GEPs. Previously, it used a number of different heuristics for analyzing the GEPs. Several of these were conservatively correct, but failed to fall back to SCEV even when SCEV might have given a reasonable answer. One was simply incorrect in how it was formulated. There was good code already to recursively evaluate the constant offsets in GEPs, look through pointer casts, etc. I gathered this into a form code like the SLP code can use in a previous commit, which allows all of this code to become quite simple. There is some performance (compile time) concern here at first glance as we're directly attempting to walk both pointers constant GEP chains. However, a couple of thoughts: 1) The very common cases where there is a dynamic pointer, and a second pointer at a constant offset (usually a stride) from it, this code will actually not do any unnecessary work. 2) InstCombine and other passes work very hard to collapse constant GEPs, so it will be rare that we iterate here for a long time. That said, if there remain performance problems here, there are some obvious things that can improve the situation immensely. Doing a vectorizer-pass-wide memoizer for each individual layer of pointer values, their base values, and the constant offset is likely to be able to completely remove redundant work and strictly limit the scaling of the work to scrape these GEPs. Since this optimization was not done on the prior version (which would still benefit from it), I've not done it here. But if folks have benchmarks that slow down it should be straight forward for them to add. I've added a test case, but I'm not really confident of the amount of testing done for different access patterns, strides, and pointer manipulation. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189007 91177308-0d34-0410-b5e6-96231b3b80d8
* Teach LoopVectorize about address space sizesMatt Arsenault2013-08-22
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188980 91177308-0d34-0410-b5e6-96231b3b80d8
* Use attribute helper functionMatt Arsenault2013-08-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188916 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix typoMatt Arsenault2013-08-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188915 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix invalid iterator errorsArnold Schwaighofer2013-08-20
| | | | | | | | | | | Update iterator when the SLP vectorizer changes the instructions in the basic block by restarting the traversal of the basic block. Patch by Yi Jiang! Fixes PR 16899. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188832 91177308-0d34-0410-b5e6-96231b3b80d8
* Add a llvm.copysign intrinsicHal Finkel2013-08-19
| | | | | | | | | | | | | | | | | | | | | This adds a llvm.copysign intrinsic; We already have Libfunc recognition for copysign (which is turned into the FCOPYSIGN SDAG node). In order to autovectorize calls to copysign in the loop vectorizer, we need a corresponding intrinsic as well. In addition to the expected changes to the language reference, the loop vectorizer, BasicTTI, and the SDAG builder (the intrinsic is transformed into an FCOPYSIGN node, just like the function call), this also adds FCOPYSIGN to a few lists in LegalizeVector{Ops,Types} so that vector copysigns can be expanded. In TargetLoweringBase::initActions, I've made the default action for FCOPYSIGN be Expand for vector types. This seems correct for all in-tree targets, and I think is the right thing to do because, previously, there was no way to generate vector-values FCOPYSIGN nodes (and most targets don't specify an action for vector-typed FCOPYSIGN). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188728 91177308-0d34-0410-b5e6-96231b3b80d8
* PR 16899: Do not modify the basic block using the iterator, but keep theJoerg Sonnenberger2013-08-17
| | | | | | | | | next value. This avoids crashes due to invalidation. Patch by Joey Gouly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188605 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix spellingMatt Arsenault2013-08-15
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188506 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Add initial stores to the write set when tracking usesHal Finkel2013-08-13
| | | | | | | | | | | | | | | | When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188329 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix PR16797 - Support PHINodes with multiple inputs from the same basic block.Nadav Rotem2013-08-12
| | | | | | | | | Do not generate new vector values for the same entries because we know that the incoming values from the same block must be identical. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188185 91177308-0d34-0410-b5e6-96231b3b80d8
* Add ISD::FROUND for libm round()Hal Finkel2013-08-07
| | | | | | | | | | | | | | | All libm floating-point rounding functions, except for round(), had their own ISD nodes. Recent PowerPC cores have an instruction for round(), and so here I'm adding ISD::FROUND so that round() can be custom lowered as well. For the most part, this is straightforward. I've added an intrinsic and a matching ISD node just like those for nearbyint() and friends. The SelectionDAG pattern I've named frnd (because ISD::FP_ROUND has already claimed fround). This will be used by the PowerPC backend in a follow-up commit. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187926 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Allow vectorization of loops with lifetime markersArnold Schwaighofer2013-08-06
| | | | | | Patch by Marc Jessome! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187825 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix PR16777. PHInodes may use multiple extracted values that ↵Nadav Rotem2013-08-02
| | | | | | | | | | come from different blocks. Thanks Alexey Samsonov. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187663 91177308-0d34-0410-b5e6-96231b3b80d8
* 80-colNadav Rotem2013-07-31
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187535 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorier: update the debug location for the new instructions.Nadav Rotem2013-07-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187363 91177308-0d34-0410-b5e6-96231b3b80d8
* Don't vectorize when the attribute NoImplicitFloat is used.Nadav Rotem2013-07-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187340 91177308-0d34-0410-b5e6-96231b3b80d8
* Update the commentNadav Rotem2013-07-27
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187316 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP Vectorier: Don't vectorize really short chains because they are already ↵Nadav Rotem2013-07-26
| | | | | | handled by the SelectionDAG store-vectorizer, which does a better job in deciding when to vectorize. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187267 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP Vectorizer: Disable the vectorization of non power of two chains, such ↵Nadav Rotem2013-07-26
| | | | | | as <3 x float>, because we dont have a good cost model for these types. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187265 91177308-0d34-0410-b5e6-96231b3b80d8
* When we vectorize across multiple basic blocks we may vectorize PHINodes ↵Nadav Rotem2013-07-22
| | | | | | that create a cycle. We already break the cycle on phi-nodes, but arithmetic operations are still uplicated. This patch adds code that checks if the operation that we are vectorizing was vectorized during the visit of the operands and uses this value if it can. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186883 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix an obvious typo in the loop vectorizer where the cost model uses the ↵Nadav Rotem2013-07-22
| | | | | | | | | | wrong variable. The variable BlockCost is ignored. We don't have tests for the effect of if-conversion loops because it requires a big test (that includes if-converted loops) and it is difficult to find and balance a loop to do the right thing. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186845 91177308-0d34-0410-b5e6-96231b3b80d8
* Delete unused helper functions.Nadav Rotem2013-07-22
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186808 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert a part of r186420. Don't forbid multiple store chains that merge.Nadav Rotem2013-07-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186786 91177308-0d34-0410-b5e6-96231b3b80d8
* fix an 80-col line.Nadav Rotem2013-07-19
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186733 91177308-0d34-0410-b5e6-96231b3b80d8
* Use LLVMs ADTs that improve the compile time of this pass.Nadav Rotem2013-07-19
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186732 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-06 20:37:03 +0000