summaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize/SLPVectorizer.cpp
Commit message (Collapse)AuthorAge
...
* Migrate metadata information from scalar to vector instructions duringChandler Carruth2013-11-23
| | | | | | | | | | | SLP vectorization. Based on the code in BBVectorizer. Fixes PR17741. Patch by Raul Silvera, reviewed by Hal and Nadav. Reformatted by my driving of clang-format. =] git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@195528 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix whitespace errors.Arnold Schwaighofer2013-11-22
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@195468 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP Vectorizer: Extract cost will only be added once even if the scalar has ↵Yi Jiang2013-11-22
| | | | | | multiple external uses. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@195406 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix stale for Value pointer arrayArnold Schwaighofer2013-11-19
| | | | | | | | | | | | | | | We are slicing an array of Value pointers and process those slices in a loop. The problem is that we might invalidate a later slice by vectorizing a former slice. Use a WeakVH to track the pointer. If the pointer is deleted or RAUW'ed we can tell. The test case will only fail when running with libgmalloc. radar://15498655 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@195162 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix whitespace errorsArnold Schwaighofer2013-11-19
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@195161 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Use properlyDominates to satisfy the irreflexivity of a ↵Benjamin Kramer2013-11-04
| | | | | | | | strict weak ordering. STL debug mode checks this. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@194015 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Add a missing pair of parens. No functionality change.Benjamin Kramer2013-11-03
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193958 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: When CSEing generated gathers only scan blocks containing them.Benjamin Kramer2013-11-03
| | | | | | | | | | | Instead of doing a RPO traversal of the whole function remember the blocks containing gathers (typically <= 2) and scan them in dominator-first order. The actual CSE is still quadratic, but I'm not confident that adding a scoped hash table here is worth it as we're only looking at the generated instructions and not arbitrary code. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193956 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Remove duplicated function.Benjamin Kramer2013-11-02
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193927 91177308-0d34-0410-b5e6-96231b3b80d8
* ARM cost model: Account for zero cost scalar SROA instructionsArnold Schwaighofer2013-10-29
| | | | | | | | | By vectorizing a series of srl, or, ... instructions we have obfuscated the intention so much that the backend does not know how to fold this code away. radar://15336950 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193573 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Use vector type for vectorized memory operationsArnold Schwaighofer2013-10-29
| | | | | | | | | No test case, because with the current cost model we don't see a difference. An upcoming ARM memory cost model change will expose and test this bug. radar://15332579 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193572 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Don't vectorize volatile memory operationsArnold Schwaighofer2013-10-16
| | | | | | | | | | radar://15231682 Reapply r192799, http://lab.llvm.org:8011/builders/lldb-x86_64-debian-clang/builds/8226 showed that the bot is still broken even with this out. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@192820 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "SLPVectorizer: Don't vectorize volatile memory operations"Arnold Schwaighofer2013-10-16
| | | | | | This speculatively reverts commit 192799. It might have broken a linux buildbot. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@192816 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Don't vectorize volatile memory operationsArnold Schwaighofer2013-10-16
| | | | | | radar://15231682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@192799 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Sort PHINodes based on their opcodeArnold Schwaighofer2013-10-12
| | | | | | | | | | | | | | Before this patch we relied on the order of phi nodes when we looked for phi nodes of the same type. This could prevent vectorization of cases where there was a phi node of a second type in between phi nodes of some type. This is important for vectorization of an internal graphics kernel. On the test suite + external on x86_64 (and on a run on armv7s) it showed no impact on either performance or compile time. radar://15024459 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@192537 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Sort inputs to commutative binary operationsArnold Schwaighofer2013-10-04
| | | | | | | | | | | | | | | | | | | | | | | Sort the operands of the other entries in the current vectorization root according to the first entry's operands opcodes. %conv0 = uitofp ... %load0 = load float ... = fmul %conv0, %load0 = fmul %load0, %conv1 = fmul %load0, %conv2 Make sure that we recursively vectorize <%conv0, %conv1, %conv2> and <%load0, %load0, %load0>. This makes it more likely to obtain vectorizable trees. We have to be careful when we sort that we don't destroy 'good' existing ordering implied by source order. radar://15080067 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191977 91177308-0d34-0410-b5e6-96231b3b80d8
* Apply slp vectorization on fully-vectorizable tree of height 2Yi Jiang2013-10-02
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191852 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Make store chain finding more aggressive with ↵Benjamin Kramer2013-10-02
| | | | | | | | | GetUnderlyingObject. This recursively strips all GEPs like the existing code. It also handles bitcasts and other operations that do not change the pointer value. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191847 91177308-0d34-0410-b5e6-96231b3b80d8
* Remove several unused variables.Rafael Espindola2013-10-01
| | | | | | Patch by Alp Toker. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191757 91177308-0d34-0410-b5e6-96231b3b80d8
* IRBuilder: Add RAII objects to reset insertion points or fast math flags.Benjamin Kramer2013-09-30
| | | | | | | | Inspired by the object from the SLPVectorizer. This found a minor bug in the debug loc restoration in the vectorizer where the location of a following instruction was attached instead of the location from the original instruction. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191673 91177308-0d34-0410-b5e6-96231b3b80d8
* Even more spelling fixes for "instruction".Robert Wilhelm2013-09-28
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191611 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix SLPVectorizer using wrong address space for load/storeMatt Arsenault2013-09-27
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191564 91177308-0d34-0410-b5e6-96231b3b80d8
* Transforms: Use getFirstNonPHI to set the insertion point for PHIsJustin Bogner2013-09-27
| | | | | | | | | | We were previously using getFirstInsertionPt to insert PHI instructions when vectorizing, but getFirstInsertionPt also skips past landingpads, causing this to generate invalid IR. We can avoid this issue by using getFirstNonPHI instead. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191526 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorize: Put horizontal reductions feeding a store under separate flagArnold Schwaighofer2013-09-25
| | | | | | | Put them under a separate flag for experimentation. They are more likely to interfere with loop vectorization which happens later in the pass pipeline. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191371 91177308-0d34-0410-b5e6-96231b3b80d8
* set the cost of tiny trees to INT_MAX in SLP vectorizer to disable ↵Yi Jiang2013-09-24
| | | | | | vectorization on them git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191314 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix multiline comment warningArnold Schwaighofer2013-09-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191135 91177308-0d34-0410-b5e6-96231b3b80d8
* Reapply "SLPVectorizer: Handle more horizontal reductions (disabled)""Arnold Schwaighofer2013-09-21
| | | | | | | | | | | | | | Reapply r191108 with a fix for a memory corruption error I introduced. Of course, we can't reference the scalars that we replace by vectorizing and then call their eraseFromParent method. I only 'needed' the scalars to get the DebugLoc. Just store the DebugLoc before actually vectorizing instead. As a nice side effect, this also simplifies the interface between BoUpSLP and the HorizontalReduction class to returning a value pointer (the vectorized tree root). radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191123 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "SLPVectorizer: Handle more horizontal reductions (disabled)"Arnold Schwaighofer2013-09-21
| | | | | | | | | This reverts commit r191108. The horizontal.ll test case fails under libgmalloc. Thanks Shuxin for pointing this out to me. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191121 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Handle more horizontal reductions (disabled)Arnold Schwaighofer2013-09-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Match reductions starting at binary operation feeding into a phi. The code handles trees like r += v1 + v2 + v3 ... and r += v1 r += v2 ... and r *= v1 + v2 + ... We currently only handle associative operations (add, fadd fast). The code can now also handle reductions feeding into stores. a[i] = v1 + v2 + v3 + ... The code is currently disabled behind the flag "-slp-vectorize-hor". The cost model for most architectures is not there yet. I found one opportunity of a horizontal reduction feeding a phi in TSVC (LoopRerolling-flt) and there are several opportunities where reductions feed into stores. radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191108 91177308-0d34-0410-b5e6-96231b3b80d8
* Prevent LoopVectorizer and SLPVectorizer running if the target has no vector ↵Robert Lytton2013-09-18
| | | | | | | | | | registers. XCore target: Add XCoreTargetTransformInfo This is where getNumberOfRegisters() resides, which in turn returns the number of vector registers (=0). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190936 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert accidental commit I had to make to get the test case in PR17268 to ↵Craig Topper2013-09-18
| | | | | | still work correctly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190917 91177308-0d34-0410-b5e6-96231b3b80d8
* Lift alignment restrictions for load/store folding on ↵Craig Topper2013-09-18
| | | | | | VINSERTF128/VEXTRACTF128. Fixes PR17268. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190916 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Don't vectorize phi nodes that use invoke valuesArnold Schwaighofer2013-09-17
| | | | | | | | | We can't insert an insertelement after an invoke. We would have to split a critical edge. So when we see a phi node that uses an invoke we just give up. radar://14990770 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190871 91177308-0d34-0410-b5e6-96231b3b80d8
* In this patch we are trying to do two things:Yi Jiang2013-09-03
| | | | | | | | | 1) If the width of vectorization list candidate is bigger than vector reg width, we will break it down to fit the vector reg. 2) We do not vectorize the width which is not power of two. The performance result shows it will help some spec benchmarks. mesa improved 6.97% and ammp improved 1.54%. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189830 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix inserting instructions before last in bundle.Matt Arsenault2013-08-26
| | | | | | | | | | | The builder inserts from before the insert point, not after, so this would insert before the last instruction in the bundle instead of after it. I'm not sure if this can actually be a problem with any of the current insertions. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189285 91177308-0d34-0410-b5e6-96231b3b80d8
* test commit. Remove blank lineYi Jiang2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189265 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix unused variable in release buildMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189264 91177308-0d34-0410-b5e6-96231b3b80d8
* Constify functionsMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189234 91177308-0d34-0410-b5e6-96231b3b80d8
* Vectorize starting from insertelements building a vectorMatt Arsenault2013-08-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189233 91177308-0d34-0410-b5e6-96231b3b80d8
* Check if in set on insertion instead of separatelyMatt Arsenault2013-08-24
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189179 91177308-0d34-0410-b5e6-96231b3b80d8
* Teach the SLP vectorizer the correct way to check for consecutive accessChandler Carruth2013-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | using GEPs. Previously, it used a number of different heuristics for analyzing the GEPs. Several of these were conservatively correct, but failed to fall back to SCEV even when SCEV might have given a reasonable answer. One was simply incorrect in how it was formulated. There was good code already to recursively evaluate the constant offsets in GEPs, look through pointer casts, etc. I gathered this into a form code like the SLP code can use in a previous commit, which allows all of this code to become quite simple. There is some performance (compile time) concern here at first glance as we're directly attempting to walk both pointers constant GEP chains. However, a couple of thoughts: 1) The very common cases where there is a dynamic pointer, and a second pointer at a constant offset (usually a stride) from it, this code will actually not do any unnecessary work. 2) InstCombine and other passes work very hard to collapse constant GEPs, so it will be rare that we iterate here for a long time. That said, if there remain performance problems here, there are some obvious things that can improve the situation immensely. Doing a vectorizer-pass-wide memoizer for each individual layer of pointer values, their base values, and the constant offset is likely to be able to completely remove redundant work and strictly limit the scaling of the work to scrape these GEPs. Since this optimization was not done on the prior version (which would still benefit from it), I've not done it here. But if folks have benchmarks that slow down it should be straight forward for them to add. I've added a test case, but I'm not really confident of the amount of testing done for different access patterns, strides, and pointer manipulation. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189007 91177308-0d34-0410-b5e6-96231b3b80d8
* Use attribute helper functionMatt Arsenault2013-08-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188916 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix typoMatt Arsenault2013-08-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188915 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix invalid iterator errorsArnold Schwaighofer2013-08-20
| | | | | | | | | | | Update iterator when the SLP vectorizer changes the instructions in the basic block by restarting the traversal of the basic block. Patch by Yi Jiang! Fixes PR 16899. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188832 91177308-0d34-0410-b5e6-96231b3b80d8
* PR 16899: Do not modify the basic block using the iterator, but keep theJoerg Sonnenberger2013-08-17
| | | | | | | | | next value. This avoids crashes due to invalidation. Patch by Joey Gouly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188605 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix spellingMatt Arsenault2013-08-15
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188506 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix PR16797 - Support PHINodes with multiple inputs from the same basic block.Nadav Rotem2013-08-12
| | | | | | | | | Do not generate new vector values for the same entries because we know that the incoming values from the same block must be identical. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188185 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix PR16777. PHInodes may use multiple extracted values that ↵Nadav Rotem2013-08-02
| | | | | | | | | | come from different blocks. Thanks Alexey Samsonov. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187663 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorier: update the debug location for the new instructions.Nadav Rotem2013-07-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187363 91177308-0d34-0410-b5e6-96231b3b80d8
* Don't vectorize when the attribute NoImplicitFloat is used.Nadav Rotem2013-07-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187340 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-12 05:55:18 +0000