summaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAge
* Revert "Introduce a string_ostream string builder facilty"Alp Toker2014-06-26
| | | | | | Temporarily back out commits r211749, r211752 and r211754. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211814 91177308-0d34-0410-b5e6-96231b3b80d8
* Introduce a string_ostream string builder faciltyAlp Toker2014-06-26
| | | | | | | | | | | | | | | | | | | | string_ostream is a safe and efficient string builder that combines opaque stack storage with a built-in ostream interface. small_string_ostream<bytes> additionally permits an explicit stack storage size other than the default 128 bytes to be provided. Beyond that, storage is transferred to the heap. This convenient class can be used in most places an std::string+raw_string_ostream pair or SmallString<>+raw_svector_ostream pair would previously have been used, in order to guarantee consistent access without byte truncation. The patch also converts much of LLVM to use the new facility. These changes include several probable bug fixes for truncated output, a programming error that's no longer possible with the new interface. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211749 91177308-0d34-0410-b5e6-96231b3b80d8
* Add Rpass-missed and Rpass-analysis reports to the loop vectorizer. The ↵Tyler Nowicki2014-06-25
| | | | | | | | | remarks give the vector width of vectorized loops and a brief analysis of loops that fail to be vectorized. For example, an analysis will be generated for loops containing control flow that cannot be simplified to a select. The optimization remarks also give the debug location of expressions that cannot be vectorized, for example the location of an unvectorizable call. Reviewed by: Arnold Schwaighofer git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211721 91177308-0d34-0410-b5e6-96231b3b80d8
* Rename loop unrolling and loop vectorizer metadata to have a common prefix.Eli Bendersky2014-06-25
| | | | | | | | | | | | | | | | | | | | | [LLVM part] These patches rename the loop unrolling and loop vectorizer metadata such that they have a common 'llvm.loop.' prefix. Metadata name changes: llvm.vectorizer.* => llvm.loop.vectorizer.* llvm.loopunroll.* => llvm.loop.unroll.* This was a suggestion from an earlier review (http://reviews.llvm.org/D4090) which added the loop unrolling metadata. Patch by Mark Heffernan. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211710 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Fix a dominance issueArnold Schwaighofer2014-06-22
| | | | | | | The induction variables start value needs to be defined before we branch (overflow check) to the scalar preheader where we used it. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211460 91177308-0d34-0410-b5e6-96231b3b80d8
* Add Support to Recognize and Vectorize NON SIMD instructions in SLPVectorizer.Karthik Bhat2014-06-20
| | | | | | | | | | This patch adds support to recognize patterns such as fadd,fsub,fadd,fsub.../add,sub,add,sub... and vectorizes them as vector shuffles if they are profitable. These patterns of vector shuffle can later be converted to instructions such as addsubpd etc on X86. Thanks to Arnold and Hal for the reviews. http://reviews.llvm.org/D4015 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@211339 91177308-0d34-0410-b5e6-96231b3b80d8
* [SLP] Enable vectorization of GEP expressions.Michael Zolotukhin2014-06-06
| | | | | | | | The use cases look like the following: x->a = y->a + 10 x->b = y->b + 12 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210342 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix PR19657 (scalar loads not combined into vector load)Karthik Bhat2014-06-06
| | | | | | | | | | If we have common uses on separate paths in the tree; process the one with greater common depth first. This makes sure that we do not assume we need to extract a load when it is actually going to be part of a vectorized tree. Review: http://reviews.llvm.org/D3800 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@210310 91177308-0d34-0410-b5e6-96231b3b80d8
* Allow vectorization of intrinsics such as powi,cttz and ctlz in Loop and SLP ↵Karthik Bhat2014-05-30
| | | | | | | | | | | Vectorizer. This patch adds support to vectorize intrinsics such as powi, cttz and ctlz in Vectorizer. These intrinsics are different from other intrinsics as second argument to these function must be same in order to vectorize them and it should be represented as a scalar. Review: http://reviews.llvm.org/D3851#inline-32769 and http://reviews.llvm.org/D3937#inline-32857 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@209873 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Add a check that the backedge taken count + 1 does not overflowArnold Schwaighofer2014-05-29
| | | | | | | | | | | The loop vectorizer instantiates be-taken-count + 1 as the loop iteration count. If this expression overflows the generated code was invalid. In case of overflow the code now jumps to the scalar loop. Fixes PR17288. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@209854 91177308-0d34-0410-b5e6-96231b3b80d8
* Add support for missed and analysis optimization remarks.Diego Novillo2014-05-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This adds two new diagnostics: -pass-remarks-missed and -pass-remarks-analysis. They take the same values as -pass-remarks but are intended to be triggered in different contexts. -pass-remarks-missed is used by LLVMContext::emitOptimizationRemarkMissed, which passes call when they tried to apply a transformation but couldn't. -pass-remarks-analysis is used by LLVMContext::emitOptimizationRemarkAnalysis, which passes call when they want to inform the user about analysis results. The patch also: 1- Adds support in the inliner for the two new remarks and a test case. 2- Moves emitOptimizationRemark* functions to the llvm namespace. 3- Adds an LLVMContext argument instead of making them member functions of LLVMContext. Reviewers: qcolombet Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D3682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@209442 91177308-0d34-0410-b5e6-96231b3b80d8
* Clean up language and grammar.Eric Christopher2014-05-20
| | | | | | | Based on a patch by jfcaron3@gmail.com! PR19806 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@209216 91177308-0d34-0410-b5e6-96231b3b80d8
* [LV][REFACTOR] One more tiny fix for printing debug locations in loop ↵Zinovy Nis2014-05-20
| | | | | | | | | | vectorizer. Now consistent with the remarks emitter. Differential Revision: http://reviews.llvm.org/D3821 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@209197 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Instead of just performing CSE on dead blocks ignore them ↵Benjamin Kramer2014-05-11
| | | | | | | | | | | | | completely. Turns out that there is a very cheap way of testing whether a block is dead, just look it up in the DomTree. We have to do this anyways so just ignore unreachable blocks before sorting by domination. This restores a proper ordering for std::stable_sort when dead code is present. Covered by existing tests & buildbots running in STL debug mode (MSVC). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@208492 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: When sorting by domination for CSE don't assert on ↵Benjamin Kramer2014-05-09
| | | | | | | | | | | | unreachable code. There is no total ordering if the CFG is disconnected. We don't care if we catch all CSE opportunities in dead code either so just exclude ignore them in the assert. PR19646 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@208461 91177308-0d34-0410-b5e6-96231b3b80d8
* [BUG][REFACTOR]Zinovy Nis2014-05-07
| | | | | | | | | | | | 1) Fix for printing debug locations for absolute paths. 2) Location printing is moved into public method DebugLoc::print() to avoid re-inventing the wheel. Differential Revision: http://reviews.llvm.org/D3513 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@208177 91177308-0d34-0410-b5e6-96231b3b80d8
* Always set alignment of vectorized LD/ST in SLP-Vectorizer. ↵Yi Jiang2014-05-05
| | | | | | <rdar://problem/16812145> git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207983 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Bring back the insertelement patch (r205965) with fixesArnold Schwaighofer2014-05-04
| | | | | | | | | | | | | | | | When can't assume a vectorized tree is rooted in an instruction. The IRBuilder could have constant folded it. When we rebuild the build_vector (the series of InsertElement instructions) use the last original InsertElement instruction. The vectorized tree root is guaranteed to be before it. Also, we can't assume that the n-th InsertElement inserts the n-th element into a vector. This reverts r207746 which reverted the revert of the revert of r205018 or so. Fixes the test case in PR19621. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207939 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Lazily allocate the map for block numbering.Benjamin Kramer2014-05-03
| | | | | | | | There is no point in creating it if we're not going to vectorize anything. Creating the map is expensive as it creates large values. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207916 91177308-0d34-0410-b5e6-96231b3b80d8
* Vectorize intrinsic math function calls in SLPVectorizer.Karthik Bhat2014-05-03
| | | | | | | | This patch adds support to recognize and vectorize intrinsic math functions in SLPVectorizer. Review: http://reviews.llvm.org/D3560 and http://reviews.llvm.org/D3559 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207901 91177308-0d34-0410-b5e6-96231b3b80d8
* Clean up constructor logic and member access for LoopVectorizeHints.Eric Christopher2014-05-02
| | | | | | | | | There are public functions that mutate various members as well as another private member already, so make all the members private to avoid the discontinuity and add accessors for the values. Should be no functional change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207868 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert r205965, which essentially reverts r205018 for the second time.Chandler Carruth2014-05-01
| | | | | | | | | | | | | | | | =[ Turns out that this was the root cause of PR19621. We found a crasher only recently (likely due to improvements elsewhere in the SLP vectorizer) but the reduced test case failed all the way back to here. I've confirmed that reverting this patch both fixes the reduced test case in PR19621 and the actual source file that led to it, so it seems to really be rooted here. I've replied to the commit thread with discussion of my (feeble) attempts to debug this. Didn't make it very far, so reverting now that we have a good test case so that things can get back to healthy while the debugging carries on. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207746 91177308-0d34-0410-b5e6-96231b3b80d8
* Add a <tuple> include to more files that aren't getting it transitively on MSVC.Benjamin Kramer2014-04-30
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207617 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix vectorization remarks.Diego Novillo2014-04-29
| | | | | | | | | This patch changes the vectorization remarks to also inform when vectorization is possible but not beneficial. Added tests to exercise some loop remarks. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207574 91177308-0d34-0410-b5e6-96231b3b80d8
* Continue slp vectorization even the BB already has vectorized store ↵Yi Jiang2014-04-29
| | | | | | radar://16641956 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207572 91177308-0d34-0410-b5e6-96231b3b80d8
* Add optimization remarks to the loop unroller and vectorizer.Diego Novillo2014-04-29
| | | | | | | | | | | | | | | Summary: This calls emitOptimizationRemark from the loop unroller and vectorizer at the point where they make a positive transformation. For the vectorizer, it reports vectorization and interleave factors. For the loop unroller, it reports all the different supported types of unrolling. Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D3456 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207528 91177308-0d34-0410-b5e6-96231b3b80d8
* [BUG] Fix -Wunused-variable warning in Release mode. Thnx to Kostya ↵Zinovy Nis2014-04-29
| | | | | | Serebryany for pointing. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207516 91177308-0d34-0410-b5e6-96231b3b80d8
* fix -Wunused-variable warning in Release modeKostya Serebryany2014-04-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207514 91177308-0d34-0410-b5e6-96231b3b80d8
* [OPENMP][LV][D3423] Respect Hints.Force meta-data for loops in LoopVectorizerZinovy Nis2014-04-29
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207512 91177308-0d34-0410-b5e6-96231b3b80d8
* [C++] Use 'nullptr'.Craig Topper2014-04-28
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207394 91177308-0d34-0410-b5e6-96231b3b80d8
* [C++] Use 'nullptr'. Transforms edition.Craig Topper2014-04-25
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207196 91177308-0d34-0410-b5e6-96231b3b80d8
* Allow vectorization of bit intrinsics in BB Vectorizer.Karthik Bhat2014-04-25
| | | | | | | | This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207174 91177308-0d34-0410-b5e6-96231b3b80d8
* Allow vectorization of few missed llvm intrinsic calls in BBVectorizor by ↵Karthik Bhat2014-04-24
| | | | | | handling them in isVectorizableIntrinsic function. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@207085 91177308-0d34-0410-b5e6-96231b3b80d8
* [LV] Statistics numbers for LoopVectorize introduced: a number of analyzed ↵Alexander Musman2014-04-23
| | | | | | | | | | | | | loops & a number of vectorized loops. Use -stats to see how many loops were analyzed for possible vectorization and how many of them were actually vectorized. Patch by Zinovy Nis Differential Revision: http://reviews.llvm.org/D3438 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@206956 91177308-0d34-0410-b5e6-96231b3b80d8
* [Modules] Fix potential ODR violations by sinking the DEBUG_TYPEChandler Carruth2014-04-22
| | | | | | | | | | | | | | | | | definition below all of the header #include lines, lib/Transforms/... edition. This one is tricky for two reasons. We again have a couple of passes that define something else before the includes as well. I've sunk their name macros with the DEBUG_TYPE. Also, InstCombine contains headers that need DEBUG_TYPE, so now those headers #define and #undef DEBUG_TYPE around their code, leaving them well formed modular headers. Fixing these headers was a large motivation for all of these changes, as "leaky" macros of this form are hard on the modules implementation. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@206844 91177308-0d34-0410-b5e6-96231b3b80d8
* D3348 - [BUG] "Rotate Loop" pass kills "llvm.vectorizer.enable" metadataAlexey Bataev2014-04-15
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@206266 91177308-0d34-0410-b5e6-96231b3b80d8
* Reapply "SLPVectorizer: Ignore users that are insertelements we can ↵Arnold Schwaighofer2014-04-10
| | | | | | | | | reschedule them" This commit reapplies 205018. After 205855 we should correctly vectorize intrinsics. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205965 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Only vectorize intrinsics whose operands are widened equallyArnold Schwaighofer2014-04-09
| | | | | | | | | The vectorizer only knows how to vectorize intrinics by widening all operands by the same factor. Patch by Tyler Nowicki! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205855 91177308-0d34-0410-b5e6-96231b3b80d8
* Add NDEBUG markers around debug only function.Eric Christopher2014-04-07
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205706 91177308-0d34-0410-b5e6-96231b3b80d8
* Add debug location information to the vectorizer debug statements.Eric Christopher2014-04-07
| | | | | | Patch by Zinovy Nis. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205705 91177308-0d34-0410-b5e6-96231b3b80d8
* Fixing typo.David Blaikie2014-04-05
| | | | | | Differential Revision: http://reviews.llvm.org/D3154 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205674 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: compare entire intrinsic for SLP compatibility.Tim Northover2014-04-02
| | | | | | | | | Some Intrinsics are overloaded to the extent that return type equality (all that's been checked up to now) does not guarantee that the arguments are the same. In these cases SLP vectorizer should not recurse into the operands, which can be achieved by comparing them as "Function *" rather than simply the ID. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205424 91177308-0d34-0410-b5e6-96231b3b80d8
* [LoopVectorizer] Count dependencies of consecutive pointers as uniformsHal Finkel2014-04-02
| | | | | | | | | | | | | | | | | | | | | For the purpose of calculating the cost of the loop at various vectorization factors, we need to count dependencies of consecutive pointers as uniforms (which means that the VF = 1 cost is used for all overall VF values). For example, the TSVC benchmark function s173 has: ... %3 = add nsw i64 %indvars.iv, 16000 %arrayidx8 = getelementptr inbounds %struct.GlobalData* @global_data, i64 0, i32 0, i64 %3 ... and we must realize that the add will be a scalar in order to correctly deduce it to be profitable to vectorize this on PowerPC with VSX enabled. In fact, all dependencies of a consecutive pointer must be a scalar (uniform), and so we simply need to add all consecutive pointers to the worklist that currently detects collects uniforms. Fixes PR19296. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205387 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "SLPVectorizer: Ignore users that are insertelements we can ↵Arnold Schwaighofer2014-03-31
| | | | | | | | | | | | | | reschedule them" This reverts commit r205018. Conflicts: lib/Transforms/Vectorize/SLPVectorizer.cpp test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll This is breaking libclc build. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205260 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Take credit for free extractelement instructionsArnold Schwaighofer2014-03-28
| | | | | | | | | Extract element instructions that will be removed when vectorzing lower the cost. Patch by Arch D. Robison! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205020 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix typosArnold Schwaighofer2014-03-28
| | | | | | Patch by Arch D. Robison! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205019 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Ignore users that are insertelements we can reschedule themArnold Schwaighofer2014-03-28
| | | | | | Patch by Arch D. Robison! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@205018 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP vectorizer: Don't hoist vector extracts of phis.Andrew Trick2014-03-25
| | | | | | | | | | | | | Extracts coming from phis were being hoisted, while all others were sunk to their uses. This was inconsistent and didn't seem to serve a purpose. Changing all extracts to be sunk to uses is a prerequisite for adding block frequency to the SLP vectorizer's cost model. I benchmarked the change in isolation (without block frequency). I only saw noise on x86 and some potentially significant improvements on ARM. No major regressions is good enough for me. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@204699 91177308-0d34-0410-b5e6-96231b3b80d8
* [LV] While I'm here, use range based for loops which are so much cleanerChandler Carruth2014-03-18
| | | | | | for this kind of walk. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@204188 91177308-0d34-0410-b5e6-96231b3b80d8
* [LV] The actual change I intended to commit in r204148. Sorry for theChandler Carruth2014-03-18
| | | | | | | | | | | | | noise. Original commit log: Replace some dead code with an assert. When I first ported this pass from a loop pass to a function pass I did so in the naive, recursive way. It doesn't actually work, we need a worklist instead. When I switched to the worklist I didn't delete the naive recursion. That recursion was also buggy because it was dead and never really exercised. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@204187 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-17 13:53:54 +0000