summaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize/BBVectorize.cpp
Commit message (Collapse)AuthorAge
* Use more type helper functionsMatt Arsenault2013-10-21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@193109 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix code duplicationMatt Arsenault2013-10-01
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191716 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix spelling intruction -> instruction.Robert Wilhelm2013-09-28
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191610 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Add initial stores to the write set when tracking usesHal Finkel2013-08-13
| | | | | | | | | | | | | | | | When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188329 91177308-0d34-0410-b5e6-96231b3b80d8
* Use SmallVectorImpl& instead of SmallVector to avoid repeating small vector ↵Craig Topper2013-07-14
| | | | | | size. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186274 91177308-0d34-0410-b5e6-96231b3b80d8
* Use SmallVectorImpl::iterator/const_iterator instead of SmallVector to avoid ↵Craig Topper2013-07-04
| | | | | | specifying the vector size. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@185606 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Fixup debugging statementsHal Finkel2013-03-10
| | | | | | | After the recent data-structure improvements, a couple of debugging statements were broken (printing pointer values). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176791 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Fix an invalid reference bugHal Finkel2013-02-17
| | | | | | | | | | | | | | This fixes PR15289. This bug was introduced (recently) in r175215; collecting all std::vector references for candidate pairs to delete at once is invalid because subsequent lookups in the owning DenseMap could invalidate the references. bugpoint was able to reduce a useful test case. Unfortunately, because whether or not this asserts depends on memory layout, this test case will sometimes appear to produce valid output. Nevertheless, running under valgrind will reveal the error. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175397 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Call a DAG and DAG instead of a treeHal Finkel2013-02-15
| | | | | | | | | | Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175278 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Cap the number of candidate pairs in each instruction groupHal Finkel2013-02-15
| | | | | | | | | | | | | | | | | | | | | | | For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175251 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Remove the remaining instances of std::multimapHal Finkel2013-02-14
| | | | | | | | | | All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175216 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Don't store candidate pairs in a std::multimapHal Finkel2013-02-14
| | | | | | | | | | This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175215 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Don't over-search when building the dependency mapHal Finkel2013-02-11
| | | | | | | | | | | | | When building the pairable-instruction dependency map, don't search past the last pairable instruction. For large blocks that have been divided into multiple instruction groups, searching past the last instruction in each group is very wasteful. This gives a 32% speedup on the csa.ll test case from PR15222 (when using 50 instructions in each group). No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174915 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Omit unnecessary entries in PairableInstUsersHal Finkel2013-02-11
| | | | | | | | | | | This map is queried only for instructions in pairs of pairable instructions; so make sure that only pairs of pairable instructions are added to the map. This gives a 3.5% speedup on the csa.ll test case from PR15222. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174914 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Eliminate one more restricted linear searchHal Finkel2013-02-11
| | | | | | | | | | This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174893 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Remove the linear searches from pair connection searchingHal Finkel2013-02-11
| | | | | | | | | This removes the last of the linear searches over ranges of std::multimap iterators, giving a 7% speedup on the doduc.bc input from PR15222. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174859 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Avoid linear searches within the load-move setHal Finkel2013-02-11
| | | | | | | | | This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174858 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: isa/cast cleanup in getInstructionTypesHal Finkel2013-02-11
| | | | | | | | | | Profiling suggests that getInstructionTypes is performance-sensitive, this cleans up some double-casting in that function in favor of using dyn_cast. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174857 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Make the bookkeeping to support full cycle checking less expensiveHal Finkel2013-02-11
| | | | | | | | | | | By itself, this does not have much of an effect, but only because in the default configuration the full cycle checks are used only for small problem sizes. This is part of a general cleanup of uses of iteration over std::multimap ranges only for the purpose of checking membership. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174856 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Use TTI->getAddressComputationCostHal Finkel2013-02-08
| | | | | | | | | | | | | This is a follow-up to the cost-model change in r174713 which splits the cost of a memory operation between the address computation and the actual memory access. In r174713, this cost is always added to the memory operation cost, and so BBVectorize will do the same. Currently, this new cost function is used only by ARM, and I don't have any ARM test cases for BBVectorize. Assistance in generating some good ARM test cases for BBVectorize would be greatly appreciated! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174743 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Better use of TTI->getShuffleCostHal Finkel2013-01-27
| | | | | | | | | | | | | | | | | | | | When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173621 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Add a additional comment about the cost computationHal Finkel2013-01-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173580 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Fix anomalous capital letter in commentHal Finkel2013-01-26
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173579 91177308-0d34-0410-b5e6-96231b3b80d8
* Switch BBVectorize to directly depend on having a TTI analysis.Chandler Carruth2013-01-07
| | | | | | | | | | | | | This could be simplified further, but Hal has a specific feature for ignoring TTI, and so I preserved that. Also, I needed to use it because a number of tests fail when switching from a null TTI to the NoTTI nonce implementation. That seems suspicious to me and so may be something that you need to look into Hal. I worked it by preserving the old behavior for these tests with the flag that ignores all target info. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171722 91177308-0d34-0410-b5e6-96231b3b80d8
* Move TargetTransformInfo to live under the Analysis library. This noChandler Carruth2013-01-07
| | | | | | | longer would violate any dependency layering and it is in fact an analysis. =] git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171686 91177308-0d34-0410-b5e6-96231b3b80d8
* Switch the BB vectorizer from the VTTI interface to the simple TTIChandler Carruth2013-01-05
| | | | | | interface. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171618 91177308-0d34-0410-b5e6-96231b3b80d8
* Move all of the header files which are involved in modelling the LLVM IRChandler Carruth2013-01-02
| | | | | | | | | | | | | | | | | | | | | into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171366 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Use VTTI to compute costs for intrinsics vectorizationHal Finkel2012-12-26
| | | | | | | | | | | | For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171079 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Enable vectorization of the fmuladd intrinsicHal Finkel2012-12-25
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171075 91177308-0d34-0410-b5e6-96231b3b80d8
* BB-Vectorizer: Check the cost of the store pointer typeNadav Rotem2012-12-21
| | | | | | | | | and not the return type, which is void. A number of test cases fail after adding the assertion in TTImpl. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170828 91177308-0d34-0410-b5e6-96231b3b80d8
* Use the new script to sort the includes of every file under lib.Chandler Carruth2012-12-03
| | | | | | | | | | | | | | | | | Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169131 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Correctly merge SubclassOptionalDataHal Finkel2012-11-28
| | | | | | | When two instructions are combined into a vector instruction, the resulting instruction must have the most-conservative flags. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168765 91177308-0d34-0410-b5e6-96231b3b80d8
* Replace std::vector -> SmallVector in BBVectorizeHal Finkel2012-11-14
| | | | | | | | | For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167966 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix the largest offender of determinism in BBVectorizeHal Finkel2012-11-14
| | | | | | | | | | | | | | | | | Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167942 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Remove temporary assert used for debuggingHal Finkel2012-11-13
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167817 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Don't vectorize vector-manipulation chainsHal Finkel2012-11-13
| | | | | | | | Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167811 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Only some insert element operand pairs are free.Hal Finkel2012-11-12
| | | | | | | | This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167784 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Use a more sophisticated check for input costHal Finkel2012-11-12
| | | | | | | | | The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167750 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Check the types of compare instructionsHal Finkel2012-11-12
| | | | | | | The pass would previously assert when trying to compute the cost of compare instructions with illegal vector types (like struct pointers). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167743 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Check the input types of shuffles for legalityHal Finkel2012-11-12
| | | | | | | | | | This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167731 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Use target costs for incoming and outgoing values instead of ↵Hal Finkel2012-11-01
| | | | | | | | | | | the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167256 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Account for internal shuffle costsHal Finkel2012-11-01
| | | | | | | | | | | | | When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167217 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Choose pair ordering to minimize shufflesHal Finkel2012-10-31
| | | | | | | | | | | | | | | | | | | | BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167122 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Cache fixed-order pairs instead of recomputing pointer info.Hal Finkel2012-10-30
| | | | | | | | | | | | Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167049 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Fix a small bug introduced in r167042.Hal Finkel2012-10-30
| | | | | | | We need to make sure that we take the correct load/store alignment when the inputs are flipped. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167044 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Simplify how input swapping is handled.Hal Finkel2012-10-30
| | | | | | | | | Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167042 91177308-0d34-0410-b5e6-96231b3b80d8
* BBVectorize: Don't make calls to SE when the result is unused.Hal Finkel2012-10-30
| | | | | | | SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167037 91177308-0d34-0410-b5e6-96231b3b80d8
* Update BBVectorize to use the new VTTI instr. cost interfaces.Hal Finkel2012-10-27
| | | | | | | | The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@166865 91177308-0d34-0410-b5e6-96231b3b80d8
* Use VTTI->getNumberOfParts in BBVectorize.Hal Finkel2012-10-26
| | | | | | This change reflects VTTI refactoring; no functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@166752 91177308-0d34-0410-b5e6-96231b3b80d8
* Disable generation of pointer vectors by BBVectorize.Hal Finkel2012-10-26
| | | | | | Once vector-of-pointer support works, then this can be reverted. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@166741 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-29 12:56:32 +0000