JS: Improve performance of ATM queries on large databases - #7475
Draft
adityasharad wants to merge 8 commits into
Draft
JS: Improve performance of ATM queries on large databases#7475adityasharad wants to merge 8 commits into
adityasharad wants to merge 8 commits into
Conversation
One of the heuristics for test files looks for source files of the form `base.ext`, then looks for sibling test files of the form `base.test.ext` or `base.spec.ext`. On large databases, the result join order computed all source files, the containers of those files, then all other files within those containers, before computing the test file names and filtering using those names. The product of all files with all other files in the same containers is of the same order of magnitude as the product of the `files` table with itself, which on large DBs like Node can be 12M+ tuples. As a performance optimisation, factor out a helper predicate that computes the likely test file names for each source file, so these can be determined with a single join against the files table. This results in much better join orders, such as computing the set of files and their containers, then the test file names, then the sibling files with those names. This loses some flexibility because the set of 'test' extension names is hardcoded in the library rather than provided by the caller predicate. The original predicate remains to avoid breaking other callers, but could eventually be deprecated.
When join-ordering and evaluating this conjunction, it is preferable to start with the relatively small set of `sanitizer` calls, then compute the set of SSA variables accessed as the arguments of those sanitizer calls, then reason about how those variables are used in phi nodes. Use directional binding pragmas to encourage this join order by picking `sanitizer` first, and discourage picking the opposite join order starting with `phi`. This impacts performance of the ATM XSS queries on large databases like Node, where computing all variable accesses from phi nodes leads to 435M+ tuples.
adityasharad
force-pushed
the
atm/perf-debugging-node
branch
from
December 22, 2021 20:13
d4bb4c8 to
6da4aa0
Compare
No behaviour change.
On databases with a large number of Exprs, it can be better to start with the set of route handlers, then find their response headers, then find the expression values set in those headers.
adityasharad
force-pushed
the
atm/perf-debugging-node
branch
from
December 23, 2021 00:10
6da4aa0 to
eb66af3
Compare
Factor the regex-independent logic of `isReadFrom` into its own predicate. Call this predicate directly from `isNumeric`, which doesn't have much restrictive context on the set of starting nodes. Use a binding hint to discourage starting with all expr nodes in this case. Other callers may have more restrictive context on the set of nodes, so they are not changed.
Should most likely refer to `AccessPathWrite` and look for write nodes. This also improves the performance of `rankedAccessPath`, since the set of candidate blocks is now limited to blocks with both a read and a write.
We have to look up the node index within the block anyway, so include it as an aggregation variable.
Check that the read node is in a *reachable* basic block before looking for a dominating write block.
adityasharad
force-pushed
the
atm/perf-debugging-node
branch
from
December 23, 2021 00:11
eb66af3 to
b43891a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress.