Query optimization - #670
Open
fstachura wants to merge 4 commits into
Open
Conversation
fstachura
force-pushed
the
query-optimization
branch
2 times, most recently
from
August 13, 2026 10:59
ce239c5 to
1a7374b
Compare
This property is not used anymore, the last reference was removed in 17726d7 (views: Move and refactor patch-forms). Computing it may execute an unnecessary count query.
Patch list is pretty slow partially because it filters on a query with a lot of extra fields. For example, I suspect that the tag count fields are computed even for rows that did not make it into the final result, same with joins. This patch changes this by separating the filtering stage, in which only patch IDs are queried, and the aggregation stage.
The new index uses project_id as the first key. This is important, as it seems that almost all web queries filter over project_id. The index also includes the submission date sorted descending, as this is the default sorting key for the patches list.
Allowing the users to set an arbitrary number of items per page may cause excessive load on the database. For example, a request to the patch list has to compute the number of tags for each listed patch.
fstachura
force-pushed
the
query-optimization
branch
from
August 13, 2026 12:39
1a7374b to
55bdbad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that queries made by Patchwork can be pretty slow on larger databases.
This PR changes two larger things that seemed to cause performance issues.
I split the query in generic_list into a filtering part and an aggregation part. In the first part, only patch ids are queried. Then, another query is made to retrieve more detailed patch data. I'm not 100% sure, but it seems that MySQL and Postgres may compute all the fields (including JOINs) even for rows that don't make it into the final result due to LIMIT/OFFSET which negatively impacts performance.
I added a new index. The first three fields of the index are fields that the default patch view uses to filter patches - project_id, archived and state. The fourth field is the field used for the default order - date, sorted descending (unlike the current covering index). I'm 100% sure that project_id should be the first field in the index, and that date should be after all the fields that are most commonly used in filters.
In my tests, I have noticed substantial performance improvements on large databases (>1 million patches), on a low-end machine (~2018-laptop tier, 16GB RAM, 2 cores, 2 threads, SSD) on both Postgres and MySQL. I can share more details about my tests, but to keep it short: on main, some queries would take 5+ seconds (average on MySQL was 2 seconds, on Postgres some would cause a timeout), with these patches the average is <300 ms. (I tested with default date descending ordering, on 5 scenarios: default filters, filtering by submitter, delegate, submitter and delegate, state. I did not do stress-tests, just "one-shot"). Query plans also became shorter, in some cases were replaced by straightforward index lookup.
https://dev.mysql.com/doc/refman/8.4/en/order-by-optimization.html