Skip to content

Speed up parseMod by pre-filtering patterns - #10216

Draft
vaisest wants to merge 1 commit into
PathOfBuildingCommunity:devfrom
vaisest:speedy-modparser
Draft

Speed up parseMod by pre-filtering patterns#10216
vaisest wants to merge 1 commit into
PathOfBuildingCommunity:devfrom
vaisest:speedy-modparser

Conversation

@vaisest

@vaisest vaisest commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Description of the problem being solved:

This is an idea PR of a way to speed up parseMod() a lot. This brings the time to generate a modcache containing the item db mods, and all item mods data.itemMods from around 18 sec to around 2 sec for the first working version of this. It's definitely possible to be many times faster.

This is more of a solution looking for a problem and was mostly done out of curiosity

I think this is somewhat similar to e.g. what the Rust regex crate or Intel hyperscan etc do. The idea is to:

  1. Pre-calculate the longest literal part of every pattern
  2. Use aho-corasick with those literals to find what patterns can match the line
  3. Continue with old parsing code, just with fewer candidates

The used aho-corasick implementation was https://gist.github.com/Validark/d493cfd1b3425c2e3073f5ccd08fbeb9, which is the first pure lua implementation I found. I have zero clue if it's slow or not.

Steps taken to verify a working solution:

Link to a build that showcases this PR:

Before screenshot:

After screenshot:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant