WHY A LANGUAGE PLUGIN NEEDS A GRAMMAR, NOT A REGEX

A regex can highlight a language for about as long as nobody nests anything. The moment a string contains a comment marker, or a comment contains something that looks like a string, or a block comment nests inside another block comment the way Julia’s #= ... =# is allowed to, the pattern either lights up the wrong span or gives up entirely. A regex is the cheap first instinct for a language plugin, and it means you will rewrite the highlighter twice.
Two generated files, not one
On the IntelliJ Platform, the standard tool for this is Grammar-Kit. The workflow is: write a .bnf grammar describing the language’s rules, check it against the live preview as you go, generate the parser, the element types, and the PSI classes from it, then separately generate a JFlex lexer, then wire both into a ParserDefinition that tells the platform how to turn a file into tokens and a tree.
Flexible Julia, my own JetBrains plugin, bundles two of these, completely independent of each other. JuliaLexer.java and JuliaParser.java sit under src/main/gen for Julia itself. Next to them, unrelated, sit DyadLexer.java and DyadParser.java, generated from a second .bnf grammar for Dyad, JuliaHub’s separate modeling language, whose .dyad files the plugin also supports and which Dyad.jl compiles down to ordinary Julia. Two languages, two grammars, two generated lexers, two generated parsers, sharing nothing but a plugin.xml.
Why there are two trees, not one
Here is the part that took me longer to internalize than I expected. Parsing on the IntelliJ Platform is a two-step process. The generated parser first builds an AST: a raw, generic tree that mirrors exactly what the grammar rules matched, node by node, with no idea yet what any of it means. Then a second tree, the PSI, gets built on top of that AST, and the PSI is where every element actually knows what it is: this node is a function definition, this one is a macro call, this identifier resolves to that other declaration three files away.
The reason for the split is that the two trees are optimized for opposite things. The AST has to be cheap to produce, because it gets rebuilt on practically every keystroke, and it has no business knowing what a symbol resolves to. The PSI is the expensive, typed, IDE-facing layer: it is what completion, navigation, refactoring, and find-usages actually walk, and it is allowed to be slower and smarter because it is not rebuilt from scratch on every character you type. A regex highlighter conflates both jobs into one pass and does neither well. A grammar-based plugin keeps them apart on purpose, and that separation is the entire reason “go to definition” on a Julia function works instead of being a fuzzy string search wearing a keyboard shortcut.
The grammar and its generated code
A slice of the actual Julia grammar, the rule for what counts as an expression:
expr ::=
compactFunction
| constGlobalStatement
| globalStatement
| qualifiedMacroOp
| applyMacroOp
| assignLevel
| arrowOp
| ternaryOp
| quoteLevel
| lambda
| ...
| primaryExpr {
implements=['com.ilscipio.language.julia.psi.impl.IJuliaExpr']
mixin='com.ilscipio.language.julia.psi.impl.JuliaExprMixin'
}
That reads almost like documentation of Julia’s own precedence table, because that is exactly what it is: every alternative is a level of the language’s grammar, ordered the way the reference parser orders them. Grammar-Kit turns this directly into recursive-descent parsing code.
Right above that rule in my own grammar file sits a comment I left for myself: the generated JuliaParser.java has two manual performance edits in it, a lookahead guard on compactFunction and a token-type dispatch that skips alternatives the current token can’t possibly start, and both are hand-written directly into generated code that Grammar-Kit would happily overwrite the next time I regenerate. Regenerating is no longer free: it means remembering to reapply the two edits I made in March.
| Education | EBS Business School, SFSU, HSG |
| Experience | 20 Years Enterprise Systems |
| Built | Scipio ERP - ERP & Ecommerce System |
| Author | "Start/Up - Set the net on fire" |
| JetBrains | 5x Plugins Published |
| Languages | Java, Python, JS, TS, Rust, Go, Swift, PHP |
| Frontend | React, Vue, Angular, HTML/CSS |
| Databases | PostgreSQL, MySQL, MongoDB |
| Caching | Redis, Memcached |
| Enterprise | L'Oréal, Amadeus, Konica Minolta, Orange |
| Digital | 1&1 Internet, Sedo.com, Creditsafe |
| Public | DRK, Kanton Basel, Aktion Mensch, Börsenverein |