Allow the user to control the process of loading grammars & themes

[tom-sherman]

The current behaviour is a good default as it mostly just works across all environments. However there are cases where I want to be in control of how/when the grammars and themes are loaded. In general: as a user I want to tightly control the IO operations of the library.

Consider the following use cases:

• I'm running lighter in an environment without network access, I instead want to choose to read the grammar/theme from the filesystem or other async IO
• I'm running lighter without access to a filesystem or any other kind of IO, instead I want to read the grammar/theme from memory
• I'm running lighter in an environment where I want to customise the fetch call. eg. in Next.js I want to pass next: { revalidate: ... }

[pomber]

We could add a couple of functions to the highlight config:

type Config = { 
  loadRawGrammar: (lang: {...}) => Promise<IRawGrammar>
  loadRawTheme: (name: string) => Promise<RawTheme>
};

But I'm not sure it will solve your use case, can you give me a more concrete example of your first two points?

---

For the Next.js case, would it make sense to add a good revalidate default in Lighter?

Also, isn't Next.js default to cache the fetch forever? Or does it need { next: { revalidate: false } }?

[tom-sherman]

can you give me a more concrete example of your first two points?

Test environments are a common one, also deno without requiring permissions. In general syntax highlighting is a pure function, it's strange to me to require a network call without the ability to swap this out.

Additionally this would allow for loading arbitrary themes and grammars not found in the standard set.

For the Next.js case, would it make sense to add a good revalidate default in Lighter?

I'd advise strongly against that. revalidate is a cost-incurring option, a library shouldn't make a default choice that incurs extra costs IMO.

Also, isn't Next.js default to cache the fetch forever?

That's correct, but this optimisation is skipped when setting export const revalidate = N on the route segment - in this case the grammars and themes will be refetched every N seconds - that's what was happening in code-hike/bright#21

[pomber]

If I add { next: { revalidate: false } } to the fetchs in lighter, that shouldn't incur in extra costs, right?

[tom-sherman]

Apologies I misread. As far as I'm aware that configuration is a complete no-op. It's equivalent to not passing revalidate to fetch at all.

[pomber]

Apologies I misread. As far as I'm aware that configuration is a complete no-op. It's equivalent to not passing revalidate to fetch at all.

Wouldn't the revalidate: false on fetch override the revalidate: N on the page? So when the page revalidates the fetch doesn't. That should fix code-hike/bright#21.

I'm just asking, I expect it to work like that but I have no idea.

[wesbos]

I found out while trying to present in front of a 1800 person conference that this package loads all the grammers from remote URLs 😆

Lucky I just had to turn the wifi on and it fixed itself, but I would love to be able to load all these grammars locally. Or at least a web worker to cache them?

[pomber]

I found out while trying to present in front of a 1800 person conference that this package loads all the grammers from remote URLs 😆

Ups, sorry about that.

How are you using it? (any bundler? framework? directly from the browser?) Lighter tries to load the grammars from the file system (node modules) first, and only goes to the network if that fails.

[tom-sherman]

Wouldn't the revalidate: false on fetch override the revalidate: N on the page?

AFAIA no. The page value takes precedence.