Describe this repository

Filter values that only differ in case, like `Mint` and `mint`, came
through as separate values. The new `--lowercase-filters` option
(`lowercase_filters`, `PAGEFIND_LOWERCASE_FILTERS`, and
`lowercaseFilters` / `lowercase_filters` in the Node and Python
wrappers) lowercases filter values when indexing and merges values on
a page that become equal. Filter names are left as they are. Off by
default.

Fixes #737

README.md

@@ -0,0 +1,10 @@
+# Contributions to Pagefind/pagefind
+
+Static low-bandwidth search at scale
+
+This repository holds only my commits to the upstream project, with their original author, date and diff.
+Each of my commits is preceded by a snapshot of the upstream files it touched, so every diff shows exactly what I changed.
+
+## Merged pull requests
+
+- [fix(pagefind): skip exclude selectors that cannot be parsed](https://github.com/Pagefind/pagefind/pull/1312) (2026-09-13)

docs/content/docs/components/results.md

@@ -0,0 +1,161 @@
+---
+title: "<pagefind-results>"
+nav_title: "<pagefind-results>"
+nav_section: Components
+weight: 20
+---
+
+Displays search results with automatic lazy loading. Results load their full data only when scrolled into view.
+
+<div class="demo-box demo-box-attached" style="flex-direction: column; gap: 1rem; height: 400px;">
+<pagefind-input instance="results-demo"></pagefind-input>
+<div class="results-scroll" style="flex: 1; overflow-y: auto; width: 100%; min-height: 0;">
+<pagefind-results instance="results-demo"></pagefind-results>
+</div>
+<pagefind-keyboard-hints instance="results-demo"></pagefind-keyboard-hints>
+</div>
+
+```html
+<pagefind-results></pagefind-results>
+```
+
+## Attributes
+
+| Attribute | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `show-images` | boolean | `false` | Show/hide result images |
+| `hide-sub-results` | boolean | `false` | Hide anchor-based sub-results |
+| `max-sub-results` | number | `3` | Maximum number of sub-results per result |
+| `max-results` | number | unlimited | Limit number of results shown |
+| `link-target` | string | — | Set `target` attribute on result links (e.g., `_blank` for new tab) |
+| `instance` | string | `"default"` | Connect to a specific Pagefind instance |
+
+## Keyboard Navigation
+
+When results are focused, use these keys to navigate:
+
+| Key | Action |
+|-----|--------|
+| `↓` / `↑` | Navigate between results |
+| `Enter` | Follow the selected link |
+| `Backspace` | Focus input and delete last character |
+| `/` | Focus the search input |
+| Any letter | Focus input and type the character |
+
+## Custom Templates
+
+Customize result rendering using `<script type="text/pagefind-template">` elements inside the component. Templates use a simple syntax with variables, conditionals, and loops.
+
+### Basic Template
+
+```html
+<pagefind-results>
+  <script type="text/pagefind-template">
+    <li class="my-result">
+      <h3>{{ meta.title }}</h3>
+      <a href="{{ url | safeUrl }}">{{ url }}</a>
+      <p>{{+ excerpt +}}</p>
+    </li>
+  </script>
+</pagefind-results>
+```
+
+Your template must have a single root element (e.g. a single `<li>`). Keyboard navigation moves between `<a>` elements in the results list, so ensure your main result link is an `<a>` tag.
+
+The full templating syntax can be seen in the [adequate-little-templates](https://github.com/bglw/adequate-little-templates?tab=readme-ov-file#syntax) package.
+
+### Available Data
+
+Templates receive this data structure:
+
+| Field | Description |
+|-------|-------------|
+| `meta.title` | Page title |
+| `meta.url` | Page URL |
+| `meta.image` | Page image (if set) |
+| `meta.image_alt` | Image alt text |
+| `meta.*` | Any custom metadata |
+| `excerpt` | Search excerpt with `<mark>` highlighting (use `{{+ excerpt +}}`) |
+| `url` | Fallback URL |
+| `sub_results` | Array of matching sections within the page |
+| `options.*` | The attributes supplied to the `<pagefind-results>` element |
+
+Each sub-result has: `title`, `url`, `excerpt`.
+
+### Template Filters
+
+The template engine provides [built-in filters](https://github.com/bglw/adequate-little-templates?tab=readme-ov-file#syntax). The Component UI registers the following additional filters:
+
+#### `resolveUrl(pageUrl)`
+
+Resolves a relative URL against a page URL. Absolute URLs are returned unchanged. If the page URL includes a domain, for example from a `baseUrl` on a merged index, the resolved URL keeps that domain.
+
+```
+"images/hero.png" | resolveUrl("/blog/post/")  →  /blog/post/images/hero.png
+"images/hero.png" | resolveUrl("https://example.com/blog/post/")  →  https://example.com/blog/post/images/hero.png
+```
+
+### Placeholder Template
+
+While results are loading, a skeleton placeholder is shown. You can customize it with a second template using `data-template="placeholder"`, as shown in the full example below. The placeholder template receives no data — it is static HTML.
+
+### Full Template Example
+
+Here's the built-in template. Copy and customize it:
+
+```html
+<pagefind-results>
+  <script type="text/pagefind-template">
+    <li class="pf-result">
+      <div class="pf-result-card">
+        {{#if and(options.show_images, meta.image)}}
+        <img class="pf-result-image" src="{{ meta.image | resolveUrl(meta.url | default(url)) }}" alt="{{ meta.image_alt | default(meta.title) }}">
+        {{/if}}
+        <div class="pf-result-content">
+          <p class="pf-result-title">
+            <a class="pf-result-link" href="{{ meta.url | default(url) | safeUrl }}"{{#if options.link_target}} target="{{ options.link_target }}"{{/if}}{{#if eq(options.link_target, "_blank")}} rel="noopener"{{/if}}>{{ meta.title }}</a>
+          </p>
+          {{#if excerpt}}
+          <p class="pf-result-excerpt">{{+ excerpt +}}</p>
+          {{/if}}
+        </div>
+      </div>
+      {{#if sub_results}}
+      <ul class="pf-heading-chips">
+        {{#each sub_results as sub}}
+        <li class="pf-heading-chip">
+          <a class="pf-heading-link" href="{{ sub.url | safeUrl }}"{{#if options.link_target}} target="{{ options.link_target }}"{{/if}}{{#if eq(options.link_target, "_blank")}} rel="noopener"{{/if}}>{{ sub.title }}</a>
+          <p class="pf-heading-excerpt">{{+ sub.excerpt +}}</p>
+        </li>
+        {{/each}}
+      </ul>
+      {{/if}}
+    </li>
+  </script>
+  <script type="text/pagefind-template" data-template="placeholder">
+    <li class="pf-result" aria-hidden="true">
+      <div class="pf-result-card">
+        <div class="pf-skeleton pf-skeleton-image"></div>
+        <div class="pf-result-content">
+          <p class="pf-result-title pf-skeleton pf-skeleton-title"></p>
+          <p class="pf-result-excerpt pf-skeleton pf-skeleton-excerpt"></p>
+        </div>
+      </div>
+    </li>
+  </script>
+</pagefind-results>
+```
+
+## Results Layout
+
+By default, results display in a single column. You can change the layout of the results container using CSS variables, for example a grid layout:
+
+```css
+:root {
+    --pf-results-display: grid;
+    --pf-results-columns: repeat(auto-fill, minmax(250px, 1fr));
+    --pf-results-gap: 16px;
+}
+```
+
+See [CSS Variables](/docs/css-variables/) for all available layout variables.

docs/content/docs/config-options.md

@@ -71,30 +71,6 @@ Note that currently Pagefind only supports lists of options via configuration fi
 |---------------------------|------------------------------|---------------------|
 | `--exclude-selectors <S>` | `PAGEFIND_EXCLUDE_SELECTORS` | `exclude_selectors` |
 
-### Rules
-Applies Pagefind attributes to the elements a selector matches, so a site can be configured with the same vocabulary its HTML would use. For example, in `pagefind.yml`:
-
-```yml
-rules:
-  - selector: "#my_navigation"
-    attributes:
-      data-pagefind-ignore: all
-  - selector: ".article-body"
-    attributes:
-      data-pagefind-body: ""
-      data-pagefind-meta: "title[data-title]"
-```
-
-Each rule behaves exactly as if the attributes had been written on the matching elements, so the documentation for [indexing attributes](/docs/indexing/) applies unchanged.
-
-An attribute written in the page always wins over a rule setting the same attribute, so rules never override markup you control. A selector Pagefind cannot parse is reported and skipped, and indexing continues without it.
-
-Rules are only available via configuration files, as a list of selectors with their attributes does not fit a CLI flag or an environment variable.
-
-| CLI Flag | ENV Variable | Config Key |
-|----------|--------------|------------|
-| n/a      | n/a          | `rules`    |
-
 ### Include characters
 Prevents Pagefind from stripping the provided characters when indexing content.
 Allows users to search for words including these characters.
@@ -145,6 +121,15 @@ This defaults to false, so playground files are not written to your live site. P
 |----------------------|--------------------|--------------------|
 | `--write-playground` | `PAGEFIND_WRITE_PLAYGROUND` | `write_playground` |
 
+### Lowercase filters
+Lowercases all filter values when indexing, so that values differing only in case are merged. With this option, `<span data-pagefind-filter="garnish">Mint</span>` and `<span data-pagefind-filter="garnish">mint</span>` both come through as the `mint` value of the `garnish` filter. Filter names are left as they are.
+
+Filter values passed to Pagefind search should then be lowercase too, and any UI that displays them will show the lowercased values.
+
+| CLI Flag              | ENV Variable                 | Config Key          |
+|-----------------------|------------------------------|---------------------|
+| `--lowercase-filters` | `PAGEFIND_LOWERCASE_FILTERS` | `lowercase_filters` |
+
 ### Verbose
 Prints extra logging while indexing the site. Only affects the CLI, does not impact web-facing search.
 

docs/content/docs/multisite.md

@@ -0,0 +1,206 @@
+---
+title: "Searching multiple sites"
+nav_title: "Searching multiple sites"
+nav_section: Searching
+weight: 80
+---
+
+Pagefind can be configured to search across multiple sites, merging results and filters into a single response. Multisite search configuration happens entirely in the browser, by pointing one Pagefind instance at multiple search bundles.
+
+The following examples reflect Pagefind running on a website at `blog.example.com` that wants to include pages from `docs.example.com` in the search results. _(Note that in a matching setup across domains, you will need to configure CORS headers. See [Cross origin indexes](#cross-origin-indexes) below.)_
+
+## Searching additional sites from the Component UI
+
+When configuring the Component UI, include a `mergeIndex` option with an array of additional indexes to merge into the main index. The URL should be the path to a pagefind bundle folder.
+
+{{< diffcode >}}
+```js
+// Running on blog.example.com
+
+const { configureInstance } = window.PagefindComponents;
+
++configureInstance("default", {
++    mergeIndex: [{
++        bundlePath: "https://docs.example.com/pagefind"
++    }]
++});
+```
+{{< /diffcode >}}
+
+Pagefind options can be passed to the additional indexes alongside the `bundlePath`:
+
+{{< diffcode >}}
+```js
+// Running on blog.example.com
+
+const { configureInstance } = window.PagefindComponents;
+
+configureInstance("default", {
++    // ... options for the blog.example.com index
+    mergeIndex: [{
+        bundlePath: "https://docs.example.com/pagefind",
++        // ... options for the docs.example.com index
+    }]
+});
+```
+{{< /diffcode >}}
+
+## Searching additional sites from the Pagefind JS API
+
+Using an initialized instance of Pagefind, await the `mergeIndex` function to add an additional index. The URL should be the path to a pagefind bundle folder.
+
+{{< diffcode >}}
+```js
+// Running on blog.example.com
+
+const pagefind = await import("/pagefind/pagefind.js");
++await pagefind.mergeIndex("https://docs.example.com/pagefind");
+```
+{{< /diffcode >}}
+
+Pagefind options can be passed to the additional index as a second argument:
+
+{{< diffcode >}}
+```js
+// Running on blog.example.com
+
+const pagefind = await import("/pagefind/pagefind.js");
++await pagefind.options({/* ... options for the blog.example.com index */});
+await pagefind.mergeIndex(
+  "https://docs.example.com/pagefind",
++  {/* ... options for the docs.example.com index */}
+);
+```
+{{< /diffcode >}}
+
+## Changing the weighting of individual indexes
+
+When searching across multiple sites you may want to rank each index higher or lower than the others. This can be achieved by passing an `indexWeight` option for each index:
+
+{{< diffcode >}}
+```js
+// Component UI:
+const { configureInstance } = window.PagefindComponents;
+
+configureInstance("default", {
++    indexWeight: 2,
+    mergeIndex: [{
+        bundlePath: "https://docs.example.com/pagefind",
++        indexWeight: 0.5
+    }]
+});
+
+// JS API:
+const pagefind = await import("/pagefind/pagefind.js");
++await pagefind.options({ indexWeight: 2 });
+await pagefind.mergeIndex("https://docs.example.com/pagefind", {
++    indexWeight: 0.5
+});
+```
+{{< /diffcode >}}
+
+## Filtering results by index
+
+When searching across multiple sites you may want to filter to each index, without having to tag every page on each site with the filter. This can be achieved with the `mergeFilter` option on each index:
+
+{{< diffcode >}}
+```js
+// Component UI:
+const { configureInstance } = window.PagefindComponents;
+
+configureInstance("default", {
++    mergeFilter: {
++        resource: "Blog"
++    },
+    mergeIndex: [{
+        bundlePath: "https://docs.example.com/pagefind",
++        mergeFilter: {
++            resource: "Documentation"
++        }
+    }]
+});
+
+// JS API:
+const pagefind = await import("/pagefind/pagefind.js");
++await pagefind.options({ mergeFilter: { resource: "Blog" } });
+await pagefind.mergeIndex("https://docs.example.com/pagefind", {
++    mergeFilter: {
++        resource: "Documentation"
++    }
+});
+```
+{{< /diffcode >}}
+
+## Merging a specific language index
+
+Pagefind will attempt to grab a matching language when merging an index, falling back to the default language for that index. You can change this behavior by passing a `language` option:
+
+{{< diffcode >}}
+```js
+// Component UI:
+const { configureInstance } = window.PagefindComponents;
+
+configureInstance("default", {
+    mergeIndex: [{
+        bundlePath: "https://docs.example.com/pagefind",
++        language: "pt-br"
+    }]
+});
+
+// JS API:
+const pagefind = await import("/pagefind/pagefind.js");
+await pagefind.mergeIndex("https://docs.example.com/pagefind", {
++    language: "pt-br"
+});
+```
+{{< /diffcode >}}
+
+## Skipping indexes that fail to load
+
+By default, a merged index that can't be loaded stops search from working. If some of your indexes might be missing, for example while one of the sites hasn't been built yet, pass an `optional` option for those indexes. Pagefind will log a warning and search across the indexes that did load:
+
+{{< diffcode >}}
+```js
+// Component UI:
+const { configureInstance } = window.PagefindComponents;
+
+configureInstance("default", {
+    mergeIndex: [{
+        bundlePath: "https://docs.example.com/pagefind",
++        optional: true
+    }]
+});
+
+// JS API:
+const pagefind = await import("/pagefind/pagefind.js");
+await pagefind.mergeIndex("https://docs.example.com/pagefind", {
++    optional: true
+});
+```
+{{< /diffcode >}}
+
+## Notes
+
+### Cross origin indexes
+
+Due to index merging happening in the browser, your additional search indexes must be configured with [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) headers if they span separate domains. For example, to configure these headers in a CloudCannon `routing.json` file:
+
+```json
+{
+  "headers": [
+    {
+      "match": "/pagefind/.*",
+      "headers": [
+        {
+          "name": "Access-Control-Allow-Origin",
+          "value": "*"
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Merging multiple languages
+
+Merged indexes will be searched using the WebAssembly module from your main instance. This means that merging an index from another language will use the language support from your main Pagefind instance. 

docs/content/docs/node-api.md

@@ -0,0 +1,226 @@
+---
+title: "Indexing content using the NodeJS API"
+nav_title: "Using the NodeJS API"
+nav_section: References
+weight: 55
+---
+
+Pagefind provides an interface to the indexing binary as a NodeJS package you can import.
+
+There are situations where using this Node library is beneficial:
+- Authors integrating Pagefind into an existing project, e.g. integrating Pagefind into the dev mode of a static site generator, can pass in-memory HTML files to Pagefind. Pagefind can also return the search index in-memory, to be hosted via the dev mode alongside the files.
+- Users looking to index their site and augment that index with extra non-HTML pages can run a standard Pagefind crawl with [`addDirectory`](#indexadddirectory) and augment it with [`addCustomRecord`](#indexaddcustomrecord).
+- Users looking to use Pagefind's engine for searching miscellaneous content such as PDFs or subtitles, where [`addCustomRecord`](#indexaddcustomrecord) can be used to build the entire index from scratch.
+
+## Example usage
+
+```js
+import * as pagefind from "pagefind";
+
+// Create a Pagefind search index to work with
+const { index } = await pagefind.createIndex();
+
+// Index all HTML files in a directory
+await index.addDirectory({
+    path: "public"
+});
+
+// Add extra content
+await index.addCustomRecord({
+    url: "/resume.pdf",
+    content: "Aenean lacinia bibendum nulla sed consectetur",
+    language: "en",
+});
+
+// Get the index files in-memory
+const { files } = await index.getFiles();
+
+// Or, write the index to disk
+await index.writeFiles({
+    outputPath: "public/pagefind"
+});
+```
+
+All interactions with Pagefind are asynchronous, as they communicate with the native Pagefind binary in the background.
+
+## pagefind.createIndex
+
+Creates a Pagefind index that files can be added to.
+The index object returned is unique, and multiple calls to `pagefind.createIndex()` can be made without conflicts.
+
+```js
+import * as pagefind from "pagefind";
+
+const { index } = await pagefind.createIndex();
+
+// ... do things with `index`
+```
+
+`createIndex` optionally takes a configuration object that can apply parts of the [Pagefind CLI config](/docs/config-options/). The options available at this level are:
+
+```js
+const { index } = await pagefind.createIndex({
+    rootSelector: "html",
+    excludeSelectors: [".my-code-blocks"],
+    forceLanguage: "en",
+    includeCharacters: "._",
+    lowercaseFilters: false,
+    keepIndexUrl: false,
+    writePlayground: false,
+    verbose: false,
+    logfile: "debug.log"
+});
+```
+
+See the relevant documentation for these configuration options in the [Configuring the Pagefind CLI](/docs/config-options/) documentation.
+
+## index.addDirectory
+
+Indexes a directory from disk using the standard Pagefind indexing behaviour.
+This is equivalent to running the Pagefind binary with `--site <dir>`.
+
+```js
+// Index all HTML files in the public directory
+const { errors, page_count } = await index.addDirectory({
+    path: "public"
+});
+
+// Index files in the directory that match the given glob
+const { errors, page_count } = await index.addDirectory({
+    path: "public",
+    glob: "**/*.{html}"
+});
+```
+
+If the `path` provided is relative, it will be relative to the current working directory of your Node process.
+
+Optionally, a custom `glob` can be supplied which controls which files Pagefind will consume within the directory. The default is shown, and the `glob` option can be omitted entirely.
+See [Wax patterns documentation](https://github.com/olson-sean-k/wax#patterns) for more details.
+
+A response with an `errors` array containing error messages indicates that Pagefind failed to process this directory.
+If successful, `page_count` will be the number of pages that were added to the index.
+
+## index.addHTMLFile
+
+Adds a virtual HTML file to the Pagefind index. Useful for files that don't exist on disk, for example a static site generator that is serving files from memory.
+
+```js
+// Index a file as if Pagefind was indexing from disk
+const { errors, file } = await index.addHTMLFile({
+    sourcePath: "contact/index.html",
+    content: "<html lang='en'><body> <h1>A Full HTML Document</h1> <p> . . . </p> </body></html>"
+});
+
+// Index HTML content, giving it a specific URL
+const { errors, file } = await index.addHTMLFile({
+    url: "/contact/",
+    content: "<html lang='en'><body> <h1>A Full HTML Document</h1> <p> . . . </p> </body></html>"
+});
+```
+
+The `sourcePath` should represent the path of this HTML file if it were to exist on disk. Pagefind will use this path to generate the URL. It should be relative, or absolute to a path within the current working directory.
+
+Instead of `sourcePath`, a `url` may be supplied to explicitly set the URL of this search result.
+
+The `content` should be the full HTML source, including the outer `<html> </html>` tags. This will be run through Pagefind's standard HTML indexing process, and should contain any required Pagefind attributes to control behaviour.
+
+A response with an `errors` array containing error messages indicates that Pagefind failed to index this content.
+If successful, the `file` object is returned containing metadata about the completed indexing.
+
+## index.addCustomRecord
+
+Adds a direct record to the Pagefind index. Useful for adding non-HTML content to the search results.
+
+```js
+const { errors, file } = await index.addCustomRecord({
+    url: "/contact/",
+    content: "My raw content to be indexed for search. Will be lightly processed by Pagefind.",
+    language: "en",
+    meta: {
+        title: "Contact",
+        category: "Landing Page"
+    },
+    filters: {
+        tags: ["landing", "company"]
+    },
+    sort: {
+        weight: "20"
+    }
+});
+```
+
+The `url`, `content`, and `language` fields are all required. `language` should be an [ISO 639-1 code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes).
+
+`meta` is optional, and is strictly a flat object of keys to string values.
+See the [Metadata documentation](https://pagefind.app/docs/metadata/) for semantics.
+
+`filters` is optional, and is strictly a flat object of keys to arrays of string values.
+See the [Filters documentation](https://pagefind.app/docs/filtering/) for semantics.
+
+`sort` is optional, and is strictly a flat object of keys to string values.
+See the [Sort documentation](https://pagefind.app/docs/sorts/) for semantics.
+*When Pagefind is processing an index, number-like strings will be sorted numerically rather than alphabetically. As such, the value passed in should be `"20"` and not `20`*
+
+A response with an `errors` array containing error messages indicates that Pagefind failed to index this content.
+If successful, the `file` object is returned containing metadata about the completed indexing.
+
+## index.getFiles
+
+Get raw data of all files in the Pagefind index. Useful for integrating a Pagefind index into the development mode of a static site generator and hosting these files yourself.
+
+```js
+const { errors, files } = await index.getFiles();
+
+for (const file of files) {
+    const output_url = `/pagefind/${file.path}`;
+    // do something with the file.content Uint8Array
+}
+```
+
+A response with an `errors` array containing error messages indicates that Pagefind failed to action this request.
+
+If successful, `files` will be an array containing file objects. Each object contains a `content` key with the raw data as a Uint8Array, and a `path` key which is the relative URL this file should be served at within a bundle directory.
+
+## index.writeFiles
+
+Writes the index files to disk, as they would be written when running the standard Pagefind binary directly.
+
+```js
+const { errors } = await index.writeFiles({
+    outputPath: "./public/pagefind"
+});
+```
+
+The `outputPath` option should contain the path to the desired Pagefind bundle directory. If relative, is relative to the current working directory of your Node process.
+
+A response with an `errors` array containing error messages indicates that Pagefind failed to action this request.
+
+## index.deleteIndex
+
+Deletes the data for the given index from the Pagefind binary service. Doesn't affect any written files or data returned by `getFiles()`.
+
+```js
+await index.deleteIndex();
+```
+
+Calling `index.getFiles()` or `index.writeFiles()` doesn't consume the index, and further modifications can be made. In situations where many indexes are being created, the `deleteIndex` call helps clear out memory from the Pagefind binary service.
+
+Reusing an `index` object after calling `index.deleteIndex()` will cause errors to be returned.
+
+Not calling this method is fine — these indexes will be cleaned up when your Node process exits.
+
+## pagefind.close
+
+Closes the Pagefind service and errors out any pending tasks, stopping the linked binary altogether.
+Called on the top-level `pagefind` object.
+
+```js
+import * as pagefind from "pagefind";
+
+const { index } = await pagefind.createIndex();
+
+// ... do things with `index`
+
+// clean up once complete
+await pagefind.close();
+```

docs/content/docs/py-api.md

@@ -0,0 +1,334 @@
+---
+title: "Indexing content using the Python API"
+nav_title: "Using the Python API"
+nav_section: References
+weight: 54 # slightly less weight than the node API
+---
+
+Pagefind provides an interface to the indexing binary as a Python package you can install and import.
+
+There are situations where using this Python package is beneficial:
+- Integrating Pagefind into an existing Python project, e.g. writing a plugin for a static site generator that can pass in-memory HTML files to Pagefind.
+  Pagefind can also return the search index in-memory, to be hosted via the dev mode alongside the files.
+- Users looking to index their site and augment that index with extra non-HTML pages can run a standard Pagefind crawl with [`add_directory`](#indexadd_directory) and augment it with [`add_custom_record`](#indexadd_custom_record).
+- Users looking to use Pagefind's engine for searching miscellaneous content such as PDFs or subtitles, where [`add_custom_record`](#indexadd_custom_record) can be used to build the entire index from scratch.
+
+## Installation
+
+To install just the Python wrapper, and use a `pagefind` executable from your system:
+```bash
+python3 -m pip install 'pagefind'
+```
+
+To install the Python wrapper as well as the standard binary for your platform:
+```bash
+python3 -m pip install 'pagefind[bin]'
+```
+
+To install the Python wrapper as well as the extended binary for your platform:
+```bash
+python3 -m pip install 'pagefind[extended]'
+```
+
+## Example Usage
+
+<!-- this example is copied verbatim from wrappers/python/src/tests/integration.py -->
+
+```py
+import asyncio
+import json
+import logging
+import os
+from pagefind.index import PagefindIndex, IndexConfig
+
+logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"))
+log = logging.getLogger(__name__)
+html_content = (
+    "<html>"
+    "  <body>"
+    "    <main>"
+    "      <h1>Example HTML</h1>"
+    "      <p>This is an example HTML page.</p>"
+    "    </main>"
+    "  </body>"
+    "</html>"
+)
+
+
+def prefix(pre: str, s: str) -> str:
+    return pre + s.replace("\n", f"\n{pre}")
+
+
+async def main():
+    config = IndexConfig(
+        root_selector="main", logfile="index.log", output_path="./output", verbose=True
+    )
+    async with PagefindIndex(config=config) as index:
+        log.debug("opened index")
+        new_file, new_record, new_dir = await asyncio.gather(
+            index.add_html_file(
+                content=html_content,
+                url="https://example.com",
+                source_path="other/example.html",
+            ),
+            index.add_custom_record(
+                url="/elephants/",
+                content="Some testing content regarding elephants",
+                language="en",
+                meta={"title": "Elephants"},
+            ),
+            index.add_directory("./public"),
+        )
+        print(prefix("new_file    ", json.dumps(new_file, indent=2)))
+        print(prefix("new_record  ", json.dumps(new_record, indent=2)))
+        print(prefix("new_dir     ", json.dumps(new_dir, indent=2)))
+
+        files = await index.get_files()
+        for file in files:
+            print(prefix("files", f"{len(file['content']):10}B {file['path']}"))
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+All interactions with Pagefind are asynchronous, as they communicate with the native Pagefind binary in the background.
+
+## PagefindIndex
+
+`pagefind.index.PagefindIndex` manages a pagefind index.
+
+`PagefindIndex` operates as an async contextmanager.
+Entering the context starts a backing Pagefind service and creates an in-memory index in the backing service.
+Exiting the context writes the in-memory index to disk and then shuts down the backing Pagefind service.
+
+```py
+from pagefind.index import PagefindIndex
+
+async def main():
+    async with PagefindIndex() as index: # open the index
+        ... # update the index
+    # the index is closed here and files are written to disk.
+```
+
+Each method of `PagefindIndex` that talks to the backing Pagefind service can raise errors.
+If an error is thrown inside `PagefindIndex`'s context, the context closes without writing the index files to disk.
+
+```py
+async def main():
+    async with PagefindIndex() as index: # open the index
+        await index.add_directory("./public")
+        raise Exception("not today")
+    # the index closes without writing anything to disk
+```
+
+`PagefindIndex` optionally takes a configuration dictionary that can apply parts of the [Pagefind CLI config](/docs/config-options/). The options available at this level are:
+
+```py
+from pagefind.index import PagefindIndex, IndexConfig
+config = IndexConfig(
+    root_selector="main",
+    exclude_selectors=["nav"],
+    force_language="en",
+    include_characters="._",
+    lowercase_filters=False,
+    verbose=True,
+    logfile="index.log",
+    keep_index_url=True,
+    write_playground=True,
+    output_path="./output",
+)
+
+async def main():
+    async with PagefindIndex(config=config) as index:
+        ...
+```
+
+See the relevant documentation for these configuration options in the [Configuring the Pagefind CLI](/docs/config-options/) documentation.
+
+## index.add_directory
+
+Indexes a directory from disk using the standard Pagefind indexing behaviour.
+This is equivalent to running the Pagefind binary with `--site <dir>`.
+
+```py
+# Index all the HTML files in the public directory
+indexed_dir = await index.add_directory("./public")
+page_count: int = new_dir["page_count"]
+```
+If the `path` provided is relative, it will be relative to the current working directory of your Python process.
+
+```py
+# Index files in a directory matching a given glob pattern.
+indexed_dir = await index.add_directory("./public", glob="**.{html}")
+```
+
+Optionally, a custom `glob` can be supplied which controls which files Pagefind will consume within the directory. The default is shown, and the `glob` option can be omitted entirely.
+See [Wax patterns documentation](https://github.com/olson-sean-k/wax#patterns) for more details.
+
+## index.add_html_file
+
+Adds a virtual HTML file to the Pagefind index. Useful for files that don't exist on disk, for example a static site generator that is serving files from memory.
+
+```py
+html_content = (
+    "<html lang='en'><body>"
+    "  <h1>A Full HTML Document</h1>"
+    "  <p> ... </p>"
+    "</body></html>"
+)
+
+# Index a file as if Pagefind was indexing from disk
+new_file = await index.add_html_file(
+    content=html_content,
+    source_path="other/example.html",
+)
+
+# Index HTML content, giving it a specific URL
+new_file = await index.add_html_file(
+    content=html_content,
+    url="https://example.com",
+)
+```
+
+The `source_path` should represent the path of this HTML file if it were to exist on disk. Pagefind will use this path to generate the URL. It should be relative, or absolute to a path within the current working directory.
+
+Instead of `source_path`, a `url` may be supplied to explicitly set the URL of this search result.
+
+The `content` should be the full HTML source, including the outer `<html> </html>` tags. This will be run through Pagefind's standard HTML indexing process, and should contain any required Pagefind attributes to control behaviour.
+
+If successful, the `file` object is returned containing metadata about the completed indexing.
+
+## index.add_custom_record
+Adds a direct record to the Pagefind index.
+Useful for adding non-HTML content to the search results.
+
+```py
+custom_record = await index.add_custom_record(
+    url="/contact/",
+    content=(
+      "My raw content to be indexed for search. "
+      "Will be lightly processed by Pagefind."
+    ),
+    language="en",
+    meta={
+        "title": "Contact",
+        "category": "Landing Page"
+    },
+    filters={"tags": ["landing", "company"]},
+    sort={"weight": "20"},
+)
+
+page_word_count: int = custom_record["page_word_count"]
+page_url: str = custom_record["page_url"]
+page_meta: dict[str, str] = custom_record["page_meta"]
+```
+
+The `url`, `content`, and `language` fields are all required. `language` should be an [ISO 639-1 code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes).
+
+`meta` is optional, and is strictly a flat object of keys to string values.
+See the [Metadata documentation](https://pagefind.app/docs/metadata/) for semantics.
+
+`filters` is optional, and is strictly a flat object of keys to arrays of string values.
+See the [Filters documentation](https://pagefind.app/docs/filtering/) for semantics.
+
+`sort` is optional, and is strictly a flat object of keys to string values.
+See the [Sort documentation](https://pagefind.app/docs/sorts/) for semantics.
+*When Pagefind is processing an index, number-like strings will be sorted numerically rather than alphabetically. As such, the value passed in should be `"20"` and not `20`*
+
+If successful, the `file` object is returned containing metadata about the completed indexing.
+
+## index.get_files
+
+Get raw data of all files in the Pagefind index.
+Useful for integrating a Pagefind index into the development mode of a static site generator and hosting these files yourself.
+
+**WATCH OUT**: these files can be large enough to clog the pipe reading from the `pagefind` binary's subprocess, causing a deadlock.
+
+```py
+for file in (await index.get_files()):
+    path: str = file["path"]
+    content: str = file["content"]
+    ...
+```
+
+## index.write_files
+
+Calling `index.write_files()` writes the index files to disk, as they would be written when running the standard Pagefind binary directly.
+
+Closing the `PagefindIndex`'s context automatically calls `index.write_files`, so calling this function is not necessary in normal operation.
+
+Calling this function won't prevent files being written when the context closes, which may cause duplicate files to be written.
+If calling this function manually, you probably want to also call `index.delete_index()`.
+
+```py
+config = IndexConfig(
+    output_path="./public/pagefind",
+)
+async with PagefindIndex(config=config) as index:
+    # ... add content to index
+
+    # write files to the configured output path for the index:
+    await index.write_files()
+
+    # write files to a different output path:
+    await index.write_files(output_path="./custom/pagefind")
+
+    # prevent also writing files when closing the `PagefindIndex`:
+    await index.delete_index()
+```
+
+The `output_path` option should contain the path to the desired Pagefind bundle directory. If relative, is relative to the current working directory of your Python process.
+
+## index.delete_index
+
+Deletes the data for the given index from its backing Pagefind service.
+Doesn't affect any written files or data returned by `get_files()`.
+
+```python
+await index.delete_index()
+```
+
+Calling `index.get_files()` or `index.write_files()` doesn't consume the index, and further modifications can be made. In situations where many indexes are being created, the `delete_index` call helps clear out memory from a shared Pagefind binary service.
+
+Reusing an `PagefindIndex` object after calling `index.delete_index()` will cause errors to be returned.
+
+Not calling this method is fine — these indexes will be cleaned up when your `PagefindIndex`'s context closes, its backing Pagefind service closes, or your Python process exits.
+
+## PagefindService
+
+`PagefindService` manages a pagefind service running in a subprocess.
+
+`PagefindService` operates as an async context manager: when the context is entered, the backing service starts, and when the context exits, the backing service shuts down.
+
+```py
+from pagefind.service import PagefindService
+
+async def main():
+    # or you can write
+    service = await PagefindService().launch()
+    ...
+    await service.close()
+
+    async with PagefindService() as service: # the service launches
+        ...
+    # the service closes
+```
+
+You should invoke `PagefindService` directly when you want to use the same backing service for many indexes:
+
+```py
+async with PagefindService() as service:
+    default_index = await service.create_index()
+    other_index = await service.create_index(
+        config=IndexConfig(output_path="./search/nonstandard"),
+    )
+    await asyncio.gather(
+        default_index.add_directory("./a"),
+        other_index.add_directory("./b"),
+    )
+    await asyncio.gather(
+        default_index.write_files(),
+        other_index.write_files(),
+    )
+```

pagefind/integration_tests/filtering/filter-values-can-be-lowercased.toolproof.yml

@@ -0,0 +1,51 @@
+name: Filtering > Filter values can be lowercased
+steps:
+  - step: I have the environment variable "PAGEFIND_SITE" set to "public"
+  - step: I have a "public/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html lang="en"><head></head><body><p
+      data-results>Nothing</p></body></html>
+  - step: I have a "public/cheeka/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html lang="en"><head></head><body><span
+      data-pagefind-filter="color">Black</span>
+
+      <span data-pagefind-filter="color">White</span>
+
+      <span data-pagefind-filter="color">white</span>
+
+      <h1>Cat</h1></body></html>
+  - step: I have a "public/theodore/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html lang="en"><head></head><body><span
+      data-pagefind-filter="color">WHITE</span>
+
+      <h1>Cat</h1></body></html>
+  - macro: I run Pagefind with "--lowercase-filters"
+  - step: stdout should contain "Running Pagefind"
+  - step: I serve the directory "public"
+  - step: In my browser, I load "/"
+  - step: In my browser, I evaluate {js}
+    js: |-
+      let pagefind = await import("/pagefind/pagefind.js");
+
+      let filters = await pagefind.filters();
+      let strings = Object.entries(filters).map(([filter, values]) => {
+          values = Object.entries(values).map(([value, count]) => {
+              return `${value}(${count})`;
+          })
+          return `${filter}:[${values.join(", ")}]`;
+      });
+
+      let search = await pagefind.search("Cat", { filters: { color: "white" } });
+      let data = await Promise.all(search.results.map(result => result.data()));
+
+      document.querySelector('[data-results]').innerText = strings.join(' ')
+          + ' ' + data.map(d => d.url).sort().join(', ');
+  - step: In my browser, the console should be empty
+  - step: In my browser, I evaluate {js}
+    js: >-
+      let val = await toolproof.querySelector("[data-results]");
+
+      toolproof.assert_eq(val.innerHTML, `color:[black(1), white(2)]
+      /cheeka/, /theodore/`);

pagefind/integration_tests/multisite/multisite_base/pagefind-skips-an-optional-index-that-fails-to-load.toolproof.yml

@@ -0,0 +1,33 @@
+name: Multisite Search > Pagefind skips an optional index that fails to load
+steps:
+  - ref: ./background.toolproof.yml
+  - macro: I run Pagefind with "--site root/website_a"
+  - step: stdout should contain "Running Pagefind"
+  - step: The file "root/website_a/pagefind/pagefind.js" should not be empty
+  - macro: I run Pagefind with "--site root/website_b"
+  - step: stdout should contain "Running Pagefind"
+  - step: The file "root/website_b/pagefind/pagefind.js" should not be empty
+  - step: I serve the directory "root"
+  - step: In my browser, I load "/"
+  - step: In my browser, I evaluate {js}
+    js: >-
+      let pagefind = await import("/website_a/pagefind/pagefind.js");
+
+      await pagefind.mergeIndex("/website_c/pagefind/", { optional: true });
+
+      await pagefind.mergeIndex("/website_b/pagefind/");
+
+
+      let search = await pagefind.search("web");
+
+
+      let pages = await Promise.all(search.results.map(r => r.data()));
+
+      document.querySelector('[data-result]').innerText = pages.map(p =>
+      p.url).join(", ");
+  - step: In my browser, I evaluate {js}
+    js: >-
+      let val = await toolproof.querySelector("[data-result]");
+
+      toolproof.assert_eq(val.innerHTML, `/website_a/hello/,
+      /website_b/lorem/`);

pagefind/integration_tests/multisite/multisite_base/pagefind-ui-skips-an-optional-index-that-fails-to-load.toolproof.yml

@@ -0,0 +1,42 @@
+name: Multisite Search > Pagefind UI skips an optional index that fails to load
+steps:
+  - ref: ./background.toolproof.yml
+  - step: I have a "root/index.html" file with the content {html}
+    html: |-
+      <!DOCTYPE html><html lang="en"><head></head><body><div id="search"></div>
+      <script src="/website_a/pagefind/pagefind-ui.js"></script></body></html>
+  - macro: I run Pagefind with "--site root/website_a"
+  - step: stdout should contain "Running Pagefind"
+  - step: The file "root/website_a/pagefind/pagefind.js" should not be empty
+  - macro: I run Pagefind with "--site root/website_b"
+  - step: stdout should contain "Running Pagefind"
+  - step: The file "root/website_b/pagefind/pagefind.js" should not be empty
+  - step: I serve the directory "root"
+  - step: In my browser, I load "/"
+  - step: In my browser, I evaluate {js}
+    js: |-
+      let pui = new PagefindUI({
+          element: "#search",
+          mergeIndex: [{
+              bundlePath: "/website_c/pagefind/",
+              optional: true
+          }, {
+              bundlePath: "/website_b/pagefind/"
+          }]
+      });
+      pui.triggerSearch("web");
+  - step: In my browser, I evaluate {js}
+    js: >-
+      let val = await
+      toolproof.querySelector(".pagefind-ui__result:nth-of-type(1)
+      .pagefind-ui__result-link");
+
+      toolproof.assert_eq(val.innerHTML, `web web world web
+      PAGEFIND_ROOT_SELECTOR`);
+  - step: In my browser, I evaluate {js}
+    js: >-
+      let val = await
+      toolproof.querySelector(".pagefind-ui__result:nth-of-type(2)
+      .pagefind-ui__result-link");
+
+      toolproof.assert_eq(val.innerHTML, `web ipsum`);

pagefind/integration_tests/node_api/node_base/pagefind-lowercase-filters-config.toolproof.yml

@@ -0,0 +1,39 @@
+name: Node API Base Tests > Pagefind lowercaseFilters config
+platforms:
+  - linux
+  - mac
+
+steps:
+  - ref: ./background.toolproof.yml
+  - step: I have a "public/index.js" file with the content {js}
+    js: |2-
+       import * as pagefind from "pagefind";
+
+       const run = async () => {
+           const { index } = await pagefind.createIndex({
+               lowercaseFilters: true,
+           });
+           await index.addHTMLFile({sourcePath: "page_a/index.html", content: "<html lang='en'><body><h1 data-pagefind-filter='color:White'>Cat</h1></body></html>"});
+           await index.addCustomRecord({
+               url: "/page_b/",
+               content: "Cat",
+               language: "en",
+               filters: { color: ["WHITE", "Black"] },
+           });
+           await index.writeFiles();
+           console.log(`Successfully wrote files`);
+       }
+
+       run();
+  - macro: I run Pagefind Node as "index.js" in "public"
+  - step: stdout should contain "Successfully wrote files"
+  - step: The file "public/pagefind/pagefind.js" should not be empty
+  - step: I serve the directory "public"
+  - step: In my browser, I load "/"
+  - step: In my browser, I evaluate {js}
+    js: |-
+      let pagefind = await import("/pagefind/pagefind.js");
+      let filters = await pagefind.filters();
+
+      toolproof.assert_eq(JSON.stringify(filters), `{"color":{"black":1,"white":2}}`);
+  - step: In my browser, the console should be empty

pagefind/integration_tests/ui/ui_base/pagefind-ui-resolves-image-urls-against-base-url.toolproof.yml

@@ -0,0 +1,67 @@
+name: Base UI Tests > Pagefind UI resolves image URLs against an absolute base URL
+steps:
+  - ref: ./background.toolproof.yml
+  - step: I have a "public/index.html" file with the content {html}
+    html: |-
+      <!DOCTYPE html><html lang="en"><head></head><body><div id="search"></div>
+      <script src="/pagefind/pagefind-ui.js"></script>
+
+      <script>
+          window.pui = new PagefindUI({ element: "#search", baseUrl: "https://example.com/" });
+      </script></body></html>
+  - step: I have a "public/subs/index.html" file with the content {html}
+    html: |-
+      <!DOCTYPE html><html lang="en"><head></head><body><div id="search"></div>
+      <script src="/pagefind/pagefind-ui.js"></script>
+
+      <script>
+          window.pui = new PagefindUI({ element: "#search", baseUrl: "https://example.com/", showSubResults: true });
+      </script></body></html>
+  - step: I have a "public/blog/post/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html
+      lang="en"><head></head><body><h1>Relative Image</h1><img
+      src="images/header.png"><p>A page with a relative image
+      URL.</p></body></html>
+  - step: I have a "public/about/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html
+      lang="en"><head></head><body><h1>Absolute Image</h1><img
+      src="/images/hero.png"><p>A page with an absolute image
+      URL.</p></body></html>
+  - macro: I run Pagefind
+  - step: stdout should contain "Running Pagefind"
+  - step: The file "public/pagefind/pagefind.js" should not be empty
+  - step: I serve the directory "public"
+  - step: In my browser, I load "/"
+  - step: In my browser, I evaluate {js}
+    js: window.pui.triggerSearch("image");
+  - step: In my browser, the console should be empty
+  - snapshot: In my browser, the result of {js}
+    js: |-
+      await toolproof.waitFor(() => document.querySelectorAll(".pagefind-ui__result-image").length === 2);
+      let items = Array.from(document.querySelectorAll(".pagefind-ui__result"));
+      items.sort((a, b) => a.textContent.localeCompare(b.textContent));
+      return items.map(li => {
+        let title = li.querySelector(".pagefind-ui__result-link").textContent;
+        return title + ": " + li.querySelector(".pagefind-ui__result-image").getAttribute("src");
+      }).join("\n");
+    snapshot_content: |-
+      ╎Absolute Image: https://example.com/images/hero.png
+      ╎Relative Image: https://example.com/blog/post/images/header.png
+  - step: In my browser, I load "/subs/"
+  - step: In my browser, I evaluate {js}
+    js: window.pui.triggerSearch("image");
+  - step: In my browser, the console should be empty
+  - snapshot: In my browser, the result of {js}
+    js: |-
+      await toolproof.waitFor(() => document.querySelectorAll(".pagefind-ui__result-image").length === 2);
+      let items = Array.from(document.querySelectorAll(".pagefind-ui__result"));
+      items.sort((a, b) => a.textContent.localeCompare(b.textContent));
+      return items.map(li => {
+        let title = li.querySelector(".pagefind-ui__result-link").textContent;
+        return title + ": " + li.querySelector(".pagefind-ui__result-image").getAttribute("src");
+      }).join("\n");
+    snapshot_content: |-
+      ╎Absolute Image: https://example.com/images/hero.png
+      ╎Relative Image: https://example.com/blog/post/images/header.png

pagefind/integration_tests/web_components/relative-image-urls-base-url.toolproof.yml

@@ -0,0 +1,49 @@
+name: Web Components Tests > Image URLs are resolved against an absolute base URL
+steps:
+  - step: I have the environment variable "PAGEFIND_SITE" set to "public"
+  - step: I have a "public/index.html" file with the content {html}
+    html: |-
+      <!DOCTYPE html><html lang="en"><head>
+      <link href="/pagefind/pagefind-component-ui.css" rel="stylesheet">
+      </head><body>
+      <pagefind-config preload base-url="https://example.com/"></pagefind-config>
+      <pagefind-input></pagefind-input>
+      <pagefind-results show-images></pagefind-results>
+      <script src="/pagefind/pagefind-component-ui.js" type="module"></script>
+      </body></html>
+  - step: I have a "public/blog/post/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html
+      lang="en"><head></head><body><h1>Relative Image</h1><img
+      src="images/header.png"><p>A page with a relative image
+      URL.</p></body></html>
+  - step: I have a "public/about/index.html" file with the content {html}
+    html: >-
+      <!DOCTYPE html><html
+      lang="en"><head></head><body><h1>Absolute Image</h1><img
+      src="/images/hero.png"><p>A page with an absolute image
+      URL.</p></body></html>
+  - macro: I run Pagefind
+  - step: stdout should contain "Running Pagefind"
+  - step: I serve the directory "public"
+  - step: In my browser, I load "/"
+  - step: In my browser, I click the selector "pagefind-input input"
+  - step: In my browser, I type "image"
+  - step: In my browser, the console should be empty
+  - step: In my browser, I evaluate {js}
+    js: |-
+      let results = await toolproof.querySelector("pagefind-results ul");
+      await toolproof.waitFor(() => results.querySelectorAll("li a").length === 2);
+  - snapshot: In my browser, the result of {js}
+    js: |-
+      let results = await toolproof.querySelector("pagefind-results ul");
+      let items = Array.from(results.querySelectorAll("li"));
+      items.sort((a, b) => a.textContent.localeCompare(b.textContent));
+      return items.map(li => {
+        let title = li.querySelector("a").textContent;
+        let img = li.querySelector("img");
+        return title + ": " + (img ? img.getAttribute("src") : "no image");
+      }).join("\n");
+    snapshot_content: |-
+      ╎Absolute Image: https://example.com/images/hero.png
+      ╎Relative Image: https://example.com/blog/post/images/header.png

pagefind/src/fossick/mod.rs

@@ -7,7 +7,6 @@ use hashbrown::HashMap;
 use lazy_static::lazy_static;
 use pagefind_stem::{Algorithm, Stemmer};
 use path_slash::PathExt as _;
-use percent_encoding::{utf8_percent_encode, AsciiSet, CONTROLS};
 use regex::Regex;
 use std::collections::BTreeMap;
 use std::io::BufRead;
@@ -282,6 +281,9 @@ impl Fossicker {
 
         let (content, word_data, anchors, word_count) = self.parse_digest(options);
         self.tidy_meta_and_filters();
+        if options.lowercase_filters {
+            self.lowercase_filters();
+        }
 
         let data = self.data.unwrap();
 
@@ -557,6 +559,22 @@ impl Fossicker {
         }
     }
 
+    /// Lowercases filter values, merging values on a page that only differed in case.
+    fn lowercase_filters(&mut self) {
+        if let Some(data) = self.data.as_mut() {
+            for values in data.filters.values_mut() {
+                let mut lowercased: Vec<String> = Vec::with_capacity(values.len());
+                for value in values.drain(..) {
+                    let value = value.to_lowercase();
+                    if !lowercased.contains(&value) {
+                        lowercased.push(value);
+                    }
+                }
+                *values = lowercased;
+            }
+        }
+    }
+
     /// Parse words from metadata fields and return them with field IDs and positions.
     fn parse_meta_words(
         meta: &BTreeMap<String, String>,
@@ -590,20 +608,6 @@ impl Fossicker {
 
 }
 
-/// The URL spec's path percent-encode set, minus `/` so that the separators
-/// between the path segments survive. `%` is deliberately left alone, so a file
-/// that is already named in encoded form keeps the URL it has today.
-const PATH_ENCODE_SET: &AsciiSet = &CONTROLS
-    .add(b' ')
-    .add(b'"')
-    .add(b'#')
-    .add(b'<')
-    .add(b'>')
-    .add(b'?')
-    .add(b'`')
-    .add(b'{')
-    .add(b'}');
-
 fn strip_index_html(url: &str) -> &str {
     if url.ends_with("/index.html") {
         &url[..url.len() - 10]
@@ -636,7 +640,13 @@ fn build_url(page_url: &Path, relative_to: Option<&Path>, options: &SearchOption
         url.to_slash_lossy().to_owned().to_string()
     };
 
-    format!("/{}", utf8_percent_encode(&final_url, PATH_ENCODE_SET))
+    format!(
+        "/{}",
+        final_url
+            .replace('%', "%25")
+            .replace('#', "%23")
+            .replace('?', "%3F")
+    )
 }
 
 fn normalize_content(content: &str) -> String {
@@ -1235,6 +1245,41 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn lowercase_filters() {
+        let html = [
+            "<html><body>",
+            "<span data-pagefind-filter=\"Garnish\">Mint</span>",
+            "<span data-pagefind-filter=\"Garnish\">mint</span>",
+            "<span data-pagefind-filter=\"Garnish\">Basil</span>",
+            "<p>Hello World!</p>",
+            "</body></html>",
+        ]
+        .concat();
+
+        let data = test_fossick(html.clone())
+            .fossick_sync(&test_opts())
+            .unwrap();
+        assert_eq!(
+            data.fragment.data.filters,
+            BTreeMap::from_iter([(
+                "Garnish".to_string(),
+                vec!["Mint".to_string(), "mint".to_string(), "Basil".to_string()]
+            )])
+        );
+
+        let mut opts = test_opts();
+        opts.lowercase_filters = true;
+        let data = test_fossick(html).fossick_sync(&opts).unwrap();
+        assert_eq!(
+            data.fragment.data.filters,
+            BTreeMap::from_iter([(
+                "Garnish".to_string(),
+                vec!["mint".to_string(), "basil".to_string()]
+            )])
+        );
+    }
+
     #[cfg(not(target_os = "windows"))]
     #[test]
     fn building_url() {
@@ -1260,29 +1305,9 @@ mod tests {
         let p: PathBuf = cwd.join::<PathBuf>("hello/world/index.html".into());
         let root: PathBuf = cwd.join::<PathBuf>("hello".into());
         assert_eq!(&build_url(&p, Some(&root), &opts), "/world/");
-    }
 
-    #[cfg(not(target_os = "windows"))]
+        let p: PathBuf = cwd.join::<PathBuf>("hello/world/100%/C#/why?.html".into());
-    #[test]
+        assert_eq!(&build_url(&p, None, &opts), "/100%25/C%23/why%3F.html");
-    fn building_urls_with_characters_that_need_encoding() {
-        std::env::set_var("PAGEFIND_SITE", "hello/world");
-        let config =
-            PagefindInboundConfig::with_layers(&[Layer::Env(Some("PAGEFIND_".into()))]).unwrap();
-        let opts = SearchOptions::load(config).unwrap();
-
-        let cwd = std::env::current_dir().unwrap();
-
-        let p: PathBuf = cwd.join::<PathBuf>("hello/world/Müsli.html".into());
-        assert_eq!(&build_url(&p, None, &opts), "/M%C3%BCsli.html");
-
-        let p: PathBuf = cwd.join::<PathBuf>("hello/world/Müsli/index.html".into());
-        assert_eq!(&build_url(&p, None, &opts), "/M%C3%BCsli/");
-
-        let p: PathBuf = cwd.join::<PathBuf>("hello/world/a b/c#d?e.html".into());
-        assert_eq!(&build_url(&p, None, &opts), "/a%20b/c%23d%3Fe.html");
-
-        let p: PathBuf = cwd.join::<PathBuf>("hello/world/M%C3%BCsli.html".into());
-        assert_eq!(&build_url(&p, None, &opts), "/M%C3%BCsli.html");
     }
 
     #[cfg(target_os = "windows")]

pagefind/src/fossick/parser.rs

@@ -21,7 +21,8 @@ lazy_static! {
 }
 
 const SENTENCE_SELECTORS: &[&'static str] = &[
-    "h1", "h2", "h3", "h4", "h5", "h6", "p", "td", "div", "ul", "li", "article", "section",
+    "h1", "h2", "h3", "h4", "h5", "h6", "p", "td", "div", "ul", "li", "dl", "dt", "dd", "article",
+    "section",
 ];
 const INLINE_SELECTORS: &[&'static str] = &[
     "a", "abbr", "acronym", "b", "bdo", "big", "br", "button", "cite", "code", "dfn", "em", "i",
@@ -361,11 +362,13 @@ impl<'a> DomParser<'a> {
                                 // Similarly, we want to separate block elements
                                 // with punctuation, so that the excerpts read nicely.
                                 // (As long as it doesn't already end with, say, a . or ?)
+                                // Description list terms lead into their details,
+                                // so they get a colon instead.
                                 if node.current_value.chars()
                                     .last()
                                     .filter(|c| SENTENCE_CHARS.is_match(&c.to_string()))
                                     .is_some() {
-                                        node.current_value.push('.');
+                                        node.current_value.push(if tag_name == "dt" { ':' } else { '.' });
                                 }
                                 node.current_value.push(' ');
                             }
@@ -837,6 +840,23 @@ mod tests {
         )
     }
 
+    #[test]
+    fn description_list_formatting() {
+        let data = test_parse(vec![
+            "<dl>",
+            "<dt>Morgawr</dt>",
+            "<dd>A sea serpent</dd>",
+            "<dt>Owlman:</dt>",
+            "<dd>A giant owl-like creature</dd>",
+            "</dl>",
+        ]);
+
+        assert_eq!(
+            data.digest,
+            "Morgawr: A sea serpent. Owlman: A giant owl-like creature."
+        )
+    }
+
     #[test]
     fn inline_tag_formatting() {
         let data = test_parse(vec![

pagefind/src/fossick/splitting.rs

@@ -0,0 +1,329 @@
+use convert_case::{Case, Casing};
+use emojis;
+use lazy_static::lazy_static;
+use pagefind_stem::Stemmer;
+use regex::Regex;
+use unicode_normalization::UnicodeNormalization;
+use unicode_segmentation::UnicodeSegmentation;
+
+lazy_static! {
+    static ref EMOJI: Regex = Regex::new("\\p{Emoji}").unwrap();
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct IndexableWord {
+    pub stemmed: String,
+    pub is_compound_part: bool,
+    /// The original word before diacritic normalization (if different)
+    pub original: Option<String>,
+}
+
+/// Returns all words that should be indexed for a given input word.
+/// This includes:
+/// - The primary normalized+stemmed word
+/// - Compound word parts (for hyphenated/camelCase/etc words)
+/// - Emoji characters
+pub fn get_indexable_words(
+    raw_word: &str,
+    stemmer: Option<&Stemmer>,
+    include_characters: &[char],
+) -> Vec<IndexableWord> {
+    let mut results = Vec::new();
+
+    let mut normalized_word = String::with_capacity(raw_word.len());
+    for mut c in raw_word.chars() {
+        let is_alpha = c.is_alphanumeric();
+        if is_alpha || include_characters.contains(&c) {
+            c.make_ascii_lowercase();
+            if c.is_uppercase() {
+                // Non-ascii uppercase can lower to multiple chars
+                normalized_word.extend(c.to_lowercase());
+            } else {
+                normalized_word.push(c);
+            }
+        }
+    }
+
+    if !normalized_word.is_empty() {
+        let diacritic_normalized = normalize_diacritics(&normalized_word);
+
+        let stemmed = if let Some(stemmer) = stemmer {
+            stemmer
+                .stem(diacritic_normalized.as_deref().unwrap_or(&normalized_word))
+                .into_owned()
+        } else {
+            diacritic_normalized
+                .clone()
+                .unwrap_or_else(|| normalized_word.to_string())
+        };
+
+        results.push(IndexableWord {
+            stemmed,
+            original: diacritic_normalized.map(|_| normalized_word.to_string()),
+            is_compound_part: false,
+        });
+    }
+
+    let possibly_compound = raw_word.chars().any(|c| !c.is_alphanumeric())
+        || raw_word.chars().skip(1).any(|c| c.is_uppercase());
+
+    if !possibly_compound {
+        return results;
+    }
+
+    let (word_parts, extras) = get_discrete_words(raw_word);
+    if !normalized_word.is_empty()
+        && (word_parts.contains(|c: char| c.is_whitespace())
+            || !normalized_word.starts_with(&word_parts))
+    {
+        let part_words: Vec<_> = word_parts.split_whitespace().collect();
+
+        // Only index two+ character words
+        for part_word in part_words.into_iter().filter(|w| w.len() > 1) {
+            let part_diacritic = normalize_diacritics(part_word);
+
+            let stemmed_part = if let Some(stemmer) = stemmer {
+                stemmer
+                    .stem(part_diacritic.as_deref().unwrap_or(part_word))
+                    .into_owned()
+            } else {
+                part_diacritic
+                    .clone()
+                    .unwrap_or_else(|| part_word.to_string())
+            };
+
+            results.push(IndexableWord {
+                stemmed: stemmed_part,
+                original: part_diacritic.map(|_| part_word.to_string()),
+                is_compound_part: true,
+            });
+        }
+    }
+
+    if let Some(extras) = extras {
+        for extra in extras {
+            results.push(IndexableWord {
+                stemmed: extra,
+                original: None,
+                is_compound_part: false,
+            });
+        }
+    }
+
+    results
+}
+
+/// Normalize diacritics, if required, before indexing
+/// (context: we index "café" as "cafe" for retrieval purposes,
+/// but we do still store each variant in the index,
+/// so matching diacritics are preferred when ranking search results)
+fn normalize_diacritics(word: &str) -> Option<String> {
+    if word.is_ascii() {
+        return None;
+    }
+
+    let normalized: String = word
+        .nfd()
+        .filter(|c| !unicode_normalization::char::is_combining_mark(*c))
+        .collect();
+
+    if normalized != word {
+        Some(normalized)
+    } else {
+        None
+    }
+}
+
+fn get_discrete_words<S: AsRef<str>>(s: S) -> (String, Option<Vec<String>>) {
+    let mut extras = None;
+
+    let words = s
+        .as_ref()
+        .replace(|c: char| c.is_ascii_punctuation() || c == '\u{AD}', " ")
+        .to_case(Case::Lower)
+        .split_whitespace()
+        .collect::<Vec<_>>()
+        .join(" ");
+
+    if EMOJI.is_match(s.as_ref()) {
+        extras = Some(
+            s.as_ref()
+                .graphemes(true)
+                .into_iter()
+                .filter_map(|x| {
+                    if emojis::get(x).is_some() {
+                        Some(x.to_string())
+                    } else {
+                        None
+                    }
+                })
+                .collect::<Vec<_>>(),
+        );
+    }
+
+    (words, extras)
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::fossick::get_stemmer;
+
+    use super::*;
+
+    #[test]
+    fn indexable_words_simple() {
+        let stemmer = get_stemmer("en");
+        let results = get_indexable_words("hello", stemmer.as_ref(), &[]);
+
+        assert_eq!(
+            results,
+            vec![IndexableWord {
+                stemmed: "hello".to_string(),
+                is_compound_part: false,
+                original: None
+            }]
+        );
+    }
+
+    #[test]
+    fn indexable_words_compound() {
+        let stemmer = get_stemmer("en");
+
+        let expected = vec![
+            IndexableWord {
+                stemmed: "threeantelop".to_string(),
+                is_compound_part: false,
+                original: None,
+            },
+            IndexableWord {
+                stemmed: "three".to_string(),
+                is_compound_part: true,
+                original: None,
+            },
+            IndexableWord {
+                stemmed: "antelop".to_string(),
+                is_compound_part: true,
+                original: None,
+            },
+        ];
+
+        let camel_results = get_indexable_words("ThreeAntelopes", stemmer.as_ref(), &[]);
+        let kebab_results = get_indexable_words("three-antelopes", stemmer.as_ref(), &[]);
+        let snake_results = get_indexable_words("three_antelopes", stemmer.as_ref(), &[]);
+        let soft_hyphen_results =
+            get_indexable_words("three\u{AD}antelopes", stemmer.as_ref(), &[]);
+
+        assert_eq!(camel_results, expected);
+        assert_eq!(kebab_results, expected);
+        assert_eq!(snake_results, expected);
+        assert_eq!(soft_hyphen_results, expected);
+    }
+
+    #[test]
+    fn indexable_words_compound_diacritics() {
+        let stemmer = get_stemmer("fr");
+        let results = get_indexable_words("café-crème", stemmer.as_ref(), &[]);
+
+        assert_eq!(
+            results,
+            vec![
+                IndexableWord {
+                    stemmed: "cafecrem".to_string(),
+                    is_compound_part: false,
+                    original: Some("cafécrème".to_string()),
+                },
+                IndexableWord {
+                    stemmed: "caf".to_string(),
+                    is_compound_part: true,
+                    original: Some("café".to_string()),
+                },
+                IndexableWord {
+                    stemmed: "crem".to_string(),
+                    is_compound_part: true,
+                    original: Some("crème".to_string()),
+                },
+            ]
+        );
+    }
+
+    #[test]
+    fn hyphenated_words() {
+        let input = "these-words-are-hyphenated";
+        assert_eq!(
+            get_discrete_words(input),
+            ("these words are hyphenated".into(), None)
+        );
+    }
+
+    #[test]
+    fn underscored_words() {
+        let input = "__array_structures";
+        assert_eq!(get_discrete_words(input), ("array structures".into(), None));
+    }
+
+    #[test]
+    fn soft_hyphenated_words() {
+        let input = "Hochspannungs\u{AD}netz\u{AD}teil";
+        assert_eq!(
+            get_discrete_words(input),
+            ("hochspannungs netz teil".into(), None)
+        );
+    }
+
+    #[test]
+    fn camel_words() {
+        let input = "WKWebVIEWComponent";
+        assert_eq!(
+            get_discrete_words(input),
+            ("wk web view component".into(), None)
+        );
+    }
+
+    #[test]
+    fn dotted_words() {
+        let input = "page.Find";
+        assert_eq!(get_discrete_words(input), ("page find".into(), None));
+    }
+
+    #[test]
+    fn misc_punctuation() {
+        let input = "cloud/cannon,page.find";
+        assert_eq!(
+            get_discrete_words(input),
+            ("cloud cannon page find".into(), None)
+        );
+    }
+
+    #[test]
+    fn french() {
+        let input = "l'alphabet";
+        assert_eq!(get_discrete_words(input), ("l alphabet".into(), None));
+    }
+
+    #[test]
+    fn html() {
+        let input = "<FormComponent data-pagefind-meta='[key:(value)]'>";
+        assert_eq!(
+            get_discrete_words(input),
+            ("form component data pagefind meta key value".into(), None)
+        );
+    }
+
+    #[test]
+    fn emoji() {
+        let input = "cloud🌦️cannon";
+        assert_eq!(
+            get_discrete_words(input),
+            ("cloud🌦️cannon".into(), Some(vec!["🌦️".into()]))
+        );
+
+        let input = "👋👨‍👩‍👧‍👦🌾";
+        assert_eq!(
+            get_discrete_words(input),
+            (
+                "👋👨‍👩‍👧‍👦🌾".into(),
+                Some(vec!["👋".into(), "👨‍👩‍👧‍👦".into(), "🌾".into()])
+            )
+        );
+    }
+}

pagefind/src/options.rs

@@ -4,7 +4,7 @@ use anyhow::{bail, Result};
 use clap::Parser;
 use rust_patch::Patch;
 use serde::{Deserialize, Serialize};
-use std::{collections::BTreeMap, env, path::PathBuf};
+use std::{env, path::PathBuf};
 use twelf::config;
 use typed_builder::TypedBuilder;
 
@@ -22,15 +22,6 @@ use crate::{
 // No options should be added that are required.
 //
 
-/// A selector paired with the Pagefind attributes to set on the elements it matches,
-/// so a site can be configured with the same vocabulary its HTML would use.
-#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
-pub(crate) struct PagefindRule {
-    pub(crate) selector: String,
-    #[serde(default)]
-    pub(crate) attributes: BTreeMap<String, String>,
-}
-
 #[config]
 #[derive(Parser, Debug, Clone)]
 #[clap(author, version, about, long_about = None)]
@@ -83,11 +74,6 @@ pub(crate) struct PagefindInboundConfig {
     #[serde(default)]
     pub(crate) exclude_selectors: Vec<String>,
 
-    /// Not a CLI flag: a list of selectors with attribute maps does not fit an argument.
-    #[clap(skip)]
-    #[serde(default)]
-    pub(crate) rules: Vec<PagefindRule>,
-
     #[clap(
         long,
         help = "The file glob Pagefind uses to find HTML files. Defaults to \"**/*.{html}\""
@@ -110,6 +96,14 @@ pub(crate) struct PagefindInboundConfig {
     #[clap(required = false)]
     pub(crate) include_characters: Option<String>,
 
+    #[clap(
+        long,
+        help = "Lowercase filter values when indexing, so that values differing only in case are merged into one."
+    )]
+    #[clap(required = false)]
+    #[serde(default = "defaults::default_false")]
+    pub(crate) lowercase_filters: bool,
+
     #[clap(
         long,
         help = "Serve the source directory after creating the search index"
@@ -213,6 +207,8 @@ pub struct PagefindServiceConfig {
     #[patch(as_option)]
     /// Include these characters when indexing and searching words.
     pub(crate) include_characters: Option<String>,
+    /// Lowercase filter values when indexing, so that values differing only in case are merged into one.
+    pub(crate) lowercase_filters: Option<bool>,
 }
 
 mod defaults {
@@ -241,10 +237,10 @@ pub(crate) struct SearchOptions {
     pub(crate) bundle_output: PathBuf,
     pub(crate) root_selector: String,
     pub(crate) exclude_selectors: Vec<String>,
-    pub(crate) rules: Vec<PagefindRule>,
     pub(crate) glob: String,
     pub(crate) force_language: Option<String>,
     pub(crate) include_characters: Vec<char>,
+    pub(crate) lowercase_filters: bool,
     pub(crate) version: &'static str,
     pub(crate) logger: Logger,
     pub(crate) keep_index_url: bool,
@@ -323,10 +319,10 @@ impl SearchOptions {
                 bundle_output,
                 root_selector: config.root_selector,
                 exclude_selectors: config.exclude_selectors,
-                rules: config.rules,
                 glob: config.glob,
                 force_language: config.force_language,
                 include_characters,
+                lowercase_filters: config.lowercase_filters,
                 version: env!("CARGO_PKG_VERSION"),
                 logger: Logger::new(
                     log_level,

pagefind_ui/component/components/index.ts

@@ -0,0 +1,46 @@
+export { getInstanceManager, configureInstance } from "./instance-manager";
+
+import { registerFunction } from "adequate-little-templates";
+
+registerFunction("resolveUrl", (url, pageUrl) => {
+  const s = String(url ?? "");
+  if (!s || /^[a-z][a-z0-9+.-]*:/i.test(s) || /^\/\//.test(s)) return s;
+  try {
+    const resolved = new URL(s, new URL(String(pageUrl ?? "/"), "https://p"));
+    return resolved.origin === "https://p"
+      ? resolved.pathname + resolved.search + resolved.hash
+      : resolved.href;
+  } catch {
+    return s;
+  }
+});
+
+import "./pagefind-config";
+import "./pagefind-input";
+import "./pagefind-summary";
+import "./pagefind-results";
+import "./pagefind-filter-pane";
+import "./pagefind-filter-dropdown";
+import "./pagefind-modal";
+import "./pagefind-modal-trigger";
+import "./pagefind-modal-header";
+import "./pagefind-modal-body";
+import "./pagefind-modal-footer";
+import "./pagefind-keyboard-hints";
+import "./pagefind-searchbox";
+
+export { PagefindConfig } from "./pagefind-config";
+export { PagefindInput } from "./pagefind-input";
+export { PagefindSummary } from "./pagefind-summary";
+export { PagefindResults } from "./pagefind-results";
+export { PagefindFilterPane } from "./pagefind-filter-pane";
+export { PagefindFilterDropdown } from "./pagefind-filter-dropdown";
+export { PagefindModal } from "./pagefind-modal";
+export { PagefindModalTrigger } from "./pagefind-modal-trigger";
+export { PagefindModalHeader } from "./pagefind-modal-header";
+export { PagefindModalBody } from "./pagefind-modal-body";
+export { PagefindModalFooter } from "./pagefind-modal-footer";
+export { PagefindKeyboardHints } from "./pagefind-keyboard-hints";
+export { PagefindSearchbox } from "./pagefind-searchbox";
+
+export { PagefindElement } from "./base-element";

pagefind_ui/default/svelte/result.svelte

@@ -0,0 +1,177 @@
+<script>
+    export let show_images = true;
+    export let process_result = null;
+    export let result = { data: async () => {} };
+
+    const skipMeta = ["title", "image", "image_alt", "url"];
+
+    let data;
+    let meta = [];
+
+    const resolveImageUrl = (src, pageUrl) => {
+        if (!src || /^[a-z][a-z0-9+.-]*:/i.test(src) || /^\/\//.test(src)) return src;
+        try {
+            const url = new URL(src, new URL(pageUrl || "/", "https://p"));
+            return url.origin === "https://p" ? url.pathname + url.search + url.hash : url.href;
+        } catch {
+            return src;
+        }
+    };
+
+    const load = async (r) => {
+        data = await r.data();
+        data = process_result?.(data) ?? data;
+        if (data.meta?.image) {
+            data = { ...data, meta: { ...data.meta, image: resolveImageUrl(data.meta.image, data.meta.url || data.url) } };
+        }
+        meta = Object.entries(data.meta).filter(
+            ([key]) => !skipMeta.includes(key),
+        );
+    };
+    $: load(result);
+
+    const placeholder = (max = 30) => {
+        return ". ".repeat(Math.floor(10 + Math.random() * max));
+    };
+</script>
+
+<li class="pagefind-ui__result">
+    {#if data}
+        {#if show_images}
+            <div class="pagefind-ui__result-thumb">
+                {#if data.meta.image}
+                    <img
+                        class="pagefind-ui__result-image"
+                        src={data.meta?.image}
+                        alt={data.meta?.image_alt || data.meta?.title}
+                    />
+                {/if}
+            </div>
+        {/if}
+        <div class="pagefind-ui__result-inner">
+            <p class="pagefind-ui__result-title">
+                <a
+                    class="pagefind-ui__result-link"
+                    href={data.meta?.url || data.url}>{data.meta?.title}</a
+                >
+            </p>
+            <p class="pagefind-ui__result-excerpt">{@html data.excerpt}</p>
+            {#if meta.length}
+                <ul class="pagefind-ui__result-tags">
+                    {#each meta as [metaTitle, metaValue]}
+                        <li
+                            class="pagefind-ui__result-tag"
+                            data-pagefind-ui-meta={metaTitle}
+                        >
+                            {metaTitle.replace(/^(\w)/, (c) =>
+                                c.toLocaleUpperCase(),
+                            )}: {metaValue}
+                        </li>
+                    {/each}
+                </ul>
+            {/if}
+        </div>
+    {:else}
+        {#if show_images}
+            <div class="pagefind-ui__result-thumb pagefind-ui__loading" />
+        {/if}
+        <div class="pagefind-ui__result-inner">
+            <p class="pagefind-ui__result-title pagefind-ui__loading">
+                {placeholder(30)}
+            </p>
+            <p class="pagefind-ui__result-excerpt pagefind-ui__loading">
+                {placeholder(40)}
+            </p>
+        </div>
+    {/if}
+</li>
+
+<style>
+    .pagefind-ui__result {
+        list-style-type: none;
+        display: flex;
+        align-items: flex-start;
+        gap: min(calc(40px * var(--pagefind-ui-scale)), 3%);
+        padding: calc(30px * var(--pagefind-ui-scale)) 0
+            calc(40px * var(--pagefind-ui-scale));
+        border-top: solid var(--pagefind-ui-border-width)
+            var(--pagefind-ui-border);
+    }
+    .pagefind-ui__result:last-of-type {
+        border-bottom: solid var(--pagefind-ui-border-width)
+            var(--pagefind-ui-border);
+    }
+    .pagefind-ui__result-thumb {
+        width: min(
+            30%,
+            calc((30% - (100px * var(--pagefind-ui-scale))) * 100000)
+        );
+        max-width: calc(120px * var(--pagefind-ui-scale));
+        margin-top: calc(10px * var(--pagefind-ui-scale));
+        aspect-ratio: var(--pagefind-ui-image-box-ratio);
+        position: relative;
+    }
+    .pagefind-ui__result-image {
+        display: block;
+        position: absolute;
+        left: 50%;
+        transform: translateX(-50%);
+        font-size: 0;
+        width: auto;
+        height: auto;
+        max-width: 100%;
+        max-height: 100%;
+        border-radius: var(--pagefind-ui-image-border-radius);
+    }
+    .pagefind-ui__result-inner {
+        flex: 1;
+        display: flex;
+        flex-direction: column;
+        align-items: flex-start;
+        margin-top: calc(10px * var(--pagefind-ui-scale));
+    }
+    .pagefind-ui__result-title {
+        display: inline-block;
+        font-weight: 700;
+        font-size: calc(21px * var(--pagefind-ui-scale));
+        margin-top: 0;
+        margin-bottom: 0;
+    }
+    .pagefind-ui__result-title .pagefind-ui__result-link {
+        color: var(--pagefind-ui-text);
+        text-decoration: none;
+    }
+    .pagefind-ui__result-title .pagefind-ui__result-link:hover {
+        text-decoration: underline;
+    }
+    .pagefind-ui__result-excerpt {
+        display: inline-block;
+        font-weight: 400;
+        font-size: calc(16px * var(--pagefind-ui-scale));
+        margin-top: calc(4px * var(--pagefind-ui-scale));
+        margin-bottom: 0;
+        min-width: calc(250px * var(--pagefind-ui-scale));
+    }
+    .pagefind-ui__loading {
+        color: var(--pagefind-ui-text);
+        background-color: var(--pagefind-ui-text);
+        border-radius: var(--pagefind-ui-border-radius);
+        opacity: 0.1;
+        pointer-events: none;
+    }
+    .pagefind-ui__result-tags {
+        list-style-type: none;
+        padding: 0;
+        display: flex;
+        gap: calc(20px * var(--pagefind-ui-scale));
+        flex-wrap: wrap;
+        margin-top: calc(20px * var(--pagefind-ui-scale));
+    }
+    .pagefind-ui__result-tag {
+        padding: calc(4px * var(--pagefind-ui-scale))
+            calc(8px * var(--pagefind-ui-scale));
+        font-size: calc(14px * var(--pagefind-ui-scale));
+        border-radius: var(--pagefind-ui-border-radius);
+        background-color: var(--pagefind-ui-tag);
+    }
+</style>

pagefind_ui/default/svelte/result_with_subs.svelte

@@ -0,0 +1,224 @@
+<script>
+  export let show_images = true;
+  export let process_result = null;
+  export let result = { data: async () => {} };
+
+  const skipMeta = ["title", "image", "image_alt", "url"];
+
+  let data;
+  let meta = [];
+  let non_root_sub_results = [];
+  let has_root_sub_result = false;
+
+  const resolveImageUrl = (src, pageUrl) => {
+    if (!src || /^[a-z][a-z0-9+.-]*:/i.test(src) || /^\/\//.test(src)) return src;
+    try {
+      const url = new URL(src, new URL(pageUrl || "/", "https://p"));
+      return url.origin === "https://p" ? url.pathname + url.search + url.hash : url.href;
+    } catch {
+      return src;
+    }
+  };
+
+  const thin_sub_results = (results, limit) => {
+    if (results.length <= limit) {
+      return results;
+    }
+
+    const top_results = [...results]
+      .sort((a, b) => b.locations.length - a.locations.length)
+      .slice(0, 3)
+      .map((r) => r.url);
+
+    return results.filter((r) => top_results.includes(r.url));
+  };
+
+  const load = async (r) => {
+    data = await r.data();
+    data = process_result?.(data) ?? data;
+    if (data.meta?.image) {
+      data = { ...data, meta: { ...data.meta, image: resolveImageUrl(data.meta.image, data.meta.url || data.url) } };
+    }
+    meta = Object.entries(data.meta).filter(([key]) => !skipMeta.includes(key));
+    if (Array.isArray(data.sub_results)) {
+      has_root_sub_result =
+        data.sub_results?.[0]?.url === (data.meta?.url || data.url);
+      if (has_root_sub_result) {
+        non_root_sub_results = thin_sub_results(data.sub_results.slice(1), 3);
+      } else {
+        non_root_sub_results = thin_sub_results([...data.sub_results], 3);
+      }
+    }
+  };
+  $: load(result);
+
+  const placeholder = (max = 30) => {
+    return ". ".repeat(Math.floor(10 + Math.random() * max));
+  };
+</script>
+
+<li class="pagefind-ui__result">
+  {#if data}
+    {#if show_images}
+      <div class="pagefind-ui__result-thumb">
+        {#if data.meta.image}
+          <img
+            class="pagefind-ui__result-image"
+            src={data.meta?.image}
+            alt={data.meta?.image_alt || data.meta?.title}
+          />
+        {/if}
+      </div>
+    {/if}
+    <div class="pagefind-ui__result-inner">
+      <p class="pagefind-ui__result-title">
+        <a class="pagefind-ui__result-link" href={data.meta?.url || data.url}
+          >{data.meta?.title}</a
+        >
+      </p>
+      {#if has_root_sub_result}
+        <p class="pagefind-ui__result-excerpt">{@html data.excerpt}</p>
+      {/if}
+
+      {#each non_root_sub_results as subres}
+        <div class="pagefind-ui__result-nested">
+          <p class="pagefind-ui__result-title">
+            <a class="pagefind-ui__result-link" href={subres.url}
+              >{subres.title}</a
+            >
+          </p>
+          <p class="pagefind-ui__result-excerpt">{@html subres.excerpt}</p>
+        </div>
+      {/each}
+
+      {#if meta.length}
+        <ul class="pagefind-ui__result-tags">
+          {#each meta as [metaTitle, metaValue]}
+              <li
+                class="pagefind-ui__result-tag"
+                data-pagefind-ui-meta={metaTitle}
+              >
+              {metaTitle.replace(/^(\w)/, (c) => c.toLocaleUpperCase())}: {metaValue}
+            </li>
+          {/each}
+        </ul>
+      {/if}
+    </div>
+  {:else}
+    {#if show_images}
+      <div class="pagefind-ui__result-thumb pagefind-ui__loading" />
+    {/if}
+    <div class="pagefind-ui__result-inner">
+      <p class="pagefind-ui__result-title pagefind-ui__loading">
+        {placeholder(30)}
+      </p>
+      <p class="pagefind-ui__result-excerpt pagefind-ui__loading">
+        {placeholder(40)}
+      </p>
+    </div>
+  {/if}
+</li>
+
+<style>
+  .pagefind-ui__result {
+    list-style-type: none;
+    display: flex;
+    align-items: flex-start;
+    gap: min(calc(40px * var(--pagefind-ui-scale)), 3%);
+    padding: calc(30px * var(--pagefind-ui-scale)) 0
+      calc(40px * var(--pagefind-ui-scale));
+    border-top: solid var(--pagefind-ui-border-width) var(--pagefind-ui-border);
+  }
+  .pagefind-ui__result:last-of-type {
+    border-bottom: solid var(--pagefind-ui-border-width)
+      var(--pagefind-ui-border);
+  }
+  .pagefind-ui__result-nested {
+    display: flex;
+    flex-direction: column;
+    padding-left: calc(20px * var(--pagefind-ui-scale));
+  }
+  .pagefind-ui__result-nested:first-of-type {
+    padding-top: calc(10px * var(--pagefind-ui-scale));
+  }
+  .pagefind-ui__result-nested .pagefind-ui__result-link {
+    font-size: 0.9em;
+    position: relative;
+  }
+  .pagefind-ui__result-nested .pagefind-ui__result-link::before {
+    content: "⤷ ";
+    position: absolute;
+    top: 0;
+    right: calc(100% + 0.1em);
+  }
+  .pagefind-ui__result-thumb {
+    width: min(30%, calc((30% - (100px * var(--pagefind-ui-scale))) * 100000));
+    max-width: calc(120px * var(--pagefind-ui-scale));
+    margin-top: calc(10px * var(--pagefind-ui-scale));
+    aspect-ratio: var(--pagefind-ui-image-box-ratio);
+    position: relative;
+  }
+  .pagefind-ui__result-image {
+    display: block;
+    position: absolute;
+    left: 50%;
+    transform: translateX(-50%);
+    font-size: 0;
+    width: auto;
+    height: auto;
+    max-width: 100%;
+    max-height: 100%;
+    border-radius: var(--pagefind-ui-image-border-radius);
+  }
+  .pagefind-ui__result-inner {
+    flex: 1;
+    display: flex;
+    flex-direction: column;
+    align-items: flex-start;
+    margin-top: calc(10px * var(--pagefind-ui-scale));
+  }
+  .pagefind-ui__result-title {
+    display: inline-block;
+    font-weight: 700;
+    font-size: calc(21px * var(--pagefind-ui-scale));
+    margin-top: 0;
+    margin-bottom: 0;
+  }
+  .pagefind-ui__result-title .pagefind-ui__result-link {
+    color: var(--pagefind-ui-text);
+    text-decoration: none;
+  }
+  .pagefind-ui__result-title .pagefind-ui__result-link:hover {
+    text-decoration: underline;
+  }
+  .pagefind-ui__result-excerpt {
+    display: inline-block;
+    font-weight: 400;
+    font-size: calc(16px * var(--pagefind-ui-scale));
+    margin-top: calc(4px * var(--pagefind-ui-scale));
+    margin-bottom: 0;
+    min-width: calc(250px * var(--pagefind-ui-scale));
+  }
+  .pagefind-ui__loading {
+    color: var(--pagefind-ui-text);
+    background-color: var(--pagefind-ui-text);
+    border-radius: var(--pagefind-ui-border-radius);
+    opacity: 0.1;
+    pointer-events: none;
+  }
+  .pagefind-ui__result-tags {
+    list-style-type: none;
+    padding: 0;
+    display: flex;
+    gap: calc(20px * var(--pagefind-ui-scale));
+    flex-wrap: wrap;
+    margin-top: calc(20px * var(--pagefind-ui-scale));
+  }
+  .pagefind-ui__result-tag {
+    padding: calc(4px * var(--pagefind-ui-scale))
+      calc(8px * var(--pagefind-ui-scale));
+    font-size: calc(14px * var(--pagefind-ui-scale));
+    border-radius: var(--pagefind-ui-border-radius);
+    background-color: var(--pagefind-ui-tag);
+  }
+</style>

pagefind_web_js/types/index.d.ts

@@ -0,0 +1,441 @@
+export {};
+
+declare global {
+  /** Global index options that can be passed to pagefind.options() */
+  type PagefindIndexOptions = {
+    /** Overrides the URL path that Pagefind uses to load its search bundle */
+    basePath?: string;
+    /** Appends the given baseURL to all search results. May be a path, or a full domain */
+    baseUrl?: string;
+    /** The maximum length of excerpts that Pagefind should generate for search results. Default to 30 */
+    excerptLength?: number;
+    /** If set, Pagefind will wrap excerpts with this string on any end that was truncated. Defaults to none */
+    excerptEllipsis?: string;
+    /**
+     * Multiply all rankings for this index by the given weight.
+     *
+     * Only applies in multisite setups, where one site should rank higher or lower than others.
+     */
+    indexWeight?: number;
+    /**
+     * Merge this filter object into all search queries in this index.
+     *
+     * Only applies in multisite setups.
+     */
+    mergeFilter?: Object;
+    /**
+     * If set, will add the search term as a query parameter under this key, for use with Pagefind's highlighting script.
+     */
+    highlightParam?: string;
+    language?: string;
+    /**
+     * If set, a merged index that fails to load is skipped instead of failing the whole search.
+     *
+     * Only applies in multisite setups, to indexes passed to mergeIndex.
+     */
+    optional?: boolean;
+    /**
+     * Whether an instance of Pagefind is the primary index or not (for multisite).
+     *
+     * This is set for you automatically, so it is unlikely you should set this directly.
+     */
+    primary?: boolean;
+    /**
+     * Provides the ability to fine tune Pagefind's ranking algorithm to better suit your dataset.
+     */
+    ranking?: PagefindRankingWeights;
+    /**
+     * If set, diacritics are treated as fully distinct words.
+     * This means searching for "café" will only match pages containing "café", not "cafe".
+     * When false (default), diacritics are normalized and all variants match,
+     * with ranking.diacriticSimilarity applied to favor close matches.
+     */
+    exactDiacritics?: boolean;
+    /**
+     * Force Pagefind to run on the main thread instead of using a web worker.
+     *
+     * By default, Pagefind will use a web worker for search operations when available,
+     * falling back to the main thread if workers are not supported or fail to initialize.
+     */
+    noWorker?: boolean;
+    /**
+     * If set, replaces the default cache-busting timestamp on the metadata request
+     * with a fixed string. Useful for PWAs and offline-capable sites so that
+     * a service worker can cache the request. Change this value each time
+     * you rebuild your site.
+     *
+     * The search index itself is unaffected and can always be cached.
+     */
+    metaCacheTag?: string;
+  };
+
+  type PagefindRankingWeights = {
+    /**
+      Controls page ranking based on similarity of terms to the search query (in length).
+      Increasing this number means pages rank higher when they contain words very close to the query,
+      e.g. if searching for `part` then `party` will boost a page higher than one containing `partition`.
+      Minimum value is 0.0, where `party` and `partition` would be viewed equally.
+    */
+    termSimilarity?: Number;
+    /**
+      Controls how much effect the average page length has on ranking.
+      Maximum value is 1.0, where ranking will strongly favour pages that are shorter than the average page on the site.
+      Minimum value is 0.0, where ranking will exclusively look at term frequency, regardless of how long a document is.
+    */
+    pageLength?: Number;
+    /**
+      Controls how quickly a term saturates on the page and reduces impact on the ranking.
+      Maximum value is 2.0, where pages will take a long time to saturate, and pages with very high term frequencies will take over.
+      As this number trends to 0, it does not take many terms to saturate and allow other parameters to influence the ranking.
+      Minimum value is 0.0, where terms will saturate immediately and results will not distinguish between one term and many.
+    */
+    termSaturation?: Number;
+    /**
+      Controls how much ranking uses term frequency versus raw term count.
+      Maximum value is 1.0, where term frequency fully applies and is the main ranking factor.
+      Minimum value is 0.0, where term frequency does not apply, and pages are ranked based on the raw sum of words and weights.
+      Values between 0.0 and 1.0 will interpolate between the two ranking methods.
+      Reducing this number is a good way to boost longer documents in your search results, as they no longer get penalized for having a low term frequency.
+      */
+    termFrequency?: Number;
+    /**
+      Controls how much boost is applied when the search query diacritics match the indexed word exactly.
+      At 1.0, searching for "café" will boost pages containing "café" by 100% over pages containing "cafe".
+      At 0.0, no boost is applied and all diacritic variants are treated equally.
+      Must be >= 0
+    */
+    diacriticSimilarity?: Number;
+    /**
+      Controls boost weights for metadata field matches.
+      Keys are meta field names (e.g., "title", "description").
+      Increasing values boost scores when a search term matches in that meta field.
+      Default: {"title": 5.0} meaning title matches get 5x boost.
+      Provided weights are merged with defaults.
+      Example: { description: 2.0 } — adds description boost while keeping the default title boost.
+    */
+    metaWeights?: Record<string, number>;
+  };
+
+  /** Options that can be passed to pagefind.search() */
+  type PagefindSearchOptions = {
+    /** If set, this call will load all assets but return before searching. Prefer using pagefind.preload() instead */
+    preload?: boolean;
+    /** Add more verbose console logging for this search query */
+    verbose?: boolean;
+    /** The set of filters to execute with this search. Input type is extremely flexible, see the filtering docs for details */
+    filters?: Object;
+    /** The set of sorts to use for this search, instead of relevancy */
+    sort?: Object;
+  };
+
+  /** Filter counts returned from pagefind.filters(), and alongside results from pagefind.search() */
+  type PagefindFilterCounts = Record<string, Record<string, number>>;
+
+  /** The main results object returned from a call to pagefind.search() */
+  type PagefindSearchResults = {
+    /** All pages that match the search query and filters provided */
+    results: PagefindSearchResult[];
+    /** How many results would there have been if you had omitted the filters */
+    unfilteredResultCount: number;
+    /** Given the query and filters provided, how many remaining results are there under each filter? */
+    filters: PagefindFilterCounts;
+    /** If the searched filters were removed, how many total results for each filter are there? */
+    totalFilters: PagefindFilterCounts;
+    /** Information on how long it took Pagefind to execute this query */
+    timings: {
+      preload: number;
+      search: number;
+      total: number;
+    };
+    /** Verbose information on stemming returned in the Pagefind Playground */
+    search_keywords?: string[];
+    /** Verbose query term IDF breakdown returned in the Pagefind Playground */
+    query_term_idfs?: PagefindQueryTermIdf[];
+  };
+
+  /** Query term IDF info for playground mode */
+  type PagefindQueryTermIdf = {
+    /** The search term (stemmed form) */
+    term: string;
+    /** IDF value for this term */
+    idf: number;
+  };
+
+  /** The main results object returned from a call to pagefind.search() */
+  type PagefindIndexesSearchResults = {
+    /** All pages that match the search query and filters provided */
+    results: PagefindSearchResult[];
+    /** How many results would there have been if you had omitted the filters */
+    unfilteredResultCount: number;
+    /** Given the query and filters provided, how many remaining results are there under each filter? */
+    filters: PagefindFilterCounts;
+    /** If the searched filters were removed, how many total results for each filter are there? */
+    totalFilters: PagefindFilterCounts;
+    /** Information on how long it took Pagefind to execute this query */
+    timings: {
+      preload: number;
+      search: number;
+      total: number;
+    }[];
+    /** Verbose information on stemming returned in the Pagefind Playground */
+    search_keywords?: string[];
+    /** Verbose query term IDF breakdown returned in the Pagefind Playground */
+    query_term_idfs?: PagefindQueryTermIdf[];
+    /** Verbose information on what environment the Pagefind search was executed in [worker, mainthread] */
+    search_environment?: string;
+  };
+
+  /** A single result from a search query, before actual data has been loaded */
+  type PagefindSearchResult = {
+    /** Pagefind's internal ID for this page, unique across the site */
+    id: string;
+    /** Pagefind's internal score for your query matching this page, that is used when ranking these results */
+    score: number;
+    /** The locations of all matching words in this page */
+    words: number[];
+    /** Verbose information returned in the Pagefind playground mode */
+    params?: PagefindTermParams;
+    /** Verbose information returned in the Pagefind playground mode */
+    scores?: PagefindTermScore[];
+    /**
+     * Which metadata fields (e.g., "title", "author") matched the search query.
+     * Present when search terms were found in metadata fields, allowing consumers to
+     * indicate matches came from metadata rather than body content.
+     */
+    matchedMetaFields?: string[];
+    /** Verbose metadata scoring returned in the Pagefind Playground */
+    verbose_meta_scores?: PagefindMetaScore[];
+    /**
+     * Calling data() loads the final data fragment needed to display this result.
+     *
+     * Only call this when you need to display the data, rather than all at once.
+     * (e.g. one page as a time, or in a scroll listener)
+     * */
+    data: () => Promise<PagefindSearchFragment>;
+  };
+
+  export type PagefindTermParams = {
+    /** Length of this result */
+    document_length: number;
+    /** Average page length */
+    average_page_length: number;
+    /** Total pages */
+    total_pages: number;
+  };
+
+  type PagefindTermScore = {
+    /** Term */
+    search_term: string;
+    /** Inverse document frequency for term — how common is this word */
+    idf: number;
+    /** Term frequency, saturating */
+    saturating_tf: number;
+    /** Term frequency, raw counts */
+    raw_tf: number;
+    /** Pagefind output term frequency */
+    pagefind_tf: number;
+    /** Final score for term */
+    score: number;
+    /** Input parameters used for the score */
+    params: PagefindTermScoreParams;
+  };
+
+  export type PagefindTermScoreParams = {
+    /** Weighted term frequency */
+    weighted_term_frequency: number;
+    /** Pages containing term */
+    pages_containing_term: number;
+    /** Length bonus */
+    length_bonus: number;
+  };
+
+  /** Verbose metadata scoring info returned in the Pagefind Playground */
+  type PagefindMetaScore = {
+    /** Metadata field name */
+    field_name: string;
+    /** Configured weight for this field */
+    field_weight: number;
+    /** Which search terms matched in this field */
+    matched_terms: string[];
+    /** Sum of IDF for matched terms */
+    matched_idf: number;
+    /** Total IDF across all query terms */
+    query_total_idf: number;
+    /** Coverage ratio (matched_idf / query_total_idf) */
+    coverage: number;
+    /** Final boost contribution from this field */
+    coverage_boost: number;
+  };
+
+  /** The useful data Pagefind provides for a search result */
+  type PagefindSearchFragment = {
+    /** Pagefind's processed URL for this page. Will include the baseUrl if configured */
+    url: string;
+    /** Pagefind's unprocessed URL for this page */
+    raw_url?: string;
+    /** The full processed content text of this page */
+    content: string;
+    /** Internal type — ignore for now */
+    raw_content?: string;
+    /** The processed excerpt for this result, with matching terms wrapping in `<mark>` elements */
+    excerpt: string;
+    /** The processed excerpt for this result, without any `<mark>` elements wrapping matching terms */
+    plain_excerpt: string;
+    /**
+     * What regions of the page matched this search query?
+     *
+     * Precalculates based on h1->6 tags with IDs, using the text between each.
+     */
+    sub_results: PagefindSubResult[];
+    /** How many total words are there on this page? */
+    word_count: number;
+    /** The locations of all matching words in this page */
+    locations: number[];
+    /**
+     * The locations of all matching words in this page,
+     * paired with data about their weight and relevance to this query
+     */
+    weighted_locations: PagefindWordLocation[];
+    /** The filter keys and values this page was tagged with */
+    filters: Record<string, string[]>;
+    /** The metadata keys and values this page was tagged with */
+    meta: Record<string, string>;
+    /**
+     * The raw anchor data that Pagefind used to generate sub_results.
+     *
+     * Contains _all_ elements that had IDs on the page, so can be used to
+     * implement your own sub result calculations with different semantics.
+     */
+    anchors: PagefindSearchAnchor[];
+  };
+
+  /** Data for a matched section within a page */
+  type PagefindSubResult = {
+    /**
+     * Title of this sub result — derived from the heading content.
+     *
+     * If this is a result for the section of the page before any headings with IDs,
+     * this will be the same as the page's meta.title value.
+     */
+    title: string;
+    /**
+     * Direct URL to this sub result, comprised of the page's URL plus the hash string of the heading.
+     *
+     * If this is a result for the section of the page before any headings with IDs,
+     * this will be the same as the page URL.
+     */
+    url: string;
+    /** The locations of all matching words in this segment */
+    locations: number[];
+    /**
+     * The locations of all matching words in this segment,
+     * paired with data about their weight and relevance to this query
+     */
+    weighted_locations: PagefindWordLocation[];
+    /** The processed excerpt for this segment, with matching terms wrapping in `<mark>` elements */
+    excerpt: string;
+    /** The processed excerpt for this segment, without any `<mark>` elements wrapping matching terms */
+    plain_excerpt: string;
+    /**
+     * Raw data about the anchor element associated with this sub result.
+     *
+     * The omission of this field means this sub result is for text found on the page
+     * before the first heading that had an ID.
+     */
+    anchor?: PagefindSearchAnchor;
+  };
+
+  /** Information about a matching word on a page */
+  type PagefindWordLocation = {
+    /** The weight that this word was originally tagged as */
+    weight: number;
+    /**
+     * An internal score that Pagefind calculated for this word.
+     *
+     * The absolute value is somewhat meaningless, but the value can be used
+     * in comparison to other values in this set of search results to perform custom ranking.
+     */
+    balanced_score: number;
+    /**
+     * The index of this word in the result content.
+     *
+     * Splitting the content key by whitespacing and indexing by this number
+     * will yield the correct word.
+     */
+    location: number;
+    /**
+     * Verbose word information returned when running Pagefind in playground mode.
+     */
+    verbose?: PagefindVerboseWordLocation;
+  };
+
+  /** Verbose playground information about a matching word on a page */
+  type PagefindVerboseWordLocation = {
+    /**
+     * The indexed string for this word, usually stemmed.
+     */
+    word_string: string;
+    /**
+     * The scoring bonus this word received based on length similarity to a search term.
+     */
+    length_bonus: number;
+  };
+
+  /**
+   * An independent Pagefind instance returned by createInstance().
+   * Each instance has its own configuration and search state.
+   * All instances share a single web worker and WASM module internally.
+   */
+  type PagefindInstance = {
+    /** Update options for this instance */
+    options: (opts: PagefindIndexOptions) => Promise<void>;
+    /** Wait for this instance to finish initializing (WASM load, etc.) */
+    init: () => Promise<void>;
+    /** Destroy this instance and terminate its worker */
+    destroy: () => Promise<void>;
+    /** Merge an additional index into this instance */
+    mergeIndex: (
+      indexPath: string,
+      options: PagefindIndexOptions,
+    ) => Promise<void>;
+    /** Search this instance's index */
+    search: (
+      term: string,
+      options?: PagefindSearchOptions,
+    ) => Promise<PagefindIndexesSearchResults>;
+    /** Debounced search — returns null if superseded by a newer call */
+    debouncedSearch: (
+      term: string,
+      options?: PagefindSearchOptions,
+      debounceTimeoutMs?: number,
+    ) => Promise<PagefindIndexesSearchResults | null>;
+    /** Preload index chunks for a search term without returning results */
+    preload: (term: string, options?: PagefindSearchOptions) => Promise<void>;
+    /** Retrieve all available filter values and counts */
+    filters: () => Promise<PagefindFilterCounts>;
+  };
+
+  /** Raw data about elements with IDs that Pagefind encountered when indexing the page */
+  type PagefindSearchAnchor = {
+    /** What element type was this anchor? e.g. `h1`, `div` */
+    element: string;
+    /** The raw id="..." attribute contents of the element */
+    id: string;
+    /** An explicit sub-result title supplied by a data-pagefind-title attribute */
+    title?: string;
+    /**
+     * The text content of this element.
+     *
+     * In order to prevent repeating most of the page data for every anchor,
+     * Pagefind will only take top level text nodes, or text nodes nested within
+     * inline elements such as <a> and <span>.
+     */
+    text?: string;
+    /**
+     * The position of this anchor in the result content.
+     * Splitting the content key by whitespacing and indexing by this number
+     * will yield the first word indexed after this element's ID was found.
+     */
+    location: number;
+  };
+}

wrappers/node/lib/index.js

@@ -0,0 +1,304 @@
+import { PagefindService } from "./service.js";
+import { decode } from "./encoding.js";
+
+/**
+ * @typedef {import('pagefindInternal').InternalResponseCallback} InternalResponseCallback
+ * @typedef {import('pagefindInternal').InternalResponsePayload} InternalResponsePayload
+ */
+
+/**
+ * @type {PagefindService?}
+ */
+let persistentService;
+const launch = () => {
+    if (!persistentService) {
+        persistentService = new PagefindService();
+    }
+    return persistentService;
+}
+
+/**
+ * @template T
+ * @param {function(any): void} resolve 
+ * @param {function(any): void} reject 
+ * @param {InternalResponseCallback} response_callback 
+ * @param {function(InternalResponsePayload): T} resultFn 
+ */
+const handleApiResponse = (resolve, reject, { exception, err, result }, resultFn) => {
+    if (exception) {
+        reject(exception);
+    } else {
+        resolve({
+            errors: err ? [err.message] : [],
+            ...(result ? resultFn(result) : {})
+        });
+    }
+}
+
+/** 
+ * @typedef {import('pagefindService').NewIndexResponse} NewIndexResponse
+ * 
+ * @param {import('pagefindService').PagefindServiceConfig=} config
+ * @type {import('pagefindService').createIndex} 
+ * */
+export const createIndex = (config) => new Promise((resolve, reject) => {
+
+    // TODO: Validate `config` here, instead of waiting for the backend to throw an error.
+    // Ideally we create a global Pagefind config JSON schema that (a subset of) can be used here.
+
+    const action = 'NewIndex';
+    launch().sendMessage(
+        {
+            type: action,
+            config: {
+                root_selector: config?.rootSelector,
+                exclude_selectors: config?.excludeSelectors,
+                force_language: config?.forceLanguage,
+                verbose: config?.verbose,
+                logfile: config?.logfile,
+                keep_index_url: config?.keepIndexUrl,
+                write_playground: config?.writePlayground,
+                include_characters: config?.includeCharacters,
+                lowercase_filters: config?.lowercaseFilters,
+            }
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<NewIndexResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== action) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    index: indexFns(success.index_id),
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+/**
+ * @type {import('pagefindService').close} 
+ */
+export const close = () => new Promise((resolve, reject) => {
+    persistentService?.close(null);
+    persistentService = null;
+    resolve(null);
+});
+
+/**
+ * @param {number} indexId 
+ * @returns {import ('pagefindService').PagefindIndex}
+ */
+const indexFns = (indexId) => {
+    return {
+        addHTMLFile: (file) => addHTMLFile(indexId, file),
+        addCustomRecord: (record) => addCustomRecord(indexId, record),
+        addDirectory: (dir) => addDirectory(indexId, dir),
+        writeFiles: (options) => writeFiles(indexId, options),
+        getFiles: () => getFiles(indexId),
+        deleteIndex: () => deleteIndex(indexId)
+    }
+}
+
+/**
+ * @typedef {import('pagefindService').NewFileResponse} NewFileResponse
+ * 
+ * @param {number} indexId 
+ * @param {import('pagefindService').HTMLFile} file
+ * @returns {Promise<NewFileResponse>}
+ */
+const addHTMLFile = (indexId, file) => new Promise((resolve, reject) => {
+    const action = 'AddFile';
+    const responseAction = 'IndexedFile';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+            file_path: file.sourcePath,
+            url: file.url,
+            file_contents: file.content
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<NewFileResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== responseAction) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    file: {
+                        uniqueWords: success.page_word_count,
+                        url: success.page_url,
+                        meta: success.page_meta,
+                    }
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+/**
+ * @param {number} indexId 
+ * @param {import('pagefindService').CustomRecord} record
+ * @returns {Promise<NewFileResponse>}
+ */
+const addCustomRecord = (indexId, record) => new Promise((resolve, reject) => {
+    const action = 'AddRecord';
+    const responseAction = 'IndexedFile';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+            url: record.url,
+            content: record.content,
+            language: record.language,
+            meta: record.meta,
+            filters: record.filters,
+            sort: record.sort,
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<NewFileResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== responseAction) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    file: {
+                        uniqueWords: success.page_word_count,
+                        url: success.page_url,
+                        meta: success.page_meta,
+                    }
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+
+/**
+ * @typedef {import('pagefindService').IndexingResponse} IndexingResponse
+ * 
+ * @param {number} indexId 
+ * @param {import('pagefindService').SiteDirectory} dir
+ * @returns {Promise<IndexingResponse>}
+ */
+const addDirectory = (indexId, dir) => new Promise((resolve, reject) => {
+    const action = 'AddDir';
+    const responseAction = 'IndexedDir';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+            path: dir.path,
+            glob: dir.glob
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<IndexingResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== responseAction) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    page_count: success.page_count
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+/**
+ * @typedef {import ('pagefindService').WriteFilesResponse} WriteFilesResponse
+ * 
+ * @param {number} indexId 
+ * @param {import('pagefindService').WriteOptions=} options
+ * @returns {Promise<WriteFilesResponse>}
+ */
+const writeFiles = (indexId, options) => new Promise((resolve, reject) => {
+    const action = 'WriteFiles';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+            output_path: options?.outputPath
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<WriteFilesResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== action) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    outputPath: success.output_path
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+/**
+ * @typedef {import ('pagefindService').GetFilesResponse} GetFilesResponse
+ * 
+ * @param {number} indexId 
+ * @returns {Promise<GetFilesResponse>}
+ */
+const getFiles = (indexId) => new Promise((resolve, reject) => {
+    const action = 'GetFiles';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<GetFilesResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== action) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return {
+                    files: success.files.map(file => {
+                        return {
+                            path: file.path,
+                            content: decode(file.content)
+                        }
+                    })
+                }
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});
+
+/**
+ * @param {number} indexId 
+ * @returns {Promise<null>}
+ */
+const deleteIndex = (indexId) => new Promise((resolve, reject) => {
+    const action = 'DeleteIndex';
+    launch().sendMessage(
+        {
+            type: action,
+            index_id: indexId,
+        }, (response) => {
+            /** @type {function(InternalResponsePayload): Omit<GetFilesResponse, 'errors'>?} */
+            const successCallback = (success) => {
+                if (success.type !== action) {
+                    reject(`Message returned from backend should have been ${action}, but was ${success.type}`);
+                    return null;
+                }
+
+                return null;
+            };
+            handleApiResponse(resolve, reject, response, successCallback);
+        }
+    );
+});

wrappers/node/types/index.d.ts

@@ -0,0 +1,231 @@
+/**
+ * Create a new Pagefind index that files can be added to
+ */
+export function createIndex(config?: PagefindServiceConfig): Promise<NewIndexResponse>;
+
+/**
+ * Close the Pagefind service and clean up, stopping the binary altogether.
+ * 
+ * Service _will_ be restarted if any new calls are made to any other function.
+ * 
+ * Calling functions on an existing index after calling close() is undefined behavior,
+ * as that index may map to a new index after the service has restarted.
+ */
+export function close(): Promise<null>;
+
+export interface PagefindServiceConfig {
+    /** 
+     * The element Pagefind should treat as the root of the document, defaults to `html`.
+     * Usually you will want to use the data-pagefind-body attribute instead.
+     * @example ".my-html-outer"
+     */
+    rootSelector?: string,
+    /** 
+     * Custom selectors that Pagefind should ignore when indexing.
+     * Usually you will want to use the data-pagefind-ignore attribute instead.
+     * @example ["svg", ".my-code-blocks"]
+     */
+    excludeSelectors?: string[],
+    /**
+     * Ignore any detected languages and index the whole site as a single language.
+     * Expects an ISO 639-1 code.
+     */
+    forceLanguage?: string,
+    /**
+     * Print verbose logging while indexing the site. Does not impact the web-facing search.
+     * When running as a service, only impacts the logfile (if present).
+     */
+    verbose?: boolean,
+    /**
+     * Path to a logfile to write to. Will replace the file on each run.
+     */
+    logfile?: string,
+    /**
+     * Keep `index.html` at the end of search result paths.
+     * Defaults to false, stripping `index.html`.
+     */
+    keepIndexUrl?: boolean,
+    /**
+     * When writing or outputting files, also write the Pagefind playground to /pagefind/playground/.
+     * Defaults to false, ensuring the playground isn't available on a live site.
+     */
+    writePlayground?: boolean,
+    /**
+     * Include these characters when indexing and searching words.
+     * Useful for sites documenting technical topics such as programming languages.
+     * @example "<>$"
+     */
+    includeCharacters?: string,
+    /**
+     * Lowercase filter values when indexing, so that values differing only in case are merged into one.
+     */
+    lowercaseFilters?: boolean,
+}
+
+
+export interface NewIndexResponse {
+    errors: string[],
+    index?: PagefindIndex
+}
+
+/**
+ * A Pagefind index that exists in the backend service before being built
+ */
+export interface PagefindIndex {
+    addHTMLFile: typeof addHTMLFile,
+    addCustomRecord: typeof addCustomRecord,
+    addDirectory: typeof addDirectory,
+    writeFiles: typeof writeFiles,
+    getFiles: typeof getFiles,
+    deleteIndex: typeof deleteIndex,
+}
+
+/**
+ * Index an HTML file that isn't on disk
+ */
+declare function addHTMLFile(file: HTMLFile): Promise<NewFileResponse>;
+/**
+ * Index a custom record that isn't backed by an HTML file
+ */
+declare function addCustomRecord(record: CustomRecord): Promise<NewFileResponse>;
+/**
+ * Index a directory of HTML files from disk
+ */
+declare function addDirectory(path: SiteDirectory): Promise<IndexingResponse>;
+
+/**
+ * The data required for Pagefind to index an HTML file that isn't on disk
+ * @example
+ * {
+ *   sourcePath: "about/index.html",
+ *   content: "<html lang='en'><body><h1>Meet the team</h1></body></html>"
+ * }
+ */
+export interface HTMLFile {
+    /** 
+     * The source path of the HTML file if it were to exist on disk.
+     * Must be a relative path, or an absolute path within the current working directory.
+     * Pagefind will compute the result URL from this path.
+     * 
+     * If not supplied, url must be supplied.
+     * 
+     * @example "about/index.html"
+     * @example "/Users/user/Documents/site/about/index.html"
+     */
+    sourcePath?: string,
+    /** 
+     * An explicit URL to use, instead of having Pagefind
+     * compute the URL based on the sourcePath.
+     * 
+     * If not supplied, sourcePath must be supplied.
+     * 
+     * @example "/about/"
+     */
+    url?: string,
+    /** The source HTML content of the file to be parsed */
+    content: string
+}
+
+/**
+ * The data required for Pagefind to index a custom record that isn't backed by an HTML file
+ * @example
+ * {
+ *   url: "/about/",
+ *   content: "Meet the team",
+ *   language: "en"
+ * }
+ */
+export interface CustomRecord {
+    /** The output URL of this record. Pagefind will not alter this */
+    url: string,
+    /** The raw content of this record */
+    content: string,
+    /** What language is this record written in. Multiple languages will be split into separate indexes. Expects an ISO 639-1 code. */
+    language: string,
+    /** The metadata to attach to this record. Supplying a `title` is highly recommended */
+    meta?: Record<string, string>,
+    /** The filters to attach to this record */
+    filters?: Record<string, string[]>,
+    /** The sort keys to attach to this record */
+    sort?: Record<string, string>
+}
+
+/**
+ * The data required for Pagefind to index the files in a directory
+ * @example
+ * {
+ *   path: "public",
+ *   glob: "**\/*.{html}"
+ * }
+ */
+export interface SiteDirectory {
+    /**
+     * The path to the directory to index.
+     * If relative, is relative to the cwd.
+     */
+    path: string,
+    /** Optionally, a custom glob to evaluate for finding files. Default to all HTML files. */
+    glob?: string
+}
+
+export interface IndexingResponse {
+    errors: string[],
+    page_count: number
+}
+
+export interface NewFileResponse {
+    errors: string[],
+    file: NewFile
+}
+
+/**
+ * Details about a new file that has been successfully added to the Pagefind index
+ */
+export interface NewFile {
+    uniqueWords: number,
+    url: string,
+    meta: Record<string, string>
+}
+
+/**
+ * Write the index files to disk
+ */
+declare function writeFiles(options?: WriteOptions): Promise<WriteFilesResponse>;
+
+/**
+ * Options for writing a Pagefind index to disk
+ */
+export interface WriteOptions {
+    /** 
+     * The path of the pagefind bundle directory to write to disk.
+     * If relative, is relative to the cwd.
+     * @example "./public/pagefind"
+     */
+    outputPath: string
+}
+
+
+export interface WriteFilesResponse {
+    errors: string[],
+    outputPath: string
+}
+
+/**
+ * Get an in-memory copy of the built index files
+ */
+declare function getFiles(): Promise<GetFilesResponse>;
+
+export interface GetFilesResponse {
+    errors: string[],
+    files: IndexFile[]
+}
+
+export interface IndexFile {
+    path: string,
+    content: Uint8Array
+}
+
+/**
+ * Delete this index and clear it from memory
+ */
+declare function deleteIndex(): Promise<null>;

wrappers/node/types/internal.d.ts

@@ -0,0 +1,155 @@
+// Requests to the backend.
+
+/**
+ * The raw object passed to the backend binary
+ */
+export interface InternalServiceRequest {
+    message_id?: number,
+    payload: InternalRequestPayload
+}
+
+/**
+ * The payload describing an action to the backend binary
+ */
+export type InternalRequestPayload =
+  | InternalNewIndexRequest
+  | InternalAddFileRequest
+  | InternalAddRecordRequest
+  | InternalAddDirRequest
+  | InternalWriteFilesRequest
+  | InternalGetFilesRequest
+  | InternalDeleteIndexRequest;
+
+export interface InternalNewIndexRequest {
+    type: 'NewIndex',
+    config?: InternalPagefindServiceConfig
+}
+
+export interface InternalPagefindServiceConfig {
+    root_selector?: string,
+    exclude_selectors?: string[],
+    force_language?: string,
+    verbose?: boolean,
+    logfile?: string,
+    keep_index_url?: boolean,
+    write_playground?: boolean,
+    include_characters?: string,
+    lowercase_filters?: boolean,
+}
+
+export interface InternalAddFileRequest {
+    type: 'AddFile',
+    index_id: number,
+    file_path?: string,
+    url?: string,
+    file_contents: string
+}
+
+export interface InternalAddRecordRequest {
+    type: 'AddRecord',
+    index_id: number,
+    url: string,
+    content: string,
+    language: string,
+    meta?: Record<string, string>,
+    filters?: Record<string, string[]>,
+    sort?: Record<string, string>
+}
+
+export interface InternalAddDirRequest {
+    type: 'AddDir',
+    index_id: number,
+    path: string,
+    glob?: string
+}
+
+export interface InternalWriteFilesRequest {
+    type: 'WriteFiles',
+    index_id: number,
+    output_path?: string
+}
+
+export interface InternalGetFilesRequest {
+    type: 'GetFiles',
+    index_id: number
+}
+
+export interface InternalDeleteIndexRequest {
+    type: 'DeleteIndex',
+    index_id: number
+}
+
+// Responses from the backend.
+
+/**
+ * The raw object returned from the backend binary
+ */
+export interface InternalServiceResponse {
+    message_id: number,
+    payload: InternalResponsePayload | InternalResponseError
+}
+
+/**
+ * The response payload in the case of an error
+ */
+export interface InternalResponseError {
+    type: 'Error',
+    original_message?: string,
+    message: string
+}
+
+/**
+ * The response payload in the case of a success
+ */
+export type InternalResponsePayload = 
+  | InternalNewIndexResponse
+  | InternalIndexedFileResponse
+  | InternalIndexedDirResponse
+  | InternalWriteFilesResponse
+  | InternalGetFilesResponse
+  | InternalDeleteIndexResponse;
+
+export interface InternalNewIndexResponse {
+    type: 'NewIndex',
+    index_id: number
+}
+
+export interface InternalIndexedFileResponse {
+    type: 'IndexedFile',
+    page_word_count: number,
+    page_url: string,
+    page_meta: Record<string, string>
+}
+
+export interface InternalIndexedDirResponse {
+    type: 'IndexedDir',
+    page_count: number
+}
+
+export interface InternalWriteFilesResponse {
+    type: 'WriteFiles',
+    output_path: string,
+}
+
+export interface InternalGetFilesResponse {
+    type: 'GetFiles',
+    files: InternalSyntheticFile[],
+}
+
+export interface InternalSyntheticFile {
+    path: string,
+    content: string
+}
+
+export interface InternalDeleteIndexResponse {
+    type: 'DeleteIndex'
+}
+
+/**
+ * What the service returns to the wrapping javascript detailing a response
+ */
+export interface InternalResponseCallback {
+    exception: Error | null,
+    err: InternalResponseError | null,
+    result: InternalResponsePayload | null
+}

wrappers/python/src/pagefind/index/__init__.py

@@ -0,0 +1,286 @@
+import logging
+import base64
+from typing import Any, Dict, List, Optional, Sequence, TypedDict, cast
+
+from ..service import PagefindService
+from ..service.types import (
+    InternalAddDirRequest,
+    InternalAddFileRequest,
+    InternalAddRecordRequest,
+    InternalDeleteIndexRequest,
+    InternalGetFilesRequest,
+    InternalGetFilesResponse,
+    InternalIndexedDirResponse,
+    InternalIndexedFileResponse,
+    InternalDecodedFile,
+    InternalWriteFilesRequest,
+)
+
+log = logging.getLogger(__name__)
+
+
+class IndexConfig(TypedDict, total=False):
+    root_selector: Optional[str]
+    """
+    The root selector to use for the index.
+    If not supplied, Pagefind will use the ``<html>`` tag.
+    """
+    exclude_selectors: Optional[Sequence[str]]
+    """Extra element selectors that Pagefind should ignore when indexing."""
+    force_language: Optional[str]
+    """
+    Ignores any detected languages and creates a single index for the entire site as the
+    provided language. Expects an ISO 639-1 code, such as ``en`` or ``pt``.
+    """
+    verbose: Optional[bool]
+    """
+    Prints extra logging while indexing the site. Only affects the CLI, does not impact
+    web-facing search.
+    """
+    logfile: Optional[str]
+    """
+    A path to a file to log indexing output to in addition to stdout.
+    The file will be created if it doesn't exist and overwritten on each run.
+    """
+    keep_index_url: Optional[bool]
+    """Whether to keep ``index.html`` at the end of search result paths.
+
+    By default, a file at ``animals/cat/index.html`` will be given the URL
+    ``/animals/cat/``. Setting this option to ``true`` will result in the URL
+    ``/animals/cat/index.html``.
+    """
+    write_playground: Optional[bool]
+    """When writing or outputting files, also write the Pagefind playground to /pagefind/playground/.
+
+    Defaults to false, ensuring the playground isn't available on a live site.
+    """
+    include_characters: Optional[str]
+    """Include these characters when indexing and searching words.
+
+    Useful for sites documenting technical topics such as programming languages.
+    """
+    lowercase_filters: Optional[bool]
+    """Lowercase filter values when indexing, so that values differing only in case are merged into one."""
+    output_path: Optional[str]
+    """
+    The folder to output the search bundle into, relative to the processed site.
+    Defaults to ``pagefind``.
+    """
+
+
+class PagefindIndex:
+    """Manages a Pagefind index.
+
+    ``PagefindIndex`` operates as an async contextmanager.
+    Entering the context starts a backing Pagefind service and creates an in-memory index in the backing service.
+    Exiting the context writes the in-memory index to disk and then shuts down the backing Pagefind service.
+
+    Each method of ``PagefindIndex`` that talks to the backing Pagefind service can raise errors.
+    If an exception is is rased inside ``PagefindIndex``'s context, the context closes without writing the index files to disk.
+
+    ``PagefindIndex`` optionally takes a configuration dictionary that can apply parts of the [Pagefind CLI config](/docs/config-options/). The options available at this level are:
+
+    See the relevant documentation for these configuration options in the
+    `Configuring the Pagefind CLI <https://pagefind.app/docs/config-options/>` documentation.
+    """
+
+    _service: Optional["PagefindService"] = None
+    _index_id: Optional[int] = None
+    _config: Optional[IndexConfig] = None
+    """Note that config should be immutable."""
+
+    def __init__(
+        self,
+        config: Optional[IndexConfig] = None,
+        *,
+        _service: Optional["PagefindService"] = None,
+        _index_id: Optional[int] = None,
+    ):
+        self._service = _service
+        self._index_id = _index_id
+        self._config = config
+
+    async def _start(self) -> "PagefindIndex":
+        """Start the backing Pagefind service and create an in-memory index."""
+        assert self._index_id is None
+        assert self._service is None
+        self._service = await PagefindService().launch()
+        _index = await self._service.create_index(self._config)
+        self._index_id = _index._index_id
+        return self
+
+    async def add_html_file(
+        self,
+        *,
+        content: str,
+        source_path: Optional[str] = None,
+        url: Optional[str] = None,
+    ) -> InternalIndexedFileResponse:
+        """Add an HTML file to the index.
+
+        :param content: The source HTML content of the file to be parsed.
+        :param source_path: The source path of the HTML file would have on disk. \
+            Must be a relative path, or an absolute path within the current working directory. \
+            Pagefind will compute the result URL from this path.
+        :param url: an explicit URL to use, instead of having Pagefind compute the \
+            URL based on the source_path. If not supplied, source_path must be supplied.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+        result = await self._service.send(
+            InternalAddFileRequest(
+                type="AddFile",
+                index_id=self._index_id,
+                url=url,
+                file_contents=content,
+                file_path=source_path,
+            )
+        )
+        assert result["type"] == "IndexedFile"
+        return cast(InternalIndexedFileResponse, result)
+
+    async def add_directory(
+        self, path: str, *, glob: Optional[str] = None
+    ) -> InternalIndexedDirResponse:
+        """Indexes a directory from disk using the standard Pagefind indexing behaviour.
+
+        This is equivalent to running the Pagefind binary with ``--site <dir>``.
+
+        :param path: the path to the directory to index. If the `path` provided is relative, \
+                it will be relative to the current working directory of your Python process.
+        :param glob: a glob pattern to filter files in the directory. If not provided, all \
+            files matching ``**.{html}`` are indexed. For more information on glob patterns, \
+            see the `Wax patterns documentation <https://github.com/olson-sean-k/wax#patterns>`.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+        result = await self._service.send(
+            InternalAddDirRequest(
+                type="AddDir",
+                index_id=self._index_id,
+                path=path,
+                glob=glob,
+            )
+        )
+        assert result["type"] == "IndexedDir"
+        return cast(InternalIndexedDirResponse, result)
+
+    async def get_files(self) -> List[InternalDecodedFile]:
+        """Get raw data of all files in the Pagefind index.
+
+        WATCH OUT: this method emits all files. This can be a lot of data, and
+        this amount of data can cause reading from the subprocess pipes to deadlock.
+
+        STRICTLY PREFER calling ``self.write_files()``.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+
+        response = await self._service.send(
+            InternalGetFilesRequest(type="GetFiles", index_id=self._index_id)
+        )
+        assert response["type"] == "GetFiles"
+        files = cast(InternalGetFilesResponse, response)["files"]
+
+        decoded_files = [
+            {"path": file["path"], "content": base64.b64decode(file["content"])}
+            for file in files
+        ]
+
+        return cast(List[InternalDecodedFile], decoded_files)
+
+    async def delete_index(self) -> None:
+        """
+        Deletes the data for the given index from its backing Pagefind service.
+        Doesn't affect any written files or data returned by ``get_files()``.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+        result = await self._service.send(
+            InternalDeleteIndexRequest(type="DeleteIndex", index_id=self._index_id)
+        )
+        assert result["type"] == "DeleteIndex"
+        self._index_id = None
+        self._service = None
+
+    async def add_custom_record(
+        self,
+        *,
+        url: str,
+        content: str,
+        language: str,
+        meta: Optional[Dict[str, str]] = None,
+        filters: Optional[Dict[str, List[str]]] = None,
+        sort: Optional[Dict[str, str]] = None,
+    ) -> InternalIndexedFileResponse:
+        """Add a direct record to the Pagefind index.
+
+        This method is useful for adding non-HTML content to the search results.
+
+        :param content: the raw content of this record.
+        :param url: the output URL of this record. Pagefind will not alter this.
+        :param language: ISO 639-1 code of the language this record is written in.
+        :param meta: the metadata to attach to this record. Supplying a ``title`` is highly recommended.
+        :param filters: the filters to attach to this record. Filters are used to group records together.
+        :param sort: the sort keys to attach to this record.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+        result = await self._service.send(
+            InternalAddRecordRequest(
+                type="AddRecord",
+                index_id=self._index_id,
+                url=url,
+                content=content,
+                language=language,
+                meta=meta,
+                filters=filters,
+                sort=sort,
+            )
+        )
+        assert result["type"] == "IndexedFile"
+        return cast(InternalIndexedFileResponse, result)
+
+    async def write_files(self, output_path: Optional[str] = None) -> None:
+        """Write the index files to disk.
+
+        If you're using PagefindIndex as a context manager, there's no need to call this method:
+        if no error occurred, closing the context automatically writes the index files to disk.
+
+        :param output_path: a path to override the configured output path for the index.
+        """
+        assert self._service is not None
+        assert self._index_id is not None
+        if not output_path:
+            if not self._config:
+                output_path = None
+            else:
+                output_path = self._config.get("output_path")
+
+        result = await self._service.send(
+            InternalWriteFilesRequest(
+                type="WriteFiles",
+                index_id=self._index_id,
+                output_path=output_path,
+            )
+        )
+        assert result["type"] == "WriteFiles"
+
+    async def __aenter__(self) -> "PagefindIndex":
+        assert self._service is None
+        assert self._index_id is None
+        return await self._start()
+
+    async def __aexit__(
+        self,
+        exc_type: Optional[Any],
+        exc_value: Optional[Any],
+        traceback: Optional[Any],
+    ) -> None:
+        if self._service is None:
+            return
+        if self._index_id is None:
+            return
+        if exc_type is None:
+            await self.write_files()
+        await self._service.close()

wrappers/python/src/pagefind/service/types.py

@@ -0,0 +1,171 @@
+from enum import Enum
+from typing import Dict, List, Literal, Optional, Sequence, TypedDict, Union
+
+
+class InternalRequestType(Enum):
+    NEW_INDEX = "NewIndex"
+    ADD_FILE = "AddFile"
+    ADD_RECORD = "AddRecord"
+    ADD_DIR = "AddDir"
+    WRITE_FILES = "WriteFiles"
+    GET_FILES = "GetFiles"
+    DELETE_INDEX = "DeleteIndex"
+
+
+class InternalPagefindServiceConfig(TypedDict, total=False):
+    # FIXME: document
+    root_selector: Optional[str]
+    exclude_selectors: Optional[Sequence[str]]
+    force_language: Optional[str]
+    verbose: Optional[bool]
+    logfile: Optional[str]
+    keep_index_url: Optional[bool]
+    write_playground: Optional[bool]
+    include_characters: Optional[str]
+    lowercase_filters: Optional[bool]
+
+
+class InternalNewIndexRequest(TypedDict):
+    type: Literal["NewIndex"]
+    config: Optional[InternalPagefindServiceConfig]
+
+
+class InternalAddFileRequest(TypedDict):
+    type: Literal["AddFile"]
+    index_id: int
+    """index_id must be positive."""
+    file_path: Optional[str]
+    url: Optional[str]
+    file_contents: str
+
+
+class InternalAddRecordRequest(TypedDict):
+    type: Literal["AddRecord"]
+    index_id: int
+    """index_id must be positive."""
+    url: str
+    content: str
+    language: str
+    meta: Optional[Dict[str, str]]
+    filters: Optional[Dict[str, List[str]]]
+    sort: Optional[Dict[str, str]]
+
+
+class InternalAddDirRequest(TypedDict, total=False):
+    type: Literal["AddDir"]
+    index_id: int
+    path: str  # TODO: support Path
+    glob: Optional[str]
+
+
+class InternalWriteFilesRequest(TypedDict, total=False):
+    type: Literal["WriteFiles"]
+    index_id: int
+    """index_id must be positive."""
+    output_path: Optional[str]
+
+
+class InternalGetFilesRequest(TypedDict):
+    type: Literal["GetFiles"]
+    index_id: int
+    """index_id must be positive."""
+
+
+class InternalDeleteIndexRequest(TypedDict):
+    type: Literal["DeleteIndex"]
+    index_id: int
+    """index_id must be positive."""
+
+
+InternalRequestPayload = Union[
+    InternalNewIndexRequest,
+    InternalAddFileRequest,
+    InternalAddRecordRequest,
+    InternalAddDirRequest,
+    InternalWriteFilesRequest,
+    InternalGetFilesRequest,
+    InternalDeleteIndexRequest,
+]
+
+
+class InternalServiceRequest(TypedDict):
+    message_id: Optional[int]
+    payload: InternalRequestPayload
+
+
+class InternalResponseType(Enum):
+    NEW_INDEX = "NewIndex"
+    INDEXED_FILE = "IndexedFile"
+    INDEXED_DIR = "IndexedDir"
+    WRITE_FILES = "WriteFiles"
+    GET_FILES = "GetFiles"
+    DELETE_INDEX = "DeleteIndex"
+    ERROR = "Error"
+
+
+class InternalResponseError(TypedDict):
+    type: Literal["Error"]
+    message: str
+    original_message: Optional[str]
+
+
+class InternalNewIndexResponse(TypedDict):
+    type: Literal["NewIndex"]
+    index_id: int
+
+
+class InternalIndexedFileResponse(TypedDict):
+    type: Literal["IndexedFile"]
+    page_word_count: int
+    page_url: str
+    page_meta: Dict[str, str]
+
+
+class InternalIndexedDirResponse(TypedDict):
+    type: str
+    page_count: int
+
+
+class InternalWriteFilesResponse(TypedDict):
+    type: Literal["IndexedFile"]
+    output_path: str
+
+
+class InternalSyntheticFile(TypedDict):
+    path: str
+    content: str
+
+
+class InternalDecodedFile(TypedDict):
+    path: str
+    content: bytes
+
+
+class InternalGetFilesResponse(TypedDict):
+    type: Literal["GetFiles"]
+    files: List[InternalSyntheticFile]
+
+
+class InternalDeleteIndexResponse(TypedDict):
+    type: Literal["DeleteIndex"]
+
+
+InternalResponsePayload = Union[
+    InternalNewIndexResponse,
+    InternalIndexedFileResponse,
+    InternalIndexedDirResponse,
+    InternalWriteFilesResponse,
+    InternalGetFilesResponse,
+    InternalDeleteIndexResponse,
+]
+
+
+class InternalServiceResponse(TypedDict):
+    message_id: Optional[int]
+    payload: Union[InternalResponsePayload, InternalResponseError]
+
+
+class InternalResponseCallback(TypedDict, total=False):
+    exception: Optional[Exception]
+    err: Optional[InternalResponseError]
+    result: Optional[InternalResponsePayload]