Wagtailmeili: integrating a blazing fast search engine with Wagtail
by Maxime Decooman
Meilisearch in a nutshell
Meilisearch is an open-source, blazing-fast search engine built for creating intuitive and delightful search experiences. It's designed with a strong emphasis on developer experience and ease of use.
Key Characteristics:
- Speed: Thanks to its optimized core written in Rust.
- Ease of Use: It offers sensible defaults and a clean, well-documented API. You can get started with a minimal configuration.
- Relevance: It uses customizable ranking rules to fine-tune relevance as needed.
- Fuzzy search: Built-in typo tolerance ensure users find what they're looking for even with minor spelling mistakes.
- Focused on Search: It excels at this specific task and doesn't try to be a general-purpose data store or analytics engine.
- Open Source & Cloud Option: It's fully open source (MIT licensed) and also offers a cloud service for managed hosting.
About other search engines
I will spare you another article adding to the noise. Read this article from meilisearch or read this extract that summarize well their place in the ecosystem.
Unlike Elasticsearch, which is a general search engine designed for large amounts of log data (for example, back-facing search), Meilisearch is intended to deliver performant instant-search experiences aimed at end-users (for example, front-facing search).
Meilisearch is a perfect choice if you need a simple and easy tool to deploy a typo-tolerant search bar. It provides prefix searching capability, makes search intuitive for users, and returns results instantly with excellent relevance out of the box.
Perfect match for Wagtail CMS
Elasticsearch is often overkill for many projects, demanding more infrastructure than you really need. Wagtail natively supports search and integrates with OpenSearch and Elasticsearch, but having another option like Meilisearch gives us a great middle ground.
Meilisearch is light enough to start on the same server and can easily scale up to its own dedicated infrastructure later on.
Meet Wagtailmeili
About two years ago, I integrated Meilisearch into a project and have now decided to refactor that integration into a public package, wagtailmeili, to enhance it and add more features.
Before starting, I did a quick search and found a package called wagtail-meilisearch, which has the same goal. I tried to understand its implementation while also learning the underlying logic in Wagtail. Perhaps it was a skill issue on my part, but I felt stuck. It didn't meet all my needs, and I was under a tight deadline.
I chose to build from scratch using a lean approach. However, the existing package was still very helpful in understanding the integration logic. Ultimately, my decision to build my own solution proved correct, as I delivered the project on time.
Wagtail integration features
The integration fundamentally uses Meilisearch's RESTful API through the Python client given by meilisearch.
In settings
- Improve the relevance of the search results by
- Customizing the RANKING_RULES
- Customizing the STOP_WORDS
- Do not index certain models (SKIP_MODELS)
- A convenient way to skip indexing instances based on attributes (SKIP_MODELS_BY_FIELD_VALUE)
By default Wagtail Pages that are not published are not indexed
WAGTAILSEARCH_BACKENDS = {
"default": {
"BACKEND": "wagtailmeili.backend",
"HOST": config("MEILISEARCH_HOST", "http://127.0.0.1"),
"PORT": config("MEILISEARCH_PORT", 7700, cast=int),
"MASTER_KEY": config("MEILISEARCH_MASTER_KEY"),
# "STOP_WORDS": ... (optional, only to override the defaults)
# "RANKING_RULES: ... (optional, only to override the defaults)
"SKIP_MODELS": ["app_label.Model1", "app_label.Model2",],
"SKIP_MODELS_BY_FIELD_VALUE": {
"wagtailmeili_testapp.MoviePage": {
"field": "status",
"value": "archived",
},
},
# ...
}
}
In your models
- Within a specific model, you can:
- Define sortable attributes
- Define specific ranking rules
class MySuperPage(Page):
"""A Super Page to do incredible things."""
sortable_attributes = [
"published_last_timestamp",
# ...
]
ranking_rules = [
"published_last_timestamp:desc",
# ...
]
Template Tag filter
Not yet documented as I am still porting some functionalities from the initial project.
{% load meilisearch %}
{% get_matches_position result %}
Searching
For convenience I have added a manager for either a Wagtail Page or a Django Model. You can either define the search with a standard backend call:
# for example
search_backend = get_search_backend("default")
# then do your search
search_backend.search(...)
Or from the Model using one of the managers
from wagtailmeili.manager import MeiliSearchPageManager
from wagtailmeili.manager import MeiliSearchModelManager
# in your page model class MyPage, for example, add this
objects = MeiliSearchPageManager()
# will use the meilisearch backend
MyPage.objects.search(...)
Search parameters
# you can configure your search parameters, the "opt_params" argument
# Let say you have default opt_params as follow
opt_params = {
"page": page_number,
"hitsPerPage": 10,
"facets": ["topics.id", "commodities.id"]
}
# you can add later some filter like this
opt_params["filter"] = search_filters
# and use them in your search
results = MyPage.objects.search(query, opt_params=opt_params).get()
Index Updates
No big magic here as it uses the standard mechanism from Wagtail (Django signals). When you create an instance of an indexed model, it will be added automatically and updated when you do a change. If you need to update the index from the command lien, you just use the standard command:
You can create or update your indexes with the wagtail command line:
python manage.py update_index
The update strategy
The strategy implemented is a swap strategy. The new index is built with a suffix _new added to the name. Then it is used to swap the old index. The old index is then deleted. For most cases, I do not think the performances will be an issue. Better to spend time on the features first.
New command line
If needed in development you can delete all the indexes, I added a new command:
python manage.py meilisearch_delete_all_indexes
Next Steps
Wagtailmeili enables enough core Meilisearch features to get you up and running. However it is only the beginning. A proper demo project with a basic frontend (HTMX and AlpineJS) will be implemented to showcase the possibilities and allow tinkering with it.
Before version 1.0 I recommend to fix the version in your requirements as things may break (See the changelog before updating).
Roadmap before 1.0.0 (unsorted)
- ✅ Adding tests (done)
- Exploring Meilisearch and bringing more of its features for Wagtail. ( I will not have the word "AI" in this post... 👀)
- Getting a leaner implementation (and give some ideas for Wagtail)
- Giving more love to the Sample project with a frontend
- Better documentation based on the sample project
- A recipe for internationalization
Like it? Give it a star and follow the project on