Custom Bangs

**LLM ahead** I used this project to test working with an LLM so most of the code here is generated by one.

Idea

Kagi is a good product, but in my opinion, the company behind it has a few issues that make it impossible for me to use. However, I have identified two main features that I find most useful and that I can replicate in my own project.

1. Customize the results

The idea behind this feature is, that you can customize the results to match your liking and needs. You don’t like a certain site? Block it. You think Wikipedia is the best site ever? Give it more weight and rank it up.

When I started with this project I tried to write my own meta search engine and aggregate results from various engines. While this would be possible it is a pain to do. One has to stay up to date with the search engines because most of them don’t provide access that you can afford if they even provide access at all. So in the end you’re left scraping their sites for results.

I had another look at SearXNG and found that it was good enough for my use case so I decided to combine it with this project. The next step might be to integrate this project directly into SearXNG as a plugin.

2. Bangs

Bangs are, as far as I know, a feature invented by DuckDuckGo. They allow you to quickly forward a query to a specific site by prefixing the query with a designated string. For example, !g foo will search for foo on Google.

This can be very powerful, especially on mobile devices, where it serves as a convenient shortcut. Having a site that supports this feature makes it particularly convenient, as you don’t have to configure each device to your preferences.

Custom default search engine

As not every user has the same preferences when it comes to search engines, the users can configure their default search engine. If the application finds a matching bang it will act on the bang. Else it will forward the query to the configured default search engine. In my case this will the an SearXNG instance.

Setup

Development Environment

To set up the development environment, you need:

• Git (required)
• Nix (required) - Used for reproducible development environments
• direnv (recommended) - Automatically loads the development environment when entering the project directory

Setting up the development environment:

1. Install Nix if you haven’t already: https://nixos.org/download.html
2. Install direnv (optional but recommended): https://direnv.net/docs/installation.html
3. Clone the repository and enter the directory:

git clone <repository-url>
cd meta-search

1. If using direnv, allow the environment:

direnv allow

1. If not using direnv, manually enter the Nix shell:

nix develop

Running tests:

Once in the development environment, you can run tests using pytest:

# Run all tests
pytest

# Run specific test file
pytest tests/test_meta_search.py

# Run tests in verbose mode
pytest -v

Coding standards

• All the Python code has to be formatted with black, docformatter, isort. The easiest way to do is, is via running dev fmt.
• For linting the following tools are used and configured in pyproject.toml:
  • mypy
  • pylint
  • ruff

Production Deployment

The flake provides a NixOS module. Add the flake as an input and import nixosModules.default:

{
  inputs.meta-search.url = "git+https://git.zweili.org/Nebucatnetzer/meta-search";

  outputs =
    { nixpkgs, meta-search, ... }:
    {
      nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
        modules = [
          meta-search.nixosModules.default
          {
            services.zweili-search = {
              enable = true;
              domain = "search.example.org";
              environmentFile = "/run/secrets/zweili-search.env";
            };
          }
        ];
      };
    };
}

The environment file must contain at least SECRET_KEY=.... Keep it outside the Nix store, for example with agenix or sops-nix.

Gunicorn listens on 127.0.0.1:8000. Put a reverse proxy with TLS in front of it.

Before the first start, zweili-search-setup.service runs the migrations and creates an admin user. The user name is adminUsername, by default admin. The password is password. Log in and change it right away. The setup service never changes an existing user.

You can create more users with the management wrapper. It runs as the service user with the service environment:

sudo zweili-search-manage createsuperuser

Module options

Option	Default	Description
services.zweili-search.enable	false	Enable the service.
services.zweili-search.domain	none, required	Public domain. Added to ALLOWED_HOSTS and CSRF origins.
services.zweili-search.environmentFile	none, required	File with SECRET_KEY. systemd reads it as root.
services.zweili-search.listenAddress	"127.0.0.1"	Address for Gunicorn.
services.zweili-search.port	8000	Port for Gunicorn.
services.zweili-search.workers	null	Number of Gunicorn workers. Null means CPUs plus one.
services.zweili-search.stateDir	"/var/lib/zweili_search"	Directory for the SQLite database.
services.zweili-search.adminUsername	"admin"	Name of the admin user that the setup service creates.
services.zweili-search.package	built with host nixpkgs	The package to run.

A stateDir outside /var/lib is created with tmpfiles. It must not be under /home, /root or /run/user.

Configuration without the module

The application reads these environment variables:

SECRET_KEY

Required. The Django secret key.

ZWEILI_SEARCH_DOMAIN

The domain under which your search site is accessible. E.g. foo.example.org. The default is the FQDN of the host.

ZWEILI_SEARCH_DB_DIR

The directory for the SQLite DB. The default is /var/lib/zweili_search.

ZWEILI_SEARCH_BIND

The address and port for Gunicorn. The default is 0.0.0.0:8000.

ZWEILI_SEARCH_WORKERS

The number of Gunicorn workers. The default is the number of CPUs plus one.

DEBUG

Enables Django debug mode. The default is false. Never enable it in production.

Usage

You can enter a search query right away and it will start searching on all the configured search engines. At the moment it is only DuckDuckGo.

When you go to https://your.domain.com/admin you can configure domains you want to block or setup custom bangs. They are always tied to a specific user so make sure you select the correct one otherwise they will not work or another will get your settings by accident. For the moment user specific settings aren’t planed from my side as I will only allow trusted users on my instance.