Skip to main content

Custom files for Aiven for OpenSearch®

Upload, list, replace, and download custom files for your Aiven for OpenSearch® service to refine text analysis and improve search relevance.

Custom files are user-defined files that control how Aiven for OpenSearch analyzes text. By adding domain-specific vocabulary and rules, these files refine search results to be more accurate and relevant.

Aiven for OpenSearch supports the following custom file types:

  • Stopwords: Exclude common words such as "the" and "is" to refine search results.
  • Synonyms: Equate similar terms, such as "car" and "automobile," to improve query matching.
  • WordNet: Provide semantic relationships between words, such as synonyms and antonyms.
  • Hyphenation patterns: Define XML hyphenation patterns for the hyphenation_decompounder token filter, which splits compound words for languages such as German, Dutch, and the Scandinavian languages.
note

Stopwords, synonyms, and WordNet files must be in plain text (UTF-8 encoded) format. Hyphenation patterns files must be well-formed XML (UTF-8 encoded).

Reference locations​

Aiven for OpenSearch stores every uploaded file at a fixed reference location in the following format:

custom/FILE_TYPE/FILE_NAME

FILE_TYPE is stopwords, synonyms, wordnet, or hyphenation_patterns, and FILE_NAME is the name you gave the file at upload, without a file extension. For example, a stopwords file named demo_stopwords is stored at custom/stopwords/demo_stopwords.

Use the reference location when you point an analyzer, token filter, or index setting at the file. The reference location does not change when you replace the file.

Upload files​

Upload new custom files to your OpenSearch service.

  1. Log in to the Aiven Console, select your project, and select your Aiven for OpenSearch service.
  2. Click Data > Indexes.
  3. In the Custom files section, click Upload file.
  4. In the Upload a custom file window:
    • Select a File type: Synonyms, Stopwords, WordNet, or Hyphenation patterns.
    • Enter a Filename. Use alphanumeric characters and underscores only.
    • Choose the file from your system and click Upload.

Aiven for OpenSearch validates the file against the selected file type and rejects uploads that do not match, for example an XML file uploaded as Stopwords.

List files​

List all custom files associated with your OpenSearch service.

In the Aiven Console, the Custom files section lists all uploaded files with their Reference location, File type, Size, and Latest upload time.

Replace files​

Once you upload a custom file, you can only replace it, not delete it. To update an existing file, replace it with a new file that contains the updated content. Replacing a file overwrites the original permanently. You cannot change the file type or the filename, and the reference location stays the same.

  1. Log in to the Aiven Console, select your project, and select your Aiven for OpenSearch service.
  2. Click Data > Indexes.
  3. In the Custom files section, locate the file to replace.
  4. Click Actions > Replace file.
  5. Choose the new file from your system and click Upload.

Download files​

Download a custom file to your local system.

  1. Log in to the Aiven Console, select your project, and select your Aiven for OpenSearch service.
  2. Click Data > Indexes.
  3. In the Custom files section, locate the file to download.
  4. Click Actions > Download file.
  5. Choose your location and click Save.

Limitations​

  • Files cannot be deleted. They can only be replaced.
  • The reference location is fixed and cannot be customized.
  • If you move to a different cloud or project, files are copied or moved accordingly.
  • For OpenSearch Cross-Cluster Replication (CCR), files must be uploaded to both services manually.
  • Use alphanumeric characters and underscores only for file names.
  • Hyphenation patterns files must be well-formed XML, or the upload is rejected.

Example: Use a stopwords file with an index​

After uploading a custom file, you can use it in your index settings by specifying custom filters or analyzers. This example shows how to create an index that uses a custom stopwords file.

Create a stopwords file​

Create a file named demo_stopwords.txt with your stopwords.

a
fox
jumps
the

Upload the stopwords file​

Upload this file as the Stopwords file type with the filename demo_stopwords. Aiven for OpenSearch stores it at custom/stopwords/demo_stopwords.

Create an index that uses the stopwords file​

Create an index using the stopwords file via the OpenSearch Dashboards or the API.

  1. Log in to the Aiven Console, select your project, and select your Aiven for OpenSearch service.

  2. Access the OpenSearch Dashboards tab in the Connection information section.

  3. Use the Service URI to access OpenSearch Dashboards in a browser.

  4. Log in with the provided User and Password.

  5. Click Index Management > Indices > Create Index.

  6. Enter the details for the index.

  7. Expand the Advanced settings section and insert the following JSON configuration to use the stopwords file:

    {
      "index.analysis.analyzer.default.filter": [
        "custom_stop_words_filter"
      ],
      "index.analysis.analyzer.default.tokenizer": "whitespace",
      "index.analysis.filter.custom_stop_words_filter.ignore_case": "true",
      "index.analysis.filter.custom_stop_words_filter.stopwords_path": "custom/stopwords/demo_stopwords",
      "index.analysis.filter.custom_stop_words_filter.type": "stop",
      "index.number_of_replicas": "1",
      "index.number_of_shards": "1"
    }

  8. Click Create.

Verify the stopwords filter​

Verify the stopwords filter by using the _analyze API.

  1. Go to Dev Tools in OpenSearch Dashboards.
  2. Use the _analyze API to verify that the stopwords filter is working.
POST demo-index/_analyze
{
  "text": "a quick brown fox jumps over the lazy dog"
}

Example: Use a hyphenation patterns file with an index​

Hyphenation patterns files are referenced the same way, through the hyphenation_patterns_path parameter of the hyphenation_decompounder token filter. After you upload an XML patterns file as the Hyphenation patterns file type with the filename de_hyphenation, create an index that uses it:

curl -X PUT -H "Content-Type: application/json" -d'{
  "settings": {
    "analysis": {
      "filter": {
        "custom_decompounder": {
          "type": "hyphenation_decompounder",
          "hyphenation_patterns_path": "custom/hyphenation_patterns/de_hyphenation",
          "word_list": ["kraft", "fahrzeug"]
        }
      },
      "analyzer": {
        "custom_decompound_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "custom_decompounder"]
        }
      }
    }
  }
}' ${SERVICE_URL}/demo-compound-index?pretty

The hyphenation_decompounder token filter also requires a word list, supplied either inline with word_list, as in the preceding example, or as a path to a file with word_list_path.

Related pages