> For the complete documentation index, see [llms.txt](https://docs.dropzone.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dropzone.ai/integrations/data/elasticsearch_data.md).

# Elasticsearch

{% hint style="info" %}
Elasticsearch is an SIEM integration. SIEM integrations are used to perform analysis of any SIEM generated alerts, and/or to use generated data as part of investigation analysis.
{% endhint %}

Dropzone AI integrates with the [Elasticsearch](https://www.elastic.co/elasticsearch) security SIEM. Many customers ingest other alert sources into Elasticsearch (e.g. IDPs) and integrate Dropzone into Elasticsearch rather than the source systems.

## Integrations Overview

To enable these integrations you will perform the following actions:

* Create an API key with the necessary permissions
* Obtain your Elasticsearch Cloud ID
* Enable the Data source within your Dropzone tenant

{% hint style="info" %}
If you are using the Elasticsearch Serverless Projects-Based Model or an On-premise Elasticsearch using the Dropzone connector, you will not need to provide a Cloud ID.
{% endhint %}

## Create an API Key

Elasticsearch requires an API Key to enable.

To obtain a least-privilege API Key, do the following:

* Navigate to your [Elastic Cloud home page](https://cloud.elastic.co/home) or deployment
* Under the Hosted Deployments section, locate the deployment you want Dropzone to be able to access
* Click "Open"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FPWWFnKhHDScYRqFn8WPM%2Felasticsearch-integration-1.png?alt=media" alt=""><figcaption><p>Click Manage</p></figcaption></figure>

* In the Deployment overview page, click "Management" in the bottom left corner
* Click the icon next to Stack Management
* Navigate to API keys

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FOPPCbHBdt73ScRdMnp47%2Felasticsearch-integration-3.png?alt=media" alt=""><figcaption><p>Navigate to API keys</p></figcaption></figure>

* Click "Create an API key"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FwdXd4z9O76WchpakQxL0%2Felasticsearch-integration-4.png?alt=media" alt=""><figcaption><p>Click "Create an API key"</p></figcaption></figure>

* Name the API key something memorable, such as "Dropzone AI"
* Under type, select User API key

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2F5ZEAxWjbNgASnDqDiHhV%2Felasticsearch-integration-5.png?alt=media" alt=""><figcaption><p>Create an API key</p></figcaption></figure>

* Check the box labeled "Control security privileges"

Dropzone requires the following security privileges across its Alert and Data Source integrations.

| Privilege             | Level   | Purpose                                           | Used In                   |
| --------------------- | ------- | ------------------------------------------------- | ------------------------- |
| `monitor`             | Cluster | Discovering index templates                       | Alert Source, Data Source |
| `read`                | Index   | Searching documents during investigations         | Alert Source, Data Source |
| `monitor`             | Index   | Listing indexes during scan (`_cat/indices`)      | Data Source               |
| `view_index_metadata` | Index   | Listing data streams during scan (`_data_stream`) | Data Source               |

* Input the following JSON into the privileges editor, replacing `"*"` in `names` with the index patterns Dropzone should use (e.g. `logs-*` or `winlogbeat-*`):

```json
{
  "read-only-role": {
    "cluster": ["monitor"],
    "indices": [
      {
        "names": ["*"],
        "privileges": ["read", "monitor", "view_index_metadata"]
      }
    ]
  }
}
```

{% hint style="info" %}
Cluster and index privileges are separate; cluster `monitor` does not grant `monitor` on index. The `*` pattern does not match hidden indexes (names that start with `.`).
{% endhint %}

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FtjrG2BL2rAeS5pCSdVds%2Felasticsearch-integration-8.png?alt=media" alt=""><figcaption><p>Enable Control security privileges and paste the required role</p></figcaption></figure>

* Click "Create API Key"
* Copy the API key generated for use later in the Dropzone UI, where it is called "API Key"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FYAqBV1gkCWHQtCnxuUgB%2Felasticsearch-integration-6.png?alt=media" alt=""><figcaption><p>Copy the key</p></figcaption></figure>

## Obtain your Elasticsearch Cloud ID

To obtain your Cloud ID, do the following:

* Navigate to your [Elastic Cloud home page](https://cloud.elastic.co/home)
* Under the Hosted Deployments section, locate the deployment you want Dropzone to be able to access
* Click "Open"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FPWWFnKhHDScYRqFn8WPM%2Felasticsearch-integration-1.png?alt=media" alt=""><figcaption><p>Click Open</p></figcaption></figure>

* In the upper right of the Overview page, click "Endpoint & API Keys"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FmPCVKX2A7F6vdBpm4ahA%2Felasticsearch-integration-2.png?alt=media" alt=""><figcaption><p>Click Endpoint &#x26; API Keys</p></figcaption></figure>

* Check "Show Cloud ID"
* Copy the value shown for use later in the Dropzone UI, where it is called "Elasticsearch Cloud ID"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FBXn9ueltXdy0gG3VrIfn%2Felasticsearch-integration-7.png?alt=media" alt=""><figcaption><p>Copy the Elasticsearch Cloud ID</p></figcaption></figure>

## Enable Elasticsearch

To enable the Data Source integration, you will need the following information:

| Dropzone Field         | Source                                                                                              |
| ---------------------- | --------------------------------------------------------------------------------------------------- |
| Elasticsearch Cloud ID | The cloud ID value copied earlier. Only necessary if you have an Elastic Cloud Hosted deployment    |
| Elasticsearch Server   | The server for your Elasticsearch project, e.g. <https://my-project.es.us-west-2.aws.elastic.cloud> |
| API Token              | The API token value generated earlier                                                               |

To enable the Data Source integration, do the following:

* Navigate to your Dropzone AI tenant home page e.g. https\://*mycompany*.dropzone.app
* In the left side bar, navigate to Platform Settings > Integrations

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FQKQGU2YK0KqsewZWvh0i%2Fui-integrations-dropdown.png?alt=media" alt=""><figcaption><p>Integrations Dropdown</p></figcaption></figure>

* Click "Library"

{% hint style="info" %}
If you have previously integrated this application, click "Configured"
{% endhint %}

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2Fx4O3FuRG5IqPwx5sFuHr%2Fapp_system_integrations_library.png?alt=media" alt=""><figcaption><p>Click Library</p></figcaption></figure>

* In the Search bar, search Elasticsearch, then click "Configure"

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FY9LSnYTcczRW2VLrA2No%2Fapp_system_integrations_available_elasticsearch.png?alt=media" alt=""><figcaption><p>The Elasticsearch Tile</p></figcaption></figure>

* Under the Data Source heading, if your Elasticsearch integration is behind an [On-premise Dropzone Connector](https://docs.dropzone.ai/platform/settings/connector), select your connector from the dropdown
* If you have a Cloud deployment, check the box labeled "Connect with Elastic Cloud ID," then input the Elasticsearch Cloud ID and API Key

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FSpclMLJazVKWDPC7JJwA%2Fapp_system_integrations_available_elasticsearch_data_config_1.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 1)</p></figcaption></figure>

* Otherwise, input the Elasticsearch Server, Port, and API Key

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2F3YmlY1JVQ8XWqJduZRWq%2Fapp_system_integrations_available_elasticsearch_config_2.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 2)</p></figcaption></figure>

* If you want Dropzone to only use [remote clusters](https://www.elastic.co/docs/deploy-manage/remote-clusters) when making queries, check the box labeled "Search Remote Indices"
* In the "Enrichment Query" section, you can help guide Dropzone's AI analyst search queries when looking for information about entities. To do so, click "Add Item." Select the type of entity you want to enrich, input the [index](https://www.elastic.co/docs/manage-data/data-store/index-basics) or [data stream](https://www.elastic.co/docs/manage-data/data-store/data-streams) the AI should search in, and input a guiding query. The query must be a JSON search or a Lucene query string

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FpHKjIB2SMcrSPwgcXUfF%2Fapp_system_integrations_available_elasticsearch_data_config_3.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 3)</p></figcaption></figure>

You can manage Dropzone's access to specific Elasticsearch [indexes](https://www.elastic.co/docs/manage-data/data-store/index-basics) directly in the Dropzone UI. By default, it can search the `logs-*` index.

* To only query specific indexes, under Index Pattern Allow List, add the Elasticsearch index patterns Dropzone may search, such as `logs-*` or `winlogbeat-*`
* To exclude specific indexes, under Index Pattern Deny List, add the Elasticsearch index patterns Dropzone may *not* search
  * These lists are a Dropzone setting, not an Elasticsearch privilege grant
  * In Elasticsearch, `.*` matches index names that start with `.`. It does not mean all indexes

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2F05pcgxi6k420ztuVA1aH%2Fapp_system_integrations_available_elasticsearch_data_config_4.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 4)</p></figcaption></figure>

* To group similar indexes together (e.g. if your indexes has been subdivided into [shards](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards), check the box labeled "Enable Smart Index Grouping"
* Dropzone pulls information from recent documents to understand what data corresponds to your indexes. Enter your Sample Lookback Window to decide how many days back it can search

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FnvoxA1rQGdcYfMHoshkl%2Fapp_system_integrations_available_elasticsearch_data_config_5.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 5)</p></figcaption></figure>

* To help guide the AI's information gathering process, enable the [Query Guidance](https://docs.dropzone.ai/integrations/data) feature. Click "Add Item" and input your Condition and Instruction pairs

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FcYfD24VJsFX1ZjoP5UDb%2Fintegration-query-guidance.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 6)</p></figcaption></figure>

* Input the maximum number of concurrent queries you want Dropzone to run against your Elasticsearch deployment. If your cluster is small or shared, consider lowering it to help limit resource competition
* Click "Test & Save" to finish

<figure><img src="https://3968177484-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDt9gEGwbp9ApiQAJP3hQ%2Fuploads%2FIPRtbPGYasgKP07NAvvc%2Fapp_system_integrations_available_elasticsearch_data_config_6.png?alt=media" alt=""><figcaption><p>The Elasticsearch Data Configuration (pt 7)</p></figcaption></figure>

{% hint style="warning" %}
Test & Save only checks that Dropzone can reach the cluster. It does not verify index listing privileges. After you save, run a scan so Dropzone can catalog your indexes. If the catalog is empty, investigation queries cannot run. Check the Index Pattern Allow List, index privileges, and scan status.
{% endhint %}

If you have any errors contact our Customer Success team.
