> For the complete documentation index, see [llms.txt](https://docs.dropzone.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dropzone.ai/edge/readme/troubleshooting.md).

# Troubleshooting

{% hint style="warning" %}
GitBook, our documentation hosting solution, is currently experiencing an outage. We are tracking this via <https://status.gitbook.com> and with our support reps.

Until it is resolved this documentation may be out of date.
{% endhint %}

## Viewing Container Logs

To see and "tail" logs for all Dropzone service:

```bash
docker compose -p dz logs -f
```

To see and "tail" logs for a specific Dropzone service:

```bash
docker compose -p dz logs -f <service-name>
```

For example, to view logs for the web service:

```bash
docker compose -p dz logs -f web
```

## A Container Is Unhealthy or Restarting

1. Check which containers are not healthy

```bash
docker compose -p dz ps
```

1. View the logs for the unhealthy service

```bash
docker compose -p dz logs --tail 100 <service-name>
```

Common causes include:

* Incorrect values in `config.yml`
* Database passwords that were not changed from defaults
* TLS certificate/key mismatch
* Insufficient system resources (CPU, memory, or disk)

## Response Actions Failing (AppArmor / SELinux)

The terrarium container requires `SETUID`, `SETGID`, and `AUDIT_WRITE` kernel capabilities. AppArmor or SELinux may block these operations, causing Response Actions to fail.

{% hint style="info" %}
This container runs customer-supplied python code. The container switches to a separate user prior to running the customer-supplied code to provide container integrity, which is why these permissions are needed.
{% endhint %}

### AppArmor (Ubuntu 24.04 and later)

Ubuntu 24.04 includes AppArmor policies that restrict unprivileged user namespace creation, which block the terrarium container. For background, see [Ubuntu 24.04 and Bubblewrap](https://etbe.coker.com.au/2024/04/24/ubuntu-24-04-bubblewrap/). To resolve this, create an AppArmor profile that allows `bwrap` to use user namespaces:

```bash
sudo tee /etc/apparmor.d/bwrap <<'EOF'
abi <abi/4.0>,
include <tunables/global>

profile bwrap /usr/bin/bwrap flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/bwrap>
}
EOF
```

Then reload AppArmor:

```bash
sudo systemctl reload apparmor
```

### SELinux (RHEL, CentOS, Fedora)

If SELinux is blocking container operations, check the audit log for denials:

```bash
sudo ausearch -m avc -ts recent
```

If denials are present for the terrarium container, set the container domain to permissive:

```bash
sudo semanage permissive -a container_t
```

{% hint style="warning" %}
Adjusting AppArmor or SELinux policies affects system-wide security. Consult your security team before making changes in production environments.
{% endhint %}

## Cannot Reach the Dropzone UI

* Verify the `nginx` container is running and healthy: `docker compose -p dz ps nginx`
* Verify DNS resolves your hostname to the instance's IP: `dig <your-hostname>`
* Verify port 443 is open in your security group or firewall
* Verify the TLS certificate in `config.yml` matches the hostname you are accessing
* Check nginx logs: `docker compose -p dz logs nginx`

## OIDC Authentication Fails

* Verify the `client-id` and `client-secret` in `config.yml` match your IdP application
* Verify all OIDC endpoint URLs in the `oidc.manual` section are correct and reachable from the instance
* Verify your IdP is configured to allow redirect URIs to `https://<your-hostname>`
* Check web service logs for detailed error messages: `docker compose -p dz logs web`

## AI Provider Errors

* Check `ai-core` logs for provider-specific error messages:

```bash
docker compose -p dz logs ai-core
```

* Verify the providers enabled in `ai_provider_config.yaml` have matching API keys set in the `apis` section of `config.yml`
* Verify the instance has outbound network access to the AI provider endpoints (see [Network Requirements](https://docs.dropzone.ai/edge/readme/prerequisites.md#network-requirements))
* Test connectivity from the instance to your configured providers:

```bash
# OpenAI
curl -s -o /dev/null -w "%{http_code}" https://api.openai.com

# Anthropic
curl -s -o /dev/null -w "%{http_code}" https://api.anthropic.com

# Google (Gemini)
curl -s -o /dev/null -w "%{http_code}" https://aiplatform.googleapis.com

# Azure OpenAI (replace with your resource name)
curl -s -o /dev/null -w "%{http_code}" https://<your-resource>.openai.azure.com
```

* Validate your configured AI models:

  ```bash
  /opt/dropzone/bin/ai-model-tool validate-config
  /opt/dropzone/bin/ai-model-tool validate-config --check-connectivity
  ```

  The first command checks model name spelling. Adding `--check-connectivity` also probes each enabled model with a live API call, reporting the specific endpoint, deployment name, and API key variable that failed.
* To see all valid model names for a provider and generate a ready-to-paste config snippet, run:

  ```bash
  /opt/dropzone/bin/ai-model-tool generate-config --provider openai
  ```

{% hint style="info" %}
For AWS Bedrock connectivity, use the AWS CLI: `aws bedrock-runtime invoke-model ...`. Bedrock does not respond to unauthenticated HTTPS probes.
{% endhint %}

For a full workflow for fixing and revalidating provider configuration, see [Managing AI Providers](https://docs.dropzone.ai/edge/readme/ai-provider-management.md).

## ai-core Fails to Start

`ai-core` intentionally exits at startup if the configuration is invalid or any enabled model is unreachable. Check the logs first:

```bash
docker compose -p dz logs ai-core
```

Common causes:

* **Invalid model key** — a model name under `models:` does not match the built-in catalog. Run `ai-model-tool validate-config` to identify and fix the offending names before restarting
* **Authentication failure (401/403)** — credentials are missing, incorrect, or do not have permission to access the model. Verify the API key in `config.yml` and your provider account permissions
* **Model or deployment not found (404)** — a `deployment_name` in your config does not match any deployed model in your provider account. Check your provider console (e.g. Azure OpenAI Studio) to verify your deployment names and compare them against the `deployment_name` values in `ai_provider_config.yaml`. Check `ai-core` logs to see which deployment(s) failed

If you need to get `ai-core` running while you debug the config, see [Managing AI Providers — Recovery](https://docs.dropzone.ai/edge/readme/ai-provider-management.md#recovery-if-ai-core-wont-start).

## Restarting After Configuration Changes

After editing any configuration files in `/opt/dropzone/deploy/`, restart the deployment:

```bash
/opt/dropzone/bin/deploy-dropzone down
/opt/dropzone/bin/deploy-dropzone up
```

## Getting Help

If the above steps do not resolve your issue, engage our Customer Success team. When reaching out, include:

{% hint style="warning" %}
Review all logs and configuration files for sensitive data (e.g. credentials, API keys, internal hostnames) before sharing. You are responsible for redacting any sensitive information from these artifacts.
{% endhint %}

* The output of `docker compose -p dz ps`
* Relevant container logs (`docker compose -p dz logs --tail 200 <service-name>`)
* Any configuration changes made since the last successful deployment
