Observability

This guide covers the installation of logging and monitoring components that provide operational visibility into your AI/Run CodeMie deployment.

Overview

The observability stack consists of three components:

• Fluent Bit - Lightweight log collection and forwarding agent that captures logs from all pods
• Kibana - Log visualization and analytics dashboard for Elasticsearch
• Kibana Dashboards - Pre-configured monitoring dashboards for CodeMie metrics and logs

Fluent Bit Installation

Fluent Bit collects and forwards data to Elasticsearch for monitoring and analysis. The default configuration collects two types of data:

1. User Metrics (Mandatory)

Collects structured usage metrics from CodeMie application pods, including:

• User activity and engagement patterns
• AI model usage and token consumption
• API request metrics and performance data
• Cost tracking and resource utilization

These metrics are required for Kibana dashboards and usage analytics. By default, they are stored in the codemie_metrics_logs index. When quarterly metrics index rotation is enabled, configure the metrics output to use the codemie_metrics_logs_write alias instead. See the Metrics Index Rotation guide for the migration and configuration steps.

2. Infrastructure Logs (Optional)

Collects application logs from the following namespaces:

• codemie - Core application logs
• oauth2-proxy - Authentication proxy logs
• security - Keycloak and security components
• elastic - Elasticsearch and Kibana logs
• ingress-nginx - Ingress controller logs
• litellm - LiteLLM proxy server logs (if deployed)

These logs are stored in the logs-codemie-infra index for troubleshooting and operational insights.

Using External Logging System?

If you have an existing logging solution (CloudWatch, Stackdriver, Splunk, etc.), you can disable infrastructure log collection by removing the first [INPUT] section (tagged kube.codemie-infra.*) and its corresponding [OUTPUT] section from fluent-bit/values.yaml. User metrics collection must remain enabled for Kibana dashboards to function.

Step 1: Create Fluent Bit Namespace

Create a dedicated namespace for Fluent Bit:

kubectl create namespace fluentbit

Step 2: Copy Elasticsearch Credentials

Fluent Bit needs Elasticsearch credentials to forward logs. Copy the credentials:

kubectl get secret elasticsearch-master-credentials -n elastic -o yaml | \
  sed '/namespace:/d' | \
  kubectl apply -n fluentbit -f -

Step 3: Install Fluent Bit Helm Chart

Deploy Fluent Bit:

helm upgrade --install fluent-bit fluent-bit/. \
  -n fluentbit \
  --values fluent-bit/values.yaml \
  --wait \
  --timeout 900s \
  --dependency-update

Step 4: Verify Fluent Bit Deployment

Check that Fluent Bit is running:

# Check DaemonSet
kubectl get daemonset -n fluentbit

# Check pods (should be one per node)
kubectl get pods -n fluentbit

# Check logs
kubectl logs -n fluentbit daemonset/fluent-bit --tail=50

The steps below are identical across cloud providers except for the Helm values file and the Kibana URL. Substitute values-<cloud-provider>.yaml and <kibana-url> with the values for your cloud:

Cloud Provider	Values File	Kibana URL
AWS	values-aws.yaml	https://kibana.<your-domain>
Azure	values-azure.yaml	https://<your-domain>/kibana
GCP	values-gcp.yaml	https://kibana.<your-domain>

AWS and GCP expose Kibana on a dedicated subdomain; Azure exposes it as a path under your main domain.

Kibana Installation

Kibana provides a web interface for searching, analyzing, and visualizing logs stored in Elasticsearch.

Step 1: Create Kibana Encryption Keys Secret

Generate encryption keys for Kibana's saved objects, reports, and security features:

kubectl create secret generic kibana-encryption-keys \
  --namespace=elastic \
  --from-literal=encryptedSavedObjects.encryptionKey="$(openssl rand -hex 16)" \
  --from-literal=reporting.encryptionKey="$(openssl rand -hex 16)" \
  --from-literal=security.encryptionKey="$(openssl rand -hex 16)" \
  --type=Opaque

Secret Structure:

apiVersion: v1
kind: Secret
metadata:
  name: kibana-encryption-keys
  namespace: elastic
type: Opaque
data:
  encryptedSavedObjects.encryptionKey: <base64-encoded-32-char-key>
  reporting.encryptionKey: <base64-encoded-32-char-key>
  security.encryptionKey: <base64-encoded-32-char-key>

Step 2: Configure Domain Name

Fill in values in kibana/values-<cloud-provider>.yaml file by replacing %%DOMAIN%% with your domain name, e.g., airun.example.com

Domain Configuration

If you followed the Getting Started steps, this should already be configured.

Step 3: Install Kibana Helm Chart

Deploy Kibana:

helm upgrade --install kibana kibana/. \
-n elastic \
--values kibana/values-<cloud-provider>.yaml \
--wait \
--timeout 900s \
--dependency-update

Step 4: Verify Kibana Deployment

Check that Kibana is running:

# Check pod status
kubectl get pods -n elastic | grep kibana

# Check deployment
kubectl get deployment -n elastic kibana

# Check service
kubectl get service -n elastic kibana

# Check logs
kubectl logs -n elastic deployment/kibana --tail=50

Step 5: Access Kibana

Kibana can be accessed at:

<kibana-url>

Login Credentials:

• Username: elastic
• Password: Retrieved from Elasticsearch secret (see Data Layer)

Step 6: Review Logs (Optional)

If infrastructure logging is enabled, you can access historical infrastructure logs in Discover section.

Metrics Index

The codemie_metrics_logs data view for user metrics is automatically configured during Kibana Dashboards installation.

Kibana Dashboards Installation

Install pre-configured dashboards that provide operational insights into CodeMie performance, usage, and health using the manage-kibana-dashboards.sh script.

Option 1: Kubernetes Secret Authentication (Recommended)

This method automatically retrieves Elasticsearch credentials from the Kubernetes secret:

bash ./kibana-dashboards/manage-kibana-dashboards.sh \
--url "<kibana-url>" \
--k8s-auth \
--non-interactive

Parameters:

• --url - Your Kibana URL (replace <your-domain> with your actual domain)
• --k8s-auth - Use Kubernetes secret for authentication
• --non-interactive - Run without prompts (uses elasticsearch-master-credentials from elastic namespace)

Example:

bash ./kibana-dashboards/manage-kibana-dashboards.sh \
--url "<kibana-url>" \
--k8s-auth \
--non-interactive

Option 2: Manual Authentication

This method prompts for Elasticsearch credentials interactively:

bash ./kibana-dashboards/manage-kibana-dashboards.sh \
--url "<kibana-url>"

You'll be prompted to enter the Elasticsearch username and password.

Verify Dashboard Installation

After running the script:

1. Log in to Kibana
2. Navigate to Dashboard section
3. You should see pre-configured CodeMie dashboards including:
   • AI/Run Adoption - Platform adoption metrics and user engagement
   • CodeMie Assistants Dashboard - Assistant usage and performance metrics
   • CodeMie Datasource Observability - Data source indexing and query performance
   • CodeMie Workflow Usage - Workflow execution metrics and analytics

Additional Options

For more configuration options and help:

bash ./kibana-dashboards/manage-kibana-dashboards.sh --help

Post-Installation Validation

After completing observability stack installation, verify the following:

# Fluent Bit is running on all nodes
kubectl get daemonset -n fluentbit
kubectl get pods -n fluentbit

# Kibana is running

kubectl get pods -n elastic | grep kibana
kubectl get deployment -n elastic kibana

# Check Kibana is accessible

curl -k <kibana-url>

All checks should return successful results.

Next Steps

Congratulations! You have successfully completed the manual deployment of all AI/Run CodeMie components.

Proceed to Accessing Applications - Learn how to access the deployed AI/Run CodeMie applications and complete the required configuration steps.