TechStar Asia - Tech News for Builders and Operators

Environment

OS: Ubuntu (snap docker)
Docker path: /var/snap/docker/common/var-lib-docker/
SigNoz path: /root/signoz/
SigNoz version: v0.117.1
ClickHouse version: 25.5.6
CPU: No AVX2 support (important — affects configuration)

1. Install SigNoz

cd /root
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/docker

2. Fix ClickHouse — Disable simdjson (REQUIRED for CPUs without AVX2)

CPUs without AVX2 will encounter CANNOT_ALLOCATE_MEMORY errors when SigNoz queries metrics via JSONExtractString.

Check if your CPU supports AVX2:

grep -o 'avx2' /proc/cpuinfo | head -1
# No output = no AVX2 support → apply the fix below

Edit /root/signoz/deploy/common/clickhouse/users.xml and add <allow_simdjson>0</allow_simdjson> to the default profile:

<profiles>
    <default>
        <max_memory_usage>10000000000</max_memory_usage>
        <allow_simdjson>0</allow_simdjson>   <!-- Fix for CPUs without AVX2 -->
        <load_balancing>random</load_balancing>
    </default>
    ...
</profiles>

Why this happens

ClickHouse uses the simdjson library for JSON parsing functions (JSONExtractString, etc.). Starting from ClickHouse 24.1+, simdjson requires AVX2 CPU instructions. On CPUs without AVX2, any query using JSONExtractString on the distributed_time_series_v4 table throws:

Code: 173. DB::Exception: Couldn't allocate N bytes when parsing JSON

Setting allow_simdjson=0 forces ClickHouse to fall back to rapidjson, which does not require AVX2.

This affects:

Infrastructure → Hosts tab (returns 500 error)
Any dashboard querying metrics with label filters

3. Disable ClickHouse System Logs (REQUIRED — prevent disk exhaustion)

By default, ClickHouse writes internal diagnostic logs (trace_log, metric_log, etc.) that can grow to 70GB+ over time.

Create /root/signoz/deploy/common/clickhouse/z_log_disable.xml:

<?xml version="1.0"?>
<clickhouse>
    <asynchronous_metric_log remove="1"/>
    <trace_log remove="1"/>
    <metric_log remove="1"/>
    <query_views_log remove="1"/>
    <query_thread_log remove="1"/>
    <part_log remove="1"/>
    <processors_profile_log remove="1"/>
    <opentelemetry_span_log remove="1"/>
</clickhouse>

The z_ prefix ensures this file is loaded last (configs are applied alphabetically), overriding any previous settings.

Mount this file into the ClickHouse container in /root/signoz/deploy/docker/docker-compose.yaml:

  clickhouse:
    volumes:
      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
      - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      - ../common/clickhouse/ttl.xml:/etc/clickhouse-server/config.d/ttl.xml
      - ../common/clickhouse/z_log_disable.xml:/etc/clickhouse-server/config.d/z_log_disable.xml  # ADD THIS LINE
      - clickhouse:/var/lib/clickhouse/

4. Configure SMTP for Email Alerts

Add the following environment variables to the signoz service in docker-compose.yaml:

  signoz:
    environment:
      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
      # SMTP configuration
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=your-email@gmail.com
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=smtp.gmail.com:587
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=your-email@gmail.com
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=your-16-char-app-password

Gmail note: You must use an App Password, not your regular Gmail password.
Generate one at: https://myaccount.google.com/apppasswords
The App Password is 16 characters — remove all spaces before using it.

Important: The correct variable names use the SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__ prefix with double underscores. Using SIGNOZ_ALERTMANAGER_SMTP_* (without SIGNOZ_GLOBAL) will not work.

5. Start SigNoz

cd /root/signoz/deploy/docker
docker compose up -d

Verify all containers are healthy:

docker compose ps

Verify ClickHouse system logs are disabled:

docker exec signoz-clickhouse clickhouse-client --query "
SELECT name FROM system.tables
WHERE database='system' AND name LIKE '%log%'
ORDER BY name"
# Only query_log should remain

6. Setup Host Metrics Monitoring (Infrastructure Tab)

Run an otel-collector agent on the main application server:

6.1 Create config file on the main server

mkdir -p /opt/otel-agent
cat > /opt/otel-agent/config.yaml << 'EOF'
receivers:
  hostmetrics:
    root_path: /hostfs
    collection_interval: 60s
    scrapers:
      cpu: {}
      disk: {}
      load: {}
      filesystem:
        exclude_mount_points:
          mount_points: [/snap/*, /boot/efi]
          match_type: regexp
      memory: {}
      network: {}
      paging: {}
      processes: {}

processors:
  batch:
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  resource/env:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp:
    endpoint: "<SIGNOZ_MONITOR_SERVER_IP>:4317"
    tls:
      insecure: true

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/env, batch]
      exporters: [otlp]
EOF

Replace <SIGNOZ_MONITOR_SERVER_IP> with your SigNoz monitor server IP.

6.2 Run the container

docker run -d \
  --name otel-agent \
  --restart unless-stopped \
  --hostname navio-main-server \
  -v /opt/otel-agent/config.yaml:/etc/otelcol-contrib/config.yaml \
  -v /:/hostfs:ro \
  -e HOST_PROC=/hostfs/proc \
  -e HOST_SYS=/hostfs/sys \
  -e HOST_ETC=/hostfs/etc \
  -e HOST_VAR=/hostfs/var \
  -e HOST_RUN=/hostfs/run \
  -e HOST_DEV=/hostfs/dev \
  otel/opentelemetry-collector-contrib:latest

After a few minutes, the host should appear in SigNoz UI → Infrastructure → Hosts.

7. Create Disk Usage Alert (> 80%)

Go to SigNoz UI → Alerts → New Alert → Metric Based Alert:

Query A (used disk):

Metric: system.filesystem.usage
Filter: state = used AND host.name = <your-hostname>
Time aggregation: avg, Space aggregation: sum

Query B (total disk — all states):

Metric: system.filesystem.usage
Filter: host.name = <your-hostname>
Time aggregation: avg, Space aggregation: sum

Formula F1: A / B * 100

Condition:

Warning: above 80
Critical: above 90
Evaluation window: last 5 min

Annotations:

Alert: Disk usage has exceeded {{$threshold}}% (current: {{$value}}%)

8. Routine Maintenance

# Check disk usage on monitor server
df -h /

# Verify ClickHouse system logs are not growing
docker exec signoz-clickhouse clickhouse-client --query "
SELECT table, formatReadableSize(sum(bytes_on_disk)) as size
FROM system.parts
WHERE database = 'system'
GROUP BY table
ORDER BY sum(bytes_on_disk) DESC"

# Clean up unused Docker images on main server
docker image prune -af

# Check Docker disk usage
docker system df

9. Troubleshooting

ClickHouse disk full due to system logs

# Enter clickhouse-client
docker exec -it signoz-clickhouse clickhouse-client

# Create force drop flag (run in a separate terminal)
docker exec signoz-clickhouse bash -c \
  "touch /var/lib/clickhouse/flags/force_drop_table && \
   chmod 666 /var/lib/clickhouse/flags/force_drop_table"

# Truncate all large log tables
TRUNCATE TABLE system.trace_log;
TRUNCATE TABLE system.metric_log;
TRUNCATE TABLE system.query_log;
TRUNCATE TABLE system.processors_profile_log;
TRUNCATE TABLE system.part_log;
TRUNCATE TABLE system.asynchronous_metric_log;
TRUNCATE TABLE system.query_views_log;

Infrastructure Hosts tab returns 500 error

Root cause: CPU does not support AVX2 instructions.

# Check AVX2 support
grep -o 'avx2' /proc/cpuinfo | head -1
# No output = no AVX2

Fix: Add <allow_simdjson>0</allow_simdjson> to ClickHouse users.xml as described in Section 2.

otel-agent not sending data

docker logs otel-agent 2>&1 | tail -20
# Check that host.name is detected correctly
# Check that the endpoint IP is correct

SigNoz Setup Guide - Monitor Server

Environment

1. Install SigNoz

2. Fix ClickHouse — Disable simdjson (REQUIRED for CPUs without AVX2)

Why this happens

3. Disable ClickHouse System Logs (REQUIRED — prevent disk exhaustion)

4. Configure SMTP for Email Alerts

5. Start SigNoz

6. Setup Host Metrics Monitoring (Infrastructure Tab)

6.1 Create config file on the main server

6.2 Run the container

7. Create Disk Usage Alert (> 80%)

8. Routine Maintenance

9. Troubleshooting

ClickHouse disk full due to system logs

Infrastructure Hosts tab returns 500 error

otel-agent not sending data

Comments (0)

United States

Related News

UCP Variant Data: The #1 Reason Agent Checkouts Fail

Amazon Employees Are 'Tokenmaxxing' Due To Pressure To Use AI Tools

How Braze’s CTO is rethinking engineering for the agentic area

Décryptage technique : Comment builder un téléchargeur de vidéos Reddit performant (DASH, HLS & WebAssembly)

How AI Reduced Manual Driver Verification by 75% — Operations Case Study. Part 2