Agents & Geo Monitoring

Monitor your services from multiple locations around the world using lightweight Go agents. Deploy private agents for internal infrastructure or use geo agents for distributed monitoring.

World map showing 8 geo monitoring agents across Europe and the US

Overview

SysTeam HealthChecks uses a Go-based agent that can run checks from remote locations. There are two modes of operation:

Private Agents

A private agent runs inside your network and executes checks assigned to it. This allows you to monitor internal services that are not accessible from the internet (databases, internal APIs, microservices, etc.).

How It Works

1. You deploy the agent binary on a server inside your network
2. The agent connects to the HealthChecks API using a secure agent key
3. You assign specific checks to the agent in the UI
4. The agent fetches its assigned checks, executes them, and reports results back
5. The agent sends heartbeats so you can monitor its own health

Deploying a Private Agent

Step 1: Generate a Registration Token

1. Go to Agents in the sidebar
2. Click Generate Token
3. Copy the one-time registration token (it starts with reg_)

Step 2: Install the Agent

The agent is a single Go binary (~15 MB). You can build it from source or use the Docker image.

docker run -d \
  --name healthcheck-agent \
  --restart unless-stopped \
  -e HC_API_URL=https://your-healthchecks-domain.com \
  -e HC_REGISTER_TOKEN=reg_your_token_here \
  -e HC_AGENT_NAME=my-private-agent \
  registry.cygal.net/healthcheck-agent:latest

# Download or build the agent binary
# Place it on your server, then:

export HC_API_URL=https://your-healthchecks-domain.com
export HC_REGISTER_TOKEN=reg_your_token_here
export HC_AGENT_NAME=my-private-agent

./healthcheck-agent

# /etc/systemd/system/healthcheck-agent.service
[Unit]
Description=HealthCheck Agent
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/healthcheck-agent
Environment=HC_API_URL=https://your-healthchecks-domain.com
Environment=HC_AGENT_KEY=your_agent_key_here
Restart=always
RestartSec=10
User=healthcheck

[Install]
WantedBy=multi-user.target

Step 3: Verify Registration

After starting the agent, go to Agents in the sidebar. You should see your agent listed with a green Online status. The agent sends heartbeats every 30 seconds.

Step 4: Assign Checks

1. Create a new check or edit an existing one
2. In the Agent dropdown, select your private agent
3. Save the check

The agent will pick up the check on its next polling cycle (typically within 10-30 seconds) and start executing it.

Agent Configuration

Supported Check Types

Private agents can execute these check types:

• HTTP/HTTPS — full support including headers, body, content matching, SSL info
• TCP — port connectivity, response time, banner grab
• ICMP — ping with packet loss, min/avg/max RTT
• DNS — record resolution with multi-server iteration
• Mail Server — SMTP/IMAP/POP3 connectivity checks

Agent Status

In the Agents page, each agent shows:

Geo Monitoring

Geo monitoring lets you check your services from multiple geographic locations simultaneously. Instead of checking from a single point, your services are tested from several locations around the world, giving you visibility into regional performance and availability.

How Geo Monitoring Works

1. When you enable Geo Monitoring on a check, it is distributed to all active geo agents
2. Each geo agent executes the check independently from its location
3. Results are collected and aggregated into an overall geo status
4. You can see per-location response times, status, and latency comparisons

Enabling Geo Monitoring

1. Create or edit a check
2. Toggle Geo Monitoring to enabled
3. Save the check

That's it. The check will automatically be picked up by all active geo agents. No agent assignment needed — geo monitoring uses all available geo agents.

Current Monitoring Locations

We currently operate 10 geo monitoring agents across Europe and the United States:

Geo Status Aggregation

When geo monitoring is enabled, the check's overall status is determined by aggregating results from all active locations:

The DEGRADED status indicates a partial outage — your service is reachable from some locations but not others. This could indicate regional DNS issues, CDN problems, or network routing problems.

Excluding Locations

You can exclude specific locations from a check's geo monitoring. This is useful when:

• A location is experiencing known issues unrelated to your service
• Your service is intentionally not available in certain regions
• You want to focus monitoring on specific regions

To exclude locations, edit the check and use the Geo Excluded Locations field. Enter location codes separated by commas (e.g., US-TX,US-WA). Excluded locations are filtered from:

• Check routing (agents in excluded locations won't execute the check)
• Status aggregation (excluded locations don't affect the overall status)
• All geo charts and visualizations

Geo Monitoring UI

When you open the detail page for a geo-enabled check, you'll see additional panels not available for standard checks:

Geo Status Map

A visual overview showing the current status of each monitoring location. Each location displays as a colored card:

• Green — Location reports UP with response time
• Red — Location reports DOWN with error message
• Gray — No data yet or location excluded

Geo Response Time Chart

A multi-line time series chart showing response time from each location over time. Each location is a separate colored line. Use this to:

• Compare latency between locations
• Identify regional slowdowns
• Spot trends in response time by region

Location Comparison Panel

Side-by-side metrics comparison between locations showing:

• Average response time
• Minimum / Maximum response time
• Current status and last check time

Geo Histogram

A visual timeline showing status history per location. Each row is a location, and colored segments show UP (green), DOWN (red), and DEGRADED (amber) periods.

Response Time Source Toggle

For geo-enabled checks, the main Response Time History chart at the top of the check detail page has a toggle switch:

• Default Location — Shows response time from your starred/default location only. This gives you a single consistent perspective.
• Average — Shows the average response time across all active (non-excluded) locations. Useful for seeing overall trends.

The default location is the one marked with a star icon in your geo preferences. You can change it in the check's geo preferences section.

Central vs Agent vs Geo Checks

Understanding the three execution modes helps you choose the right approach:

Mail Server Geo Monitoring

Mail server checks with geo monitoring use a special split execution model:

• Geo agents run connectivity checks: SMTP, IMAP, POP3 port availability and response time from each location
• Central server runs DNS-based checks once: SPF, DKIM, DMARC validation, PTR record, and RBL blacklist checks

This avoids redundant DNS queries from multiple locations (SPF/DKIM/DMARC results are the same regardless of where you check from) while still giving you regional connectivity data.

DNS Check Geo Monitoring

DNS checks with geo monitoring also use a split model:

• Geo agents run basic DNS resolution from each location (measuring DNS response time per region)
• Central server runs advanced checks: multi-record monitoring, SOA tracking, WHOIS expiry

Auto-Diagnostic MTR/Traceroute

When any agent-executed check fails with a timeout, the agent automatically runs an MTR (My Traceroute) diagnostic to the target host. This captures the full network path, showing exactly where packets are being lost or delayed.

How It Works

1. Agent runs a check (HTTP, TCP, DNS, etc.) and it times out
2. Agent detects the timeout error and runs mtr -c 3 -n --json <host>
3. MTR results (hop-by-hop data) are attached to the check result
4. Results appear in the check detail page under the failed log entry

What You See

When a timed-out log entry contains traceroute data, you'll see a Network Path (MTR) section showing a hop table with:

• Hop # — Position in the network path (1 = first router, last = destination)
• Host — IP address of each hop (or * if unreachable)
• Loss % — Packet loss at each hop (color-coded: green = 0%, amber = >10%, red = >50%)
• Avg / Best / Worst — Round-trip latency in milliseconds

This helps you quickly identify whether a timeout is caused by:

• A problem at the destination server (loss only at the last hop)
• A transit/ISP issue (loss at an intermediate hop)
• A network routing problem (high latency jumps between hops)

Configuring MTR

MTR on timeout is enabled by default for all checks. You can disable it per check:

1. Edit the check
2. Uncheck Run MTR on timeout
3. Save the check

Managing Agents

The Agents page in the sidebar provides a full overview of all registered agents:

• Agent list — Name, status, mode (Private/Geo), location, last heartbeat
• Generate Token — Create new one-time registration tokens
• Agent details — Click an agent to see its assigned checks, recent results, and configuration

Agent Authentication

Agents authenticate using an X-Agent-Key header. The key is a long random string generated during registration. It is hashed with SHA-256 on the server side — the raw key is never stored. If you lose the key, you must re-register the agent with a new token.

Next Steps

• Learn how to read dashboard charts
• Set up SLA/SLO tracking
• Supported check types
• API reference for agent endpoints