Nagios
Nagios monitoring starts with a concrete contract between a check and its scheduler: the plugin reports a state and concise diagnostic output, and the configured monitoring logic decides what happens next. A useful deployment also defines retries, dependencies, maintenance, contacts, and recovery handling.
Begin with one service and one action an operator can take. Test the plugin’s healthy, warning, critical, and unknown outcomes before integrating it. A perfectly formatted message is not enough if the process exits with the wrong code or silently reports a collector failure as healthy.
TL;DR
- Keep plugin state codes and human-readable output consistent.
- Represent invalid or unavailable collection distinctly from measured success.
- Review retry and soft/hard state behavior before expecting a notification.
- Use dependencies and maintenance for defined operating conditions.
- Verify the actual receiver path, recovery, and monitoring freshness.
Quick Example
Save this complete Python script as check_demo_queue.py and supply one fictional numeric depth: 0 yields OK, 10 yields WARNING, 20 yields CRITICAL, and nan yields UNKNOWN. The thresholds are illustrative and are not capacity guidance.
Ten actual subprocess cases passed, including all four return codes, missing/extra arguments, negative values, nonfinite values, invalid text, and performance-data separation. This script takes a supplied number rather than collecting a real queue. No Nagios Core scheduler, host-state handling, remote execution, notification, or service integration was tested.
Core Concepts
Plugin contract
For service checks, Nagios Core interprets return codes 0, 1, 2, and 3 as OK, WARNING, CRITICAL, and UNKNOWN. Plugins provide text output and may append performance data after a pipe separator. Host-check interpretation has additional rules, so do not assume this service mapping describes every host state. Plugin API
State transitions
Nagios distinguishes SOFT and HARD state types; configured check attempts influence that transition and notification behavior. State types
Delivery is another boundary
Notification processing includes service/host state, configured contacts, and multiple filters. A critical check result does not by itself prove a message reached an operator. Notification logic
Build a check contract before writing the collector
Avoid embedding credentials or arbitrary user-controlled commands in plugin arguments. Use an approved credential mechanism and the minimum access required by the actual collector. A local numeric demonstration needs neither remote credentials nor command execution.
Keep the first line useful under pressure: the state, measured condition, and scope should be understandable without reading a large dump. Performance data is machine-facing output with its own format; maintain it separately from prose so a wording improvement does not break downstream interpretation.
Choose thresholds from the service’s behavior and required response. A queue depth alone may be less informative than age, progress, or a relationship to available capacity. The example’s 10 and 20 boundaries exist only to make code paths easy to test.
Follow the result through scheduler state
Run the plugin under the same identity, working assumptions, environment, and timeout budget as the monitoring process. A command that works in an administrator’s interactive shell may fail under the service account or return a different locale-dependent format.
Inspect actual execution and freshness before adjusting notification rules. Distinguish a check that never ran, a plugin that returned UNKNOWN, a result waiting for another attempt, and a HARD problem with delivery suppressed by configuration.
Review dependencies as explicit relationships. Nagios documents that dependencies can suppress check execution or notifications under selected conditions, and inheritance must be configured where required. Suppression should follow a known operational relationship, not merely make a noisy dashboard quieter. Dependencies
Keep independent symptoms when they help assess user impact. A failed upstream dependency may explain many child failures, while one child can also have a separate problem. Choose which observations to retain and which notifications to group or suppress based on diagnostic value.
Exercise maintenance and recovery
Nagios Core distinguishes fixed downtime with explicit start/end times from flexible downtime that starts under documented state conditions within a window. Select the behavior that matches the planned work and verify scope, timing, and expiry. Scheduled downtime
Before relying on the monitor, use an approved environment or drill to verify problem delivery to the intended contact and the corresponding recovery behavior. Check time periods, contact filters, escalation, and the diagnostic context required for a first action.
Observe the monitoring system itself. Check scheduling delays, plugin runtime, unavailable collectors, and result age. The last recorded OK value should not remain a current health claim when no fresh check has completed.
Review the check after service or plugin changes. Re-test boundary values and unavailable-input behavior, then confirm the service account and receiver path still work. A source-controlled plugin plus an owned deployment procedure is easier to maintain than an undocumented command on one server.
Best Practices
Test every state explicitly
Verify exit status and output together, including invalid input and timeout handling. Include exact threshold boundaries.
Trace the notification path
Inspect state type, filters, dependencies, contacts, and receiver behavior separately. Record what each approved exercise actually demonstrated.
Keep maintenance bounded
Use the correct scope and timing, verify expiry, and confirm normal detection resumes after planned work.
Common Mistakes
Bad: The message says CRITICAL, so Nagios must see CRITICAL.
Correct: Check the process exit code; text and state code must agree.
Bad: Failed collection can return zero as a harmless default.
Correct: Preserve unavailable or invalid measurement as a distinct state.
Bad: Downtime and dependencies are generic noise switches.
Correct: Apply them to documented operating relationships and verify the intended detection and delivery consequences.
FAQ
Does this cover Nagios Core or every Nagios product?
The cited contracts are from Nagios Core documentation. Verify differences in the exact product and version you operate.
Why use UNKNOWN for invalid input?
The example cannot establish a measured service state from invalid input. Returning healthy would misrepresent collection failure.
Does the plugin test prove notifications work?
No. It proves local process outputs and return codes for supplied inputs. Scheduler transitions and actual delivery require separate integration checks.
Related Topics
- Monitoring — Coverage, freshness, and observation methods.
- Zabbix — A different monitoring architecture and operating workflow.
- Actionable Alerts and Noise Reduction — Response contracts and alert usefulness.