Logdive: A leap forward in log querying for developers

In the fast-paced world of programming and software development, debugging production applications can become a nightmare. When something breaks in the middle of the night, your options are often limited and inefficient: from grep on rotated log files to complex jq command chains that quickly become unmanageable. More robust solutions, such as Loki or Elastic, often involve infrastructure overhead and configuration that isn't always desirable, especially for small projects or personal research.

This is the scenario where logdive emerges: a new open source tool that seeks to democratize access and querying of structured logs locally and efficiently. Born from the frustration of needing a version of jq with memory, time ranges, and advanced filters, logdive presents itself as a pragmatic and powerful solution.

The problem: Log tools, between simplicity and complexity

Backend engineers often deal with well-structured JSON logs, but the available tools operate at opposite extremes. On one hand, jq is excellent for individual files and one-off queries, but it lacks memory, time ranges, or the ability to filter across multiple files. On the other hand, platforms like Datadog, Splunk, or Elastic demand a considerable investment in infrastructure, costs, and configuration, which is overkill for many scenarios.

Logdive positions itself precisely in this niche, offering a single binary written in Rust that, once installed (cargo install logdive), allows you to index and query logs directly on your local machine. No daemons to manage, no cloud dependencies, and no YAML configuration files.

Logdive in action: Efficient ingestion and querying

Logdive's operation is surprisingly simple:

• Log ingestion: You can feed logdive from a log file or via a pipe from the output of other commands, such as docker logs.

  logdive ingest --file ./logs/app.log
  docker logs my-container | logdive ingest --tag my-container
  

• Powerful queries: The query language allows for filtering by specific fields, time ranges, and the use of the CONTAINS operator. Known fields (timestamp, level, message, tag) benefit from direct SQLite indexes, while arbitrary fields are queried via json_extract().

  logdive query 'level=error AND service=payments last 2h'
  logdive query 'message contains "timeout"' --format json | jq
  

• Stats and API: The tool also offers logdive stats to inspect indexed data and, optionally, exposes a read-only HTTP API (logdive-api) for remote queries.

Design and performance: The power of Rust and SQLite

The choice of Rust for developing logdive is no coincidence. It allows for creating a self-contained binary, with no external runtime, zero-copy parsing, direct SQLite integration, and real concurrency for ingestion. The result is a surprisingly small binary (around 3.7 MB for logdive) and remarkable performance.

Performance metrics are impressive:

• Ingestion: up to ~210k lines/second.
• Queries on known fields: latencies of ~17 μs.
• Queries on JSON fields: up to ~3.6 ms.

Logdive is designed to be small in scope, but expressive enough to be useful. Version 0.1.0 focuses on the essential needs of a side project or a small team.

Who is it for (and who is it not for)

Logdive is ideal for:

• Backend engineers debugging production incidents with local copies of logs.
• Small teams without a dedicated budget for observability.
• Developers who use Docker locally and want to query docker logs instantly.
• CI pipelines that need to filter structured outputs.

However, it is not a solution for multi-machine indexing, real-time log tailing, or massive enterprise-scale log volumes. It also does not support non-JSON log formats in its current version.

A growing ecosystem

Logdive is an open source project that welcomes contributions. It functions as a workspace of three crates: logdive-core (the main logic), logdive (the CLI), and logdive-api (the HTTP server). Future versions (v2) plan to include the OR operator, support for log formats like plaintext and logfmt, a follow mode, and even a browser-based graphical user interface (GUI).

In a landscape where observability solutions can be overwhelming, logdive offers a lightweight, fast, and accessible alternative. It is a tool that, inspired by the power of jq but overcoming its limitations, demonstrates the value of well-designed open source solutions written in languages like Rust, which can compete even with the most complex commercial solutions in specific niches. If you are a developer who values simplicity and local efficiency, logdive deserves your attention, just like other promising tools in the open source space such as Utilyze, which seeks to provide precision in GPU measurement.

Sources:

• Dev.to: I wanted jq with memory, time ranges, and filters. So I built logdive
• GitHub Repository: Aryagorjipour/logdive