Skip to content

danielwaichcesl/queue

Repository files navigation

⚠️ Work in Progress — This project is under active development and not ready for production yet.

🚀 Ultra-lightweight Queue Engine in Go

An ultra-lightweight queue engine, written in Golang, designed to support thousands of sequential queues with high performance, low memory usage, and reliable persistence.


🎯 Goal

This project aims to create a simple, fast, and independent queue solution — without relying on complex external systems like RabbitMQ or Kafka.
The goal is to achieve an architecture capable of handling millions of messages per second, focusing on:

  • ⚡ Extreme efficiency: minimal overhead per queue.
  • 🧱 Horizontal scalability: native support for thousands of concurrent queues.
  • 🎛️ Efficient batch processing: a publisher and consumer designed to work in batches, maximizing performance and resource utilization.
  • 🚀 Ultra-low latency & high throughput: optimized communication for near real-time processing at massive scale.
  • 💾 Reliable persistence: no message loss, even under failure conditions.
  • 🧩 Simplicity: easy to use, embed, and understand.

🧩 Current State

The project is in an early development stage — currently defining the architecture and building the first prototypes.
The core is being written entirely in Go, focusing on performance and code clarity.

⚠️ This repository is not ready for production yet, but contributions and ideas are welcome!


🤝 Contributing

Want to help build a modern, simple, and powerful open-source queue engine?

  • Fork the repository
  • Send PRs with improvements, ideas, or fixes
  • Open issues to discuss architecture, API design, or optimizations
  • Even small contributions are valuable — documentation, benchmarks, tests, or design feedback

🔮 Roadmap (initial vision)

  • Basic in-memory queue structure
  • Simple disk persistence
  • Efficient consumption
  • Multiple consumers
  • Basic monitoring (metrics)
  • Benchmark and comparison with other solutions
  • Optional CLI / REST API interface

💡 Philosophy

“Less protocol, more purpose.”

Most modern queue systems are designed to solve every possible problem — but in doing so, they carry a significant cost in complexity and overhead.
Protocols, brokers, network layers, replication — all of these have a price, often paid in latency and resource consumption.

This project takes a different path:
focus on pure performance and a clear purpose — to create and manage thousands of sequential queues with minimal client impact and no unnecessary dependencies.

Each queue is simple, predictable, and direct — it processes messages in order, respects available resources, and requires no heavy infrastructure.

The result is a transparent, optimized, and efficient queue engine — built to solve a specific problem with clarity and total control.


🧠 Technologies

  • 🐹 Golang — main programming language
  • 💾 Local storage / in-memory (optional) — for persistence and shared state
  • 🧩 Modular architecture — for easy future extension

About

⚡ Ultra-lightweight and predictable queue engine in Go — built for thousands of sequential queues with minimal client impact.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors