smartnessai/queen-mq repository overview

⁠Queen MQ - PostgreSQL-backed Message Queue

Queen MQ is a modern PostgreSQL-backed Message Queue, high-performance, feature-rich system with unlimited FIFO partitions, consumer groups, transactions, tracing and streaming capabilities with exact-once delivery guarantees.

⁠Why Queen?

Born at Smartness⁠ to power Smartchat, Queen solves a unique problem: unlimited FIFO partitions where slow processing in one partition doesn't block others.

Perfect for:

• Processing messages in order, without losing them somewhere in the middle
• Do not want that a slow message processing in one partition blocks other partitions
• When you need thousands of thousands of partitions to process messages in parallel, respecting the order of the messages
• Process the same messages in multiple pipelines
• Have a clear view of the message processing flow and traceability
• Build event-driven microservices with exactly-once delivery guarantees
• Critical systems that need to be highly available and reliable with zero message loss

⁠Quick Start

Run PostgreSQL and Queen server in Docker:

# Start PostgreSQL and Queen
docker network create queen

docker run --name postgres --network queen \
  -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres

docker run -p 6632:6632 --network queen \
  -e PG_HOST=postgres -e PG_PASSWORD=postgres \
  smartnessai/queen-mq

Install client:

npm install queen-mq

Use the client:

import { Queen } from 'queen-mq'

const queen = new Queen('http://localhost:6632')

// Create queue
await queen.queue('orders')
  .config({ leaseTime: 30, retryLimit: 3 })
  .create()

// Push with guaranteed order per partition
await queen.queue('orders')
  .partition('customer-123')
  .push([{ data: { orderId: 'ORD-001', amount: 99.99 } }])

// Consume with consumer groups
await queen.queue('orders')
  .group('order-processor')
  .concurrency(10)
  .batch(20)
  .consume(async (message) => {
    await processOrder(message.data)
  })

Complete Tutorial →⁠

⁠Key Features

• 🎯 Unlimited FIFO Partitions - No limits on ordered message streams
• 👥 Consumer Groups - Kafka-style with replay from any timestamp
• 🔄 Transactions - Atomic push+ack for exactly-once delivery
• 📡 Streaming - Real-time message streams via WebSocket
• 🛡️ Zero Message Loss - Automatic failover to disk buffer
• 💀 Dead Letter Queue - Automatic handling of failed messages
• 🔍 Message Tracing - Debug distributed workflows
• 📊 Modern Dashboard - Vue 3 web interface with analytics
• 🔐 Encryption - Per-queue message encryption
• ⚡ High Performance - 200K+ msg/s with proper batching

Compare with Kafka, RabbitMQ, NATS →⁠

⁠Performance

Detailed Benchmarks →⁠

⁠Webapp

A modern Vue 3 dashboard for monitoring and managing Queen MQ:

Webapp Documentation →⁠

⁠Documentation

📚 Complete Documentation⁠

⁠Getting Started

• Quick Start Guide⁠
• Installation⁠
• Basic Concepts⁠

⁠Client Libraries

• JavaScript Client⁠
• C++ Client⁠
• HTTP API Reference⁠

⁠Server

• Architecture⁠
• Installation & Build⁠
• Configuration⁠
• Environment Variables⁠
• Deployment (Docker, K8s, systemd)⁠
• Performance Tuning⁠

⁠Features

• Queues & Partitions⁠
• Consumer Groups⁠
• Transactions⁠
• Long Polling⁠
• Streaming⁠
• Dead Letter Queue⁠
• Message Tracing⁠
• Failover & Recovery⁠

⁠Production Ready

Queen is currently running in production at Smartness⁠, handling 200k messages daily.

⁠Community & Support

• 🌐 Website: smartpricing.github.io/queen⁠
• 📦 Docker Hub: smartnessai/queen-mq⁠
• 💼 LinkedIn: Smartness Company Page⁠
• 🐛 Issues: GitHub Issues⁠

⁠License

Queen MQ is released under the Apache 2.0 License⁠.