Shortas Documentation

Architecture Overview

Shortas is built as a microservices architecture designed for high performance, scalability, and reliability. This document provides a comprehensive overview of the system architecture.

🏗️ System Architecture

High-Level Overview

graph TB
    A[Client Request] --> B[Load Balancer]
    B --> C[Click Router]
    C --> D[Click Tracker]
    D --> E[Click Aggregator]
    E --> F[Analytics Storage - ClickHouse]
    C --> G[Route & Settings DB - MongoDB/DynamoDB]
    C --> H[Cache - Redis/Moka]
    D --> I[Message Queue - Kafka/Fluvio]
    E --> I
    J(Click Router API) --> G
    K(Click Aggregator API) --> F
    L(Admin/User UI) --> J
    L --> K

🧩 Core Microservices

Shortas is built around five primary microservices:

1. Click Router

Function: A high-performance, intelligent URL redirection service built in Rust. Provides advanced routing capabilities with conditional logic, analytics, and multi-database support for enterprise-grade URL shortening and redirection services.

Key Features:

Advanced Routing:

Technologies: Rust, Salvo, MongoDB/DynamoDB, Moka (in-memory cache), Kafka/Fluvio, GeoIP, UA Parser.

2. Click Tracker

Function: Processes and enriches click event data in real-time. It captures details like user agent, IP address, geographic location, and device information.

Key Features:

Technologies: Rust, Kafka/Fluvio, GeoIP, UA Parser.

3. Click Aggregator

Function: Consumes enriched click data from the message queue, aggregates it, and stores it in the analytics database for reporting and analysis.

Key Features:

Technologies: Rust, ClickHouse, Kafka/Fluvio.

4. Click Router API

Function: A high-performance, secure click aggregation API with JWT authentication via Keycloak, comprehensive OpenAPI documentation, and support for multiple database backends.

Key Features:

Technologies: Rust, Salvo, MongoDB/DynamoDB, Keycloak (for JWT).

5. Click Aggregator API

Function: A high-performance, secure click aggregation API with JWT authentication via Keycloak, comprehensive OpenAPI documentation, and ClickHouse integration for analytics.

Key Features:

Technologies: Rust, Salvo, ClickHouse, Keycloak (for JWT).

🏗️ Click Router Architecture

Click Router uses a modular, pipeline-based architecture:

Request → Flow Router → Modules → Adapters → Response

Core Components

Request Processing Pipeline

  1. Start: Initial request processing and validation
  2. UrlExtract: URL analysis and route matching
  3. Register: Hit logging and analytics
  4. BuildResult: Response generation
  5. End: Final response processing

Project Structure

src/
├── adapters/          # Service integrations
│   ├── aws/          # DynamoDB integration
│   ├── mongodb/      # MongoDB integration
│   ├── moka/         # Caching layer
│   └── fluvio/       # Analytics streaming
├── core/             # Core routing logic
│   ├── flow_router.rs # Main router
│   └── modules/      # Processing modules
├── model/            # Data models
└── settings.rs       # Configuration

🔄 Data Flow

The data flow within Shortas is designed for high throughput and real-time processing:

  1. Incoming Request: A user clicks a short URL, sending an HTTP request to the Click Router.
  2. Route Resolution: The Click Router resolves the short URL to its long destination, potentially applying conditional logic based on request parameters (e.g., user agent, geo-location). It queries MongoDB/DynamoDB for route information, utilizing Redis/Moka for caching.
  3. Hit Tracking: Before redirection, the Click Router sends a raw click event to the Click Tracker via a message queue (Kafka/Fluvio).
  4. Data Enrichment: The Click Tracker enriches the raw click event with additional metadata (e.g., device type, OS, browser, country from GeoIP/UA Parser) and publishes the enriched event back to the message queue.
  5. Data Aggregation: The Click Aggregator consumes the enriched click events from the message queue, performs necessary aggregations, and stores the data in ClickHouse.
  6. Redirection: The Click Router issues an HTTP redirect (301, 302, etc.) to the user’s browser, sending them to the long destination URL.
  7. API Access:
    • The Click Router API is used by administrators or user interfaces to create, update, or delete short URLs and manage settings.
    • The Click Aggregator API is used to retrieve analytics reports and raw click stream data from ClickHouse.

🗄️ Data Storage and Caching

🌐 Network and Communication

🔒 Security Considerations

Next Steps: Explore the API Reference for detailed information on interacting with Shortas services.