Visla GPS Backend: Complete Data Flow & Communication Analysis

This document provides a comprehensive mapping of all data flows, APIs, protocols, sockets, and communication patterns in the Java monolithic backend (backend.vislagps.com).

---

📋 Table of Contents

1. System Architecture
2. External Access Layer (Nginx Gateway)
3. Real-time & Device Layer
4. REST API Inventory
5. Data Forwarding & Integration Layer
6. Notificators & External Services
7. Database Schema
8. Complete Data Flow Summary

---

1. 🏗️ System Architecture

The system is divided into two main communication layers: the HTTP Layer for user applications and the TCP/UDP Layer for GPS devices.

High-Level Architecture Diagram

                            ┌─────────────────────────────┐
                            │         INTERNET            │
                            └──────────┬──────────────────┘
                                       │
                    ┌──────────────────┴──────────────────┐
                    │                                     │
                    ▼                                     ▼
         ┌──────────────────────┐            ┌──────────────────────┐
         │   Web/Mobile Apps    │            │     GPS Devices      │
         │   (HTTPS/WSS + JWT)  │            │   (TCP/UDP :5000+)   │
         └──────────┬───────────┘            └──────────┬───────────┘
                    │                                   │
                    ▼                                   ▼
         ┌──────────────────────┐            ┌──────────────────────┐
         │   Nginx Gateway      │            │  Java Device Gateway │
         │     (Port 443)       │            │(backend.vislagps.com)│
         └──────────┬───────────┘            └──────────┬───────────┘
                    │                                   │
                    │ ALL requests                      │ Decode Protocol
                    │ (HTTP + WebSocket)                │
                    ▼                                   ▼
         ┌──────────────────────┐            ┌──────────────────────┐
         │  Unified Auth Flow   │            │  Position Handler    │
         │                      │            │                      │
         │ 1. auth_request to   │            │ • Parse Protocol     │
         │    /auth/validate    │            │ • Generate Position  │
         │                      │            │ • Detect Events      │
         │ 2. Validate JWT      │            │ • Trigger Rules      │
         │    Token             │            │                      │
         │                      │            └─────────┬────────────┘
         │ 3. Return 200/401    │                      │
         │                      │                      │
         │ 4. IF 200:           │                      │
         │    • HTTP API →      │                      │
         │      microservice    │                      │
         │    • WebSocket →     │                      │
         │      upgrade to      │                      │
         │      :8086           │                      │
         │                      │                      │
         │ 5. IF 401:           │                      │
         │    Reject request    │                      │
         └──────────┬───────────┘                      │
                    │                                  │
        ┌───────────┴─────────┐                        │
        │                     │                        │
        ▼                     ▼                        │
┌────────────────┐  ┌────────────────────┐             │
│ HTTP APIs      │  │ WebSocket          │             │
│ (after auth)   │  │ (after auth)       │             │
└───────┬────────┘  └────────┬───────────┘             │
        │                    │                         │
        └─────────┬──────────┘                         │
                  ▼                                    │
       ┌──────────────────────┐                        │
       │   Microservices      │                        │
       │  ┌────────────────┐  │                        │
       │  │ Auth :8081     │  │                        │
       │  │ Device :8082   │  │                        │
       │  │ Docs :8090     │  │  (dev only)            │
       │  │ Tracking :8083 │  │                        │                        │
       │  │ Command :8084  │  │                        │
       │  │ Notif  :8085   │  │                        │
       │  │ Report:8086    │◄─┼────WebSocket───────────┼─────┐
       │  │ Config :8087   │  │  (persistent)          │     │
       │  └────────┬───────┘  │                        │     │
       └───────────┼──────────┘                        │     │
                   │                                   │     │
                   │                                   │     │
       ┌───────────┴───────────────────────────────────┴─────┴─────┐
       │                                                           │
       ▼                                                           ▼
┌──────────────────┐                                    ┌──────────────────┐
│ MySQL Database   │                                    │  Redis Instance  │
│                  │                                    │                  │
│ • positions      │                                    │ • JWT Cache      │
│ • events         │                                    │ • Pub/Sub        │
│ • devices        │                                    │   - positions    │
│ • users          │                                    │   - events       │
│ • permissions    │                                    │   - devices      │
└──────────────────┘                                    └────────┬─────────┘
                                                                 │
                                                                 │ Subscribe
                                                                 ▼
                                                      ┌──────────────────────┐
                                                      │ Notification Service │
                                                      │   (Port 8086)        │
                                                      │                      │
                                                      │ • Subscribe to Redis │
                                                      │ • WebSocket Server   │
                                                      │ • Push Notifications │
                                                      │ • Email/Phone        │
                                                      └──────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                              DATA FLOWS                                     │
├─────────────────────────────────────────────────────────────────────────────┤
│ [1] HTTP API Flow (Web/Mobile → Microservices)                              │
│     Client (JWT) → Nginx → /auth/validate → 200 OK → Microservice           │
│                                                                             │
│ [2] WebSocket Flow (Web/Mobile → Notification Service)                      │
│     Client (JWT) → Nginx → /auth/validate → 200 OK → WS Upgrade :8086       │
│     → Notification Service establishes persistent WebSocket connection      │
│     Note: Auth happens ONCE during handshake, then connection stays open    │
│                                                                             │
│ [3] Device Ingestion Flow (GPS → Backend)                                   │
│     Device → Java Gateway → MySQL + Redis Pub/Sub + Forwarder               │
│                                                                             │
│ [4] Real-time Notification Flow (Backend → Client)                          │
│     Redis Pub/Sub → Notification Service → WebSocket → Client               │
│                                                                             │
│ [5] Authentication Flow (Login)                                             │
│     Client → Nginx → Auth Service → MySQL → JWT Token → Client              │
└─────────────────────────────────────────────────────────────────────────────┘

---

2. 🌐 External Access Layer (Nginx Gateway)

The Nginx Gateway is the only entry point for HTTP/WebSocket traffic. It handles SSL termination, CORS, and routing.

Routing Table

Path Prefix	Target Service	Port	Notes
/api/swagger	Swagger Service	8090	Unified Swagger UI (dev only)
/api/auth/*	Auth Service	8081	Identity, JWT Auth, OAuth
/api/socket	Notification Service	8086	WebSocket Upgrade
/api/devices, /api/groups	device Service	8082	Asset Management
/api/positions, /api/events	Tracking Service	8083	Historical Data
/api/commands	Command Service	8084	Device Control
/api/reports, /api/statistics	Report Service	8085	Analytics
/api/notifications	Notification Service	8086	Rules & Alerts
/api/server, /api/attributes	Config Service	8087	System Settings

Authentication Flow (Nginx auth_request)

Nginx performs an internal subrequest to validate ALL requests (both HTTP and WebSocket) before forwarding them to a microservice.

Unified Authentication:

• HTTP APIs: Validated via auth_request, then forwarded to microservice
• WebSocket: Validated via auth_request, then upgraded to WebSocket (if valid)
• Authentication: JWT access token only (stateless, no sessions)

1. Client → Nginx (Request + JWT Access Token in Authorization header)
2. Nginx → Auth Service (/api/auth/validate)
3. Auth Service validates JWT → Returns 200/401/403
4a. HTTP API: Nginx forwards to Target Service (if 200)
4b. WebSocket: Nginx upgrades to WebSocket on :8086 (if 200)
5. If 401/403: Nginx returns Error to client

---

3. 📡 Real-time & Device Layer

This layer handles the high-frequency data ingestion from devices and real-time updates to clients.

A. Device Communication (Java Gateway)

The Java Gateway (backend.vislagps.com) acts as the Protocol Translator. It listens on specific ports, decodes proprietary GPS protocols into standardized data, and ingests it into the system.

Supported Protocols & Ports: Note: A protocol is only active if configured in traccar.xml.

Protocol	Port	Model Map	Description
s21l	5052	S21L	Dedicated Huabao for S21L
g11lse	5058	G11L-SE	Dedicated GT06 for G11L-SE
c059	5059	C059	Dedicated P5 for C059
p5	5054	P5	Dedicated Huabao for P5

B. Real-time Data Flow (The "Live" Loop)

This flow explains how a position moves from a device to a user's screen in milliseconds.

1. Ingestion
   Device -> [TCP/UDP Port] -> Java Gateway
   Java Gateway decodes binary data -> Standardized Position Object

2. Broadcasting (Internal)
   Java Gateway -> Redis Stream (Channel: "positions")
   Java Gateway -> PSQL (Persistence)

3. Delivery (WebSocket)
   Redis -> Notification Service (Subscribed to "positions")
   Notification Service -> WebSocket Client (User)

C. WebSocket Protocol (/api/socket)

Connection:

• Endpoint: wss://backend.vislagps.com/api/socket or ws://localhost:8086/api/socket
• Handled by: Notification Service (Port 8086)
• Routing: Nginx Gateway → /auth/validate (JWT validation) → Notification Service :8086
• Authentication: JWT token validated by Auth Service via auth_request before WebSocket upgrade
• Protocol: After successful auth, HTTP upgrades to persistent WebSocket connection

Messages Sent to Client:

{
  "positions": [{ ... }],
  "devices": [{ ... }],
  "events": [{ ... }],
  "logs": [{ ... }]
}

Messages Received from Client:

{
  "logs": true  // Enable/disable log streaming
}

Keepalive:

• Server sends empty {} messages every N seconds
• Client should respond or maintain connection

---

4. 📦 REST API Inventory

🅰️ Auth Service (Port 8081)

FastAPI-based service responsible for identity, access control, and JWT authentication.

Base path: /api/auth

Endpoint	Method	Auth Required	Description
/api/auth/login	POST	❌	Login with email/password. Returns access token (15-min) + refresh token (30-day).
/api/auth/register	POST	❌	Register new user (requires terms consent, sends verification email).
/api/auth/verify	POST	❌	Verify email with token from registration email.
/api/auth/resend-verification	POST	❌	Resend verification email (if not received).
/api/auth/profile	GET	✅	Get current authenticated user info from JWT.
/api/auth/logout	DELETE	✅	Logout (revoke all refresh tokens for user).
/api/auth/refresh	POST	❌	Exchange refresh token for new access token.
/api/auth/validate	GET	❌	Internal JWT validation endpoint (used by Nginx auth_request).
/api/auth/health	GET	❌	Health check (database connectivity).
/api/auth/delete	POST	✅	Delete user account (immediate token invalidation).
/api/auth/openid/auth	GET	❌	Initiate OAuth flow (Google/Facebook/Apple).
/api/auth/openid/callback	GET/POST	❌	OAuth callback handler (receives code, returns tokens).
/api/auth/openid/token	POST	❌	Mobile OAuth token exchange (native SDK → JWT tokens).
/api/auth/password/reset	POST	❌	Request password reset email.
/api/auth/password/update	POST	❌	Update password using reset token.
/api/auth/2fa/setup	POST	✅	Generate TOTP secret and QR code.
/api/auth/2fa/verify	POST	✅	Verify TOTP code to enable 2FA.
/api/auth/2fa/disable	POST	✅	Disable 2FA (requires password).
/api/auth/2fa/backup-codes	POST	✅	Generate new backup codes.

Authentication:

• Type: JWT-based (stateless)
• Access Token: 15-minute expiration, signed with RSA private key
• Refresh Token: 30-day expiration, stored in database, revocable
• Token Blacklist: On logout, access tokens are blacklisted in Redis until expiration
• Header: Authorization: Bearer <access_token> or Cookie: access_token=<token>

Token Response Format:

{
  "access_token": "eyJ...",
  "refresh_token": "random-64-char-token",
  "token_type": "bearer",
  "user": { "id": 1, "email": "...", "administrator": false, ... }
}

🅱️ Device Service (Port 8082)

FastAPI-based service responsible for the static inventory of devices.

Base path: /api/devices

Endpoint	Method	Auth Required	Description
/api/devices	GET	✅	List all devices the user has access to.
/api/devices	POST	✅	Register a new device.
/api/devices/{id}	GET	✅	Get a single device by ID.
/api/devices/{id}	PUT	✅	Update device details (name, group, phone).
/api/devices/{id}	DELETE	✅	Delete a device.
/api/devices/claim	POST	✅	Claim a device using a unique token (QR code).
/api/devices/validate-token	POST	❌	Validate a device claim token (public).
/api/devices/health	GET	❌	Health check (database connectivity).

📚 Swagger Service (Port 8090)

Lightweight service providing unified API documentation in development mode.

Base path: /api/swagger

Endpoint	Method	Auth Required	Description
/api/swagger	GET	❌	Unified Swagger UI with dropdown to select Auth, Device, or Tracking service.
/api/swagger/health	GET	❌	Health check.

Note: This service only runs when ENVIRONMENT=dev. In production, it returns 404.

📍 Tracking Service (Port 8083)

Responsible for high-volume dynamic data.

Endpoint	Method	Description
/api/positions	GET	Get latest positions or history (params: from, to).
/api/events	GET	List system events (overspeed, geofence enter/exit).
/api/events/{id}	GET	Get details of a specific event.
/api/geofences	GET	List geofences.
/api/geofences	POST	Create a geofence (Circle/Polygon).
/api/geofences/{id}	PUT	Update geofence area.
/api/geofences/{id}	DELETE	Delete geofence.

⚙️ Command Service (Port 8084)

Responsible for sending instructions to devices.

Endpoint	Method	Description
/api/commands	GET	List saved command templates.
/api/commands	POST	Create a saved command template.
/api/commands/send	POST	Send a command immediately to a device.
/api/commands/types	GET	Get supported command types for a specific device model.

🔔 Notification Service (Port 8086)

Responsible for alerting users and real-time push.

Endpoint	Method	Description
/api/notifications	GET	List notification rules.
/api/notifications	POST	Create a notification rule (link event -> channel).
/api/notifications/test	POST	[NEW] Send a test notification.
/api/notifications/test/{notificator}	POST	[NEW] Test a specific channel (e.g., 'web', 'mail').
/api/notifications/types	GET	[NEW] List available notification types.
/api/notifications/notificators	GET	[NEW] List available notificator backends.
/api/socket	GET	[NEW] WebSocket endpoint for real-time updates.
/api/push-token	POST	Register a Firebase (FCM) token for mobile push.
/api/push-token	DELETE	Unregister a push token.

---

5. 💾 Data Forwarding & Integration Layer

The Java backend supports multiple data forwarding mechanisms to integrate with external systems.

Position Forwarding (Outbound)

Configuration: forward.url and forward.type in traccar.xml

Type	Description	Implementation
URL (default)	HTTP POST to custom endpoint	PositionForwarderUrl.java
JSON	HTTP POST with JSON payload + device enrichment	PositionForwarderJson.java
AMQP	RabbitMQ / AMQP exchange publish	PositionForwarderAmqp.java
Kafka	Apache Kafka topic publish	PositionForwarderKafka.java
MQTT	MQTT broker publish	PositionForwarderMqtt.java
Redis	Redis Pub/Sub channel	PositionForwarderRedis.java
Wialon	Wialon Hosting IPS protocol	PositionForwarderWialon.java

Data Flow:

Device -> Java Gateway -> Position Saved to MySQL
                       -> Position Forwarded via Selected Protocol

Event Forwarding (Outbound)

Configuration: event.forward.url and event.forward.type in traccar.xml

Type	Description	Implementation
JSON (default)	HTTP POST with JSON payload	EventForwarderJson.java
AMQP	RabbitMQ / AMQP exchange publish	EventForwarderAmqp.java
Kafka	Apache Kafka topic publish	EventForwarderKafka.java
MQTT	MQTT broker publish	EventForwarderMqtt.java

Data Flow:

Device Event (overspeed, geofence, alarm, etc.)
  -> Event Saved to MySQL
  -> Event Forwarded via Selected Protocol

Current Configuration (traccar.xml):

<entry key='event.forward.enable'>false</entry>
<entry key='event.forward.url'>http://events:8083/events</entry>
<entry key='event.forward.header'>API-KEY: AIzaSyCnk5s9MO9tVpTUiAY9be_qd0ZTB_jOT0I</entry>

Broadcast Service (Internal Synchronization)

Configuration: broadcast.type in traccar.xml

Type	Description	Implementation
Multicast	UDP multicast for same-network cluster	MulticastBroadcastService.java
Redis	Redis Pub/Sub for distributed cluster	RedisBroadcastService.java
Null (default)	No broadcasting (single instance)	NullBroadcastService.java

Purpose: Synchronize cache updates across multiple backend instances in a cluster.

---

6. 🔔 Notificators & External Services

The Java backend forwards notifications to external microservices for processing.

Available Notificator Channels

Configuration: notificator.types in traccar.xml

Current active channels:

<entry key='notificator.types'>firebase,web,mailExternal,command,phone</entry>

Channel	Description	Implementation	External Endpoint
firebase	Firebase Cloud Messaging (FCM) push	NotificatorFirebase.java	Firebase API
web	WebSocket real-time notifications	NotificatorWeb.java	Internal (ConnectionManager)
mailExternal	External email service (microservice)	NotificatorEmailExternal.java	http://events:8083/emails
command	Send SMS command to device	NotificatorCommand.java	Internal
phone	External phone call service (microservice)	NotificatorPhone.java	http://events:8083/calls
mail	Internal SMTP email	NotificatorMail.java	SMTP server
sms	SMS gateway	NotificatorSms.java	SMS provider
telegram	Telegram bot	NotificatorTelegram.java	Telegram API
traccar	Traccar cloud service	NotificatorTraccar.java	Traccar cloud
pushover	Pushover service	NotificatorPushover.java	Pushover API

External Microservice Integrations

A. Phone Call Service (NotificatorPhone)

Endpoint: http://events:8083/calls Method: POST Authentication: API-KEY: AIzaSyCnk5s9MO9tVpTUiAY9be_qd0ZTB_jOT0I

Payload Format:

{
  "phone": "+393931952336",
  "userId": 123,
  "deviceId": 456,
  "eventId": 789,
  "eventType": "alarm",
  "alarmType": "sos",
  "title": "SOS Alert",
  "body": "Device ABC triggered SOS alarm",
  "deviceName": "Vehicle ABC",
  "position": {
    "latitude": 41.9028,
    "longitude": 12.4964,
    "accuracy": 10.0
  },
  "address": "Via Roma, 123, Roma, Italia"
}

Phone Number Sources:

• device.attributes.alarmPhones (CSV)
• device.attributes.alarmPhone1
• device.attributes.alarmPhone2
• device.attributes.alarmPhone3

Test Configuration:

<entry key='notificator.phone.testNumber'>3931952336</entry>

B. External Email Service (NotificatorEmailExternal)

Endpoint: http://events:8083/emails Method: POST Authentication: API-KEY: AIzaSyCnk5s9MO9tVpTUiAY9be_qd0ZTB_jOT0I

Payload Format:

{
  "to": "user@example.com",
  "subject": "Overspeed Alert",
  "text": "Device ABC exceeded speed limit",
  "userId": 123,
  "deviceId": 456,
  "eventId": 789,
  "eventType": "deviceOverspeed",
  "alarmType": null,
  "deviceName": "Vehicle ABC",
  "position": {
    "latitude": 41.9028,
    "longitude": 12.4964,
    "accuracy": 10.0
  },
  "address": "Via Roma, 123, Roma, Italia"
}

Email Recipient Sources:

• device.attributes.alarmEmails (CSV)
• device.attributes.alarmEmail1
• device.attributes.alarmEmail2
• device.attributes.alarmEmail3
• Fallback: user.email

Test Configuration:

<entry key='notificator.mailExternal.testEmail'>vislasrls@gmail.com</entry>

C. Log Forwarding (ELK Stack)

Endpoint: http://logstash:8080/ Method: POST Purpose: Forward client-side logs to Logstash for centralized logging

Configuration:

<entry key='logs.forward.url'>http://logstash:8080/</entry>

Triggered by: /api/logs endpoint

---

7. 💾 Database Schema

All services share a single MySQL instance but own distinct tables.

Table Ownership

Service	Tables
Auth	tc_users, tc_keystore, tc_refresh_tokens
device	devices, groups, drivers, calendars, maintenances, orders
Tracking	positions, events, geofences
Notification	notifications, notification_rules
Config	server, attributes, computed_attributes
Java Gateway	Writes to positions, events (ReadOnly for others)

Cache & Messaging (Redis)

Usage:

• Caching: User sessions, device state, config
• Pub/Sub: Real-time message bus for inter-service communication
  • Channel: "positions" - Position updates from devices
  • Channel: "devices" - Device metadata updates
  • Channel: "events" - System events

---

8. 🔍 Complete Data Flow Summary

Data Flow Diagram

Complete Data Flows

1️⃣ Device Position Update (Real-time)

GPS Device 
  → TCP/UDP Port (5000+) 
  → Protocol Decoder (Java) 
  → Position Handler 
  → [PARALLEL]
      ├─ MySQL (positions table)
      ├─ Redis Pub/Sub ("positions" channel)
      ├─ Position Forwarder (if configured)
      │    └─ External System (HTTP/Kafka/AMQP/MQTT/Redis/Wialon)
      └─ Event Generator (if rules triggered)
           ├─ MySQL (events table)
           ├─ Redis Pub/Sub ("events" channel)
           ├─ Event Forwarder (if configured)
           │    └─ External System
           └─ Notificators (if rules matched)
                ├─ WebSocket → Client
                ├─ Firebase FCM → Mobile App
                ├─ HTTP → Microservice (phone, email)
                └─ SMTP → Email

2️⃣ User Login & Authentication

Client 
  → POST /api/auth/login (email, password)
  → Auth Service :8081
  → LDAP (if configured) OR MySQL users table
  → JWT Access Token Created (15-min expiration)
  → JWT Refresh Token Created (30-day, stored in DB)
  → Response ({ access_token, refresh_token, user })
  → Client stores both tokens
  
Client (authenticated)
  → GET /api/devices (Authorization: Bearer <access_token>)
  → Nginx Gateway
  → Internal: GET /api/auth/validate (validates JWT)
  → Auth Service → Returns 200 OK + userId from JWT
  → Nginx → Forward to device Service :8082
  → device Service → MySQL devices table
  → Response to Client

Client (after 15 minutes, access token expired)
  → POST /api/auth/refresh (refresh_token)
  → Auth Service :8081
  → Validate refresh token from DB
  → New access token created
  → Response ({ access_token })

3️⃣ WebSocket Real-time Updates

Client 
  → WSS /api/socket (with JWT access token in Authorization header)
  → Nginx Gateway :443
  → Internal: GET /api/auth/validate (validates JWT)
  → Auth Service → Returns 200 OK (userId extracted from JWT)
  → Nginx → Upgrades to WebSocket connection to :8086
  → Notification Service establishes persistent connection
  → ConnectionManager.addListener(userId)
  
[Device sends position]
  → ... (flow #1) ...
  → Redis Pub/Sub ("positions")
  → Notification Service (subscribed)
  → ConnectionManager.onUpdatePosition()
  → WebSocket → Client receives JSON (real-time)

4️⃣ Device Command Send

Client 
  → POST /api/commands/send
  → Command Service :8084
  → Device Command Queue
  → Device Gateway
  → TCP/UDP Connection to Device
  → Device executes command
  → Device sends ACK (optional)
  → Position update with command result

5️⃣ Notification Flow (Alarm)

Device sends alarm position
  → Protocol Decoder
  → Position Handler
  → Event Generator detects alarm
  → MySQL events table
  → Notification Rules Matcher
  → User has notification rule: alarm → phone
  → NotificatorPhone.send()
  → HTTP POST http://events:8083/calls
  → External Microservice
  → Phone call initiated

6️⃣ OpenID Authentication (Google/Facebook/Apple)

Client 
  → GET /api/auth/openid/auth?provider=google
  → OpenIdProvider.getAuthUrl()
  → Redirect to https://accounts.google.com/...
  
User authenticates with Google
  → Google redirects to /api/auth/openid/callback?code=...
  → OpenIdProvider.processCallback()
  → Exchange code for access_token
  → Fetch user info from Google
  → Create or update user in MySQL
  → Generate JWT access token + refresh token
  → Redirect to frontend with tokens (URL params or POST message)
  
Mobile App (native SDK)
  → Google SDK → id_token
  → POST /api/auth/openid/token (id_token)
  → Validate Google token
  → Create/find user in MySQL
  → Generate JWT access token + refresh token
  → Response ({ access_token, refresh_token, user })

7️⃣ Report Generation

Client 
  → GET /api/reports/trips?deviceId=123&from=...&to=...
  → Report Service :8085
  → MySQL positions table (range query)
  → Trip Detection Algorithm
  → Response: JSON array of trips
  
Client
  → GET /api/reports/route/xlsx?deviceId=123&from=...&to=...
  → Report Service :8085
  → Generate Excel file
  → Response: File download

---

9. ☕ Java Backend Internal Data Flow

This section details the internal lifecycle of a data packet within the Java monolith (backend.vislagps.com), from the socket ingress to the database and Redis.

A. Netty Pipeline (Ingestion)

Every device protocol (e.g., Teltonika, GT06) runs on a dedicated TCP/UDP port using Netty.

Pipeline Stages:

1. Frame Decoder: Splits the TCP stream into discrete packets (e.g., by length, delimiter).
2. String Decoder (Optional): Converts bytes to string if the protocol is text-based.
3. Protocol Decoder: The core logic. Parses the raw bytes/string into a Position object.
   • Extracts: Latitude, Longitude, Speed, Course, Altitude, Battery, Ignition, etc.
   • Normalizes: Converts vendor-specific status codes into standard attributes.
4. Position Handler: The main business logic processor.

B. Position Handler Logic

Once a Position object is created, it passes through PositionHandler (and other handlers like FilterHandler, GeocoderHandler).

Processing Steps:

1. Filtering: Checks if the position should be discarded (e.g., invalid GPS, duplicate, zero speed).
   • Config: filter.enable, filter.invalid, filter.duplicate
2. Geocoding: If enabled, fetches address from external provider (Google, LocationIQ).
   • Config: geocoder.enable
3. Computed Attributes: Calculates virtual attributes based on expressions (e.g., fuel = adc1 * 0.5).
   • Config: processing.computedAttributes.enable
4. Database Persistence: Saves the position to the tc_positions table in MySQL.
   • Update: Updates the tc_devices table with the latest position ID and status.

C. Data Forwarding & Broadcasting

After persistence, the position is broadcasted to other systems.

1. Redis Pub/Sub (Real-time)

   • Channel: positions
   • Payload: JSON representation of the Position object.
   • Purpose: Consumed by the Notification Service to push updates to WebSockets.
   • Implementation: PositionForwarderRedis (configured to use publish).

2. Event Generation

   • Checks for events: Overspeed, Geofence Entry/Exit, Maintenance.
   • If Event Detected:
     • Saved to tc_events table.
     • Published to Redis channel events.
     • Forwarded to configured Notificators (Firebase, Web, Email, Phone).

D. Java Internal Diagram

---

📝 Missing Documentation Checklist

✅ Documented:

• All REST API endpoints (22 resources)
• WebSocket protocol (/api/socket)
• TCP/UDP device protocols (16 protocols, ports 5000+)
• Position forwarding (7 types: URL, JSON, AMQP, Kafka, MQTT, Redis, Wialon)
• Event forwarding (4 types: JSON, AMQP, Kafka, MQTT)
• Notificators (10 channels: firebase, web, mailExternal, phone, command, mail, sms, telegram, traccar, pushover)
• External microservice integrations (phone, email, logs)
• Redis Pub/Sub channels
• Broadcast service (Multicast, Redis)
• Database schema ownership
• Authentication flows (standard, OpenID)
• Real-time data flows

✅ All data flows, APIs, protocols, and sockets are now comprehensively documented.

---

🔄 Migration Notes

When migrating from the monolithic Java backend to microservices:

1. Preserve all protocol decoders - These are critical and not easily replaceable
2. Maintain Redis Pub/Sub - Essential for real-time WebSocket updates
3. Keep position/event forwarding - External integrations depend on this
4. Migrate notificators carefully - Complex business logic for phone/email services
5. Document all custom device attributes - alarmPhones, alarmEmails, etc.
6. Test WebSocket protocol - Critical for dashboard real-time updates
7. Validate all API endpoints - 100+ endpoints across 7 services

---

📚 Related Documentation

• PORTS_E_MODELLI.md - Protocol ports and model mapping
• RAW_LOGGING.md - Device raw packet logging
• CUSTOM_PROTOCOL.md - Custom protocol implementation
• NOTIFICATOR_PHONE.md - Phone notificator details
• QR_CLAIM_FLOW.md - Device claiming workflow
• OPENID_FLOW.md - OpenID authentication flow

---

Last Updated: 2025-12-08
Status: ✅ Complete - All data flows documented