Developer Guide

DataMate is an enterprise-level data processing platform using microservices architecture, supporting large-scale data processing and custom extensions.

Architecture Documentation

• Backend Architecture - Java microservices architecture design
• Frontend Architecture - React frontend architecture design

Development Guide

• Development Setup - Local development environment configuration
• Contribution Guide - Code contribution process
• Deployment Guide - Production environment deployment

Tech Stack

Frontend

Backend (Java)

Backend (Python)

Project Structure

DataMate/
├── backend/                 # Java backend
│   ├── services/           # Microservice modules
│   ├── openapi/            # OpenAPI specs
│   └── scripts/            # Build scripts
├── frontend/               # React frontend
│   ├── src/
│   │   ├── components/    # Common components
│   │   ├── pages/         # Page components
│   │   ├── services/      # API services
│   │   └── store/         # Redux store
│   └── package.json
├── runtime/                # Python runtime
│   └── datamate/          # DataMate runtime
└── deployment/             # Deployment config
    ├── docker/            # Docker config
    └── helm/              # Helm Charts

Quick Start

1. Clone Code

git clone https://github.com/ModelEngine-Group/DataMate.git
cd DataMate

2. Start Services

# Start basic services
make install

# Access frontend
open http://localhost:30000

3. Development Mode

# Backend development
cd backend/services/main-application
mvn spring-boot:run

# Frontend development
cd frontend
pnpm dev

# Python service development
cd runtime/datamate
python operator_runtime.py --port 8081

Core Concepts

Microservices Architecture

DataMate uses microservices architecture, each service handles specific business functions:

• API Gateway: Unified entry, routing, authentication
• Main Application: Core business logic
• Data Management Service: Dataset management
• Data Cleaning Service: Data cleaning
• Data Synthesis Service: Data synthesis
• Runtime Service: Operator execution

Operator System

Operators are basic units of data processing:

• Built-in operators: Common operators provided by platform
• Custom operators: User-developed custom operators
• Operator execution: Executed by Runtime Service

Pipeline Orchestration

Pipelines are implemented through visual orchestration:

• Nodes: Basic units of data processing
• Connections: Data flow between nodes
• Execution: Automatic execution according to workflow

Extension Development

Develop Custom Operators

Operator development guide:

1. Operator Market - Operator usage guide
2. Python operator development examples
3. Operator testing and debugging

Integrate External Systems

• API integration: Integration via REST API
• Webhook: Event notifications
• Plugin system: (Coming soon)

Testing

Unit Tests

# Backend tests
cd backend
mvn test

# Frontend tests
cd frontend
pnpm test

# Python tests
cd runtime
pytest

Integration Tests

# Start test environment
make test-env-up

# Run integration tests
make integration-test

# Clean test environment
make test-env-down

Performance Optimization

Backend Optimization

• Database connection pool configuration
• Query optimization
• Caching strategies
• Asynchronous processing

Frontend Optimization

• Code splitting
• Lazy loading
• Caching strategies

Security

Authentication and Authorization

• JWT authentication
• RBAC permission control
• API Key authentication

Data Security

• Transport encryption (HTTPS/TLS)
• Storage encryption
• Sensitive data masking

Related Resources

• GitHub Repository
• Issue Tracking
• OpenAPI Specs
• API Reference - Complete API documentation