Developer Guide

• 1: Backend Architecture
• 2: Frontend Architecture

DataMate/
├── backend/                 # Java backend
│   ├── services/           # Microservice modules
│   ├── openapi/            # OpenAPI specs
│   └── scripts/            # Build scripts
├── frontend/               # React frontend
│   ├── src/
│   │   ├── components/    # Common components
│   │   ├── pages/         # Page components
│   │   ├── services/      # API services
│   │   └── store/         # Redux store
│   └── package.json
├── runtime/                # Python runtime
│   └── datamate/          # DataMate runtime
└── deployment/             # Deployment config
    ├── docker/            # Docker config
    └── helm/              # Helm Charts

git clone https://github.com/ModelEngine-Group/DataMate.git
cd DataMate

# Start basic services
make install

# Access frontend
open http://localhost:30000

# Backend development
cd backend/services/main-application
mvn spring-boot:run

# Frontend development
cd frontend
pnpm dev

# Python service development
cd runtime/datamate
python operator_runtime.py --port 8081

# Backend tests
cd backend
mvn test

# Frontend tests
cd frontend
pnpm test

# Python tests
cd runtime
pytest

# Start test environment
make test-env-up

# Run integration tests
make integration-test

# Clean test environment
make test-env-down

1 - Backend Architecture

Architecture Overview

DataMate backend uses microservices architecture, splitting into multiple independent services:

┌─────────────────────────────────────────────┐
│              API Gateway                    │
│         (Spring Cloud Gateway)              │
│              Port: 8080                     │
└──────────────┬──────────────────────────────┘
               │
       ┌───────┴───────┬───────────────┐
       ▼               ▼               ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│   Main       │ │  Data        │ │  Data        │
│ Application  │ │  Management  │ │  Collection  │
└──────────────┘ └──────────────┘ └──────────────┘
       │               │               │
       └───────────────┴───────────────┘
                       │
                       ▼
              ┌────────────────┐
              │   PostgreSQL   │
              │   Port: 5432   │
              └────────────────┘

Tech Stack

Core Frameworks

Support Components

Microservices List

API Gateway

Port: 8080

Functions:

• Unified entry point
• Route forwarding
• Authentication and authorization
• Rate limiting and circuit breaking

Tech: Spring Cloud Gateway, JWT authentication

Main Application

Functions:

• User management
• Permission management
• System configuration
• Task scheduling

Data Management Service

Port: 8092

Functions:

• Dataset management
• File management
• Tag management
• Statistics

API Endpoints:

• /data-management/datasets - Dataset management
• /data-management/datasets/{id}/files - File management

Runtime Service

Port: 8081

Functions:

• Operator execution
• Ray integration
• Task scheduling

Tech: Python + Ray, FastAPI

Database Design

Main Tables

users (User Table)

datasets (Dataset Table)

Service Communication

Synchronous Communication

Services communicate via HTTP/REST:

// Using Feign Client
@FeignClient(name = "data-management-service")
public interface DataManagementClient {
    @GetMapping("/data-management/datasets/{id}")
    DatasetResponse getDataset(@PathVariable String id);
}

Asynchronous Communication

Using Redis for async messaging:

// Send message
redisTemplate.convertAndSend("task.created", taskMessage);

// Receive message
@RedisListener(topic = "task.created")
public void handleTaskCreated(TaskMessage message) {
    // Handle task creation event
}

Authentication & Authorization

JWT Authentication

@Configuration
public class JwtConfig {
    @Value("${datamate.jwt.secret}")
    private String secret;

    @Value("${datamate.jwt.expiration}")
    private Long expiration;
}

RBAC

@PreAuthorize("hasRole('ADMIN')")
public void adminOperation() {
    // Admin operations
}

Performance Optimization

Database Connection Pool

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000

Caching Strategy

@Cacheable(value = "datasets", key = "#id")
public Dataset getDataset(String id) {
    return datasetRepository.findById(id);
}

Related Documentation

• Frontend Architecture - Frontend architecture
• Database Schema - Database tables
• Development Setup - Dev environment

2 - Frontend Architecture

Architecture Overview

DataMate frontend adopts SPA architecture:

┌─────────────────────────────────────────────┐
│              Browser                        │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│              React App                      │
│  ┌──────────────────────────────────────┐  │
│  │         Components                   │  │
│  └──────────────────────────────────────┘  │
│  ┌──────────────────────────────────────┐  │
│  │         State Management             │  │
│  │         (Redux Toolkit)              │  │
│  └──────────────────────────────────────┘  │
│  ┌──────────────────────────────────────┐  │
│  │         Services (API)               │  │
│  └──────────────────────────────────────┘  │
│  ┌──────────────────────────────────────┐  │
│  │         Routing                      │  │
│  └──────────────────────────────────────┘  │
└─────────────────────────────────────────────┘

Tech Stack

Core Frameworks

State Management

Project Structure

frontend/
├── src/
│   ├── components/     # Common components
│   ├── pages/          # Page components
│   ├── services/       # API services
│   ├── store/          # Redux store
│   ├── hooks/          # Custom hooks
│   ├── routes/         # Routes config
│   └── main.tsx        # Entry point

Routing Design

const router = createBrowserRouter([
  { path: "/", Component: Home },
  { path: "/chat", Component: AgentPage },
  {
    path: "/data",
    Component: MainLayout,
    children: [
      {
        path: "management",
        Component: DatasetManagement
      }
    ]
  }
]);

State Management

Redux Toolkit Configuration

export const store = configureStore({
  reducer: {
    dataManagement: dataManagementSlice,
    user: userSlice,
  },
});

Slice Example

export const fetchDatasets = createAsyncThunk(
  'dataManagement/fetchDatasets',
  async (params: GetDatasetsParams) => {
    const response = await getDatasets(params);
    return response.data;
  }
);

Component Design

Page Component

export const DataManagement: React.FC = () => {
  const dispatch = useAppDispatch();
  const { datasets, loading } = useAppSelector(
    (state) => state.dataManagement
  );

  useEffect(() => {
    dispatch(fetchDatasets({ page: 0, size: 20 }));
  }, [dispatch]);

  return (
    <div className="p-6">
      <h1>Data Management</h1>
      <DataTable data={datasets} loading={loading} />
    </div>
  );
};

API Services

Axios Configuration

const request = axios.create({
  baseURL: import.meta.env.VITE_API_BASE_URL,
  timeout: 30000,
});

// Request interceptor
request.interceptors.request.use((config) => {
  const token = localStorage.getItem('token');
  if (token) {
    config.headers.Authorization = `Bearer ${token}`;
  }
  return config;
});

Performance Optimization

Code Splitting

const DataManagement = lazy(() =>
  import('@/pages/DataManagement/Home/DataManagement')
);

React.memo

export const DataCard = React.memo<DataCardProps>(({ data }) => {
  return <div>{data.name}</div>;
});

Related Documentation

• Backend Architecture - Backend architecture
• Development Setup - Dev environment