This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Appendix

Configuration, troubleshooting, and other reference information

Appendix contains configuration parameters, troubleshooting, and other reference information.

Appendix Content

Configuration

Detailed system configuration documentation:

  • Environment Variables: All configurable environment variables
  • application.yml: Spring Boot configuration file
  • Docker Compose: Container configuration
  • Kubernetes: K8s configuration

Troubleshooting

Common issue troubleshooting steps and solutions:

  • Service startup issues: Container startup failures
  • Database connection issues: Database connection failures
  • Frontend issues: Page loading, API requests
  • Task execution issues: Tasks stuck, execution failures
  • Performance issues: Slow response, memory overflow

Other References

Technical Support

If you encounter issues:

  1. Check Troubleshooting documentation
  2. Search GitHub Issues
  3. Submit a new issue with detailed information

Contributing

Contributions to DataMate are welcome:

  • Report bugs
  • Propose new features
  • Submit code contributions
  • Improve documentation

See Contribution Guide for details.

1 - Configuration

DataMate system configuration parameters

This document details various configuration parameters of the DataMate system.

Environment Variables

Common Configuration

VariableDefaultDescription
DB_PASSWORDpasswordDatabase password
DATAMATE_JWT_ENABLEfalseEnable JWT authentication
REGISTRYghcr.io/modelengine-group/Image registry
VERSIONlatestImage version tag

Database Configuration

VariableDefaultDescription
DB_HOSTdatamate-databaseDatabase host
DB_PORT5432Database port
DB_NAMEdatamateDatabase name
DB_USERpostgresDatabase username
DB_PASSWORDpasswordDatabase password

Redis Configuration

VariableDefaultDescription
REDIS_HOSTdatamate-redisRedis host
REDIS_PORT6379Redis port
REDIS_PASSWORD-Redis password (optional)
REDIS_DB0Redis database number

Milvus Configuration

VariableDefaultDescription
MILVUS_HOSTmilvusMilvus host
MILVUS_PORT19530Milvus port
MILVUS_INDEX_TYPEIVF_FLATVector index type
MILVUS_EMBEDDING_DIM768Vector dimension

MinIO Configuration

VariableDefaultDescription
MINIO_ENDPOINTminio:9000MinIO endpoint
MINIO_ACCESS_KEYminioadminAccess key
MINIO_SECRET_KEYminioadminSecret key
MINIO_BUCKETdatamateBucket name

LLM Configuration

VariableDefaultDescription
OPENAI_API_KEY-OpenAI API key
OPENAI_BASE_URLhttps://api.openai.com/v1API base URL
OPENAI_MODELgpt-4Model to use

JWT Configuration

VariableDefaultDescription
JWT_SECRETdefault-insecure-keyJWT secret (CHANGE IN PRODUCTION)
JWT_EXPIRATION86400Token expiration (seconds)

Logging Configuration

VariableDefaultDescription
LOG_LEVELINFOLog level
LOG_PATH/var/log/datamateLog path

application.yml Configuration

Main Config

datamate:
  jwt:
    enable: ${DATAMATE_JWT_ENABLE:false}
    secret: ${JWT_SECRET:default-insecure-key}
    expiration: ${JWT_EXPIRATION:86400}

  storage:
    type: minio
    endpoint: ${MINIO_ENDPOINT:minio:9000}
    access-key: ${MINIO_ACCESS_KEY:minioadmin}
    secret-key: ${MINIO_SECRET_KEY:minioadmin}

Spring Boot Config

spring:
  datasource:
    url: jdbc:postgresql://${DB_HOST:datamate-database}:${DB_PORT:5432}/${DB_NAME:datamate}
    username: ${DB_USER:postgres}
    password: ${DB_PASSWORD:password}

  jpa:
    hibernate:
      ddl-auto: validate
    show-sql: false

server:
  port: 8092

Docker Compose Configuration

Environment Variables

services:
  datamate-backend:
    environment:
      - DB_PASSWORD=${DB_PASSWORD:-password}
      - LOG_LEVEL=${LOG_LEVEL:-INFO}

Resource Limits

services:
  datamate-backend:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

Kubernetes Configuration

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: datamate-config
data:
  LOG_LEVEL: "INFO"

Secret

apiVersion: v1
kind: Secret
metadata:
  name: datamate-secret
type: Opaque
data:
  DB_PASSWORD: cGFzc3dvcmQ=  # base64 encoded

Performance Tuning

Database Connection Pool

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000

JVM Parameters

JAVA_OPTS="-Xms2g -Xmx4g -XX:+UseG1GC"

2 - Troubleshooting

Common issues and solutions for DataMate

This document provides troubleshooting steps and solutions for common DataMate issues.

Service Startup Issues

Service Won’t Start

Symptoms

Service fails to start or exits immediately after running make install.

Troubleshooting Steps

  1. Check Port Conflicts
# Check port usage
lsof -i :8080  # API Gateway
lsof -i :30000 # Frontend

If port is occupied:

# Kill process
kill -9 <PID>
  1. View Container Logs
# View all containers
docker ps -a

# View specific container logs
docker logs datamate-backend
  1. Check Docker Resources
# View Docker system info
docker system df

# Clean unused resources
docker system prune -a

Common Causes and Solutions

CauseSolution
Port occupiedKill process or modify port mapping
Insufficient memoryIncrease Docker memory limit
Image not pulledRun docker pull
Network issuesCheck firewall and network config

Container Exits Immediately

Troubleshooting

# View exit code
docker ps -a

# View detailed logs
docker logs <container-name> --tail 100

Database Connection Issues

Cannot Connect to Database

Troubleshooting Steps

  1. Check Database Container
docker ps | grep datamate-database
docker logs datamate-database
  1. Test Database Connection
# Enter database container
docker exec -it datamate-database psql -U postgres -d datamate
  1. Check Database Config
# Check environment variables
docker exec datamate-backend env | grep DB_

Frontend Issues

Frontend Not Accessible

Symptoms

Browser cannot access http://localhost:30000

Troubleshooting

  1. Check Frontend Container
docker ps | grep datamate-frontend
docker logs datamate-frontend
  1. Check Port Mapping
docker port datamate-frontend

API Request Failed

Troubleshooting

  1. Check Browser Console

Open browser DevTools → Network tab

  1. Check API Gateway
docker ps | grep datamate-gateway
docker logs datamate-gateway
  1. Test API
curl http://localhost:8080/actuator/health

Task Execution Issues

Task Stuck

Troubleshooting

  1. View Task Logs
docker logs datamate-backend --tail 100 | grep <task-id>
docker logs datamate-runtime --tail 100
  1. Check System Resources
docker stats

Performance Issues

Slow System Response

Troubleshooting

  1. Check System Resources
docker stats
  1. Check Database Performance
-- View active queries
SELECT * FROM pg_stat_activity WHERE state = 'active';

Memory Overflow

Troubleshooting

# Check exit reason
docker inspect <container> | grep OOMKilled

Log Viewing

View Application Logs

# Backend logs
docker logs datamate-backend --tail 100 -f

# Frontend logs
docker logs datamate-frontend --tail 100 -f

Log File Locations

ServiceLog Path
Backend/var/log/datamate/backend/app.log
Frontend/var/log/datamate/frontend/
Database/var/log/datamate/database/
Runtime/var/log/datamate/runtime/

Getting Help

If issues persist:

  1. Collect Information

    • Error messages
    • Log files
    • System environment
    • Reproduction steps
  2. Search Existing Issues

Visit GitHub Issues

  1. Submit New Issue

Include:

  • DataMate version
  • OS version
  • Docker version
  • Detailed error messages
  • Reproduction steps