Appendix
Configuration, troubleshooting, and other reference information
Appendix contains configuration parameters, troubleshooting, and other reference information.
Appendix Content
Configuration
Detailed system configuration documentation:
- Environment Variables: All configurable environment variables
- application.yml: Spring Boot configuration file
- Docker Compose: Container configuration
- Kubernetes: K8s configuration
Troubleshooting
Common issue troubleshooting steps and solutions:
- Service startup issues: Container startup failures
- Database connection issues: Database connection failures
- Frontend issues: Page loading, API requests
- Task execution issues: Tasks stuck, execution failures
- Performance issues: Slow response, memory overflow
Other References
Technical Support
If you encounter issues:
- Check Troubleshooting documentation
- Search GitHub Issues
- Submit a new issue with detailed information
Contributing
Contributions to DataMate are welcome:
- Report bugs
- Propose new features
- Submit code contributions
- Improve documentation
See Contribution Guide for details.
1 - Configuration
DataMate system configuration parameters
This document details various configuration parameters of the DataMate system.
Environment Variables
Common Configuration
| Variable | Default | Description |
|---|
DB_PASSWORD | password | Database password |
DATAMATE_JWT_ENABLE | false | Enable JWT authentication |
REGISTRY | ghcr.io/modelengine-group/ | Image registry |
VERSION | latest | Image version tag |
Database Configuration
| Variable | Default | Description |
|---|
DB_HOST | datamate-database | Database host |
DB_PORT | 5432 | Database port |
DB_NAME | datamate | Database name |
DB_USER | postgres | Database username |
DB_PASSWORD | password | Database password |
Redis Configuration
| Variable | Default | Description |
|---|
REDIS_HOST | datamate-redis | Redis host |
REDIS_PORT | 6379 | Redis port |
REDIS_PASSWORD | - | Redis password (optional) |
REDIS_DB | 0 | Redis database number |
Milvus Configuration
| Variable | Default | Description |
|---|
MILVUS_HOST | milvus | Milvus host |
MILVUS_PORT | 19530 | Milvus port |
MILVUS_INDEX_TYPE | IVF_FLAT | Vector index type |
MILVUS_EMBEDDING_DIM | 768 | Vector dimension |
MinIO Configuration
| Variable | Default | Description |
|---|
MINIO_ENDPOINT | minio:9000 | MinIO endpoint |
MINIO_ACCESS_KEY | minioadmin | Access key |
MINIO_SECRET_KEY | minioadmin | Secret key |
MINIO_BUCKET | datamate | Bucket name |
LLM Configuration
| Variable | Default | Description |
|---|
OPENAI_API_KEY | - | OpenAI API key |
OPENAI_BASE_URL | https://api.openai.com/v1 | API base URL |
OPENAI_MODEL | gpt-4 | Model to use |
JWT Configuration
| Variable | Default | Description |
|---|
JWT_SECRET | default-insecure-key | JWT secret (CHANGE IN PRODUCTION) |
JWT_EXPIRATION | 86400 | Token expiration (seconds) |
Logging Configuration
| Variable | Default | Description |
|---|
LOG_LEVEL | INFO | Log level |
LOG_PATH | /var/log/datamate | Log path |
application.yml Configuration
Main Config
datamate:
jwt:
enable: ${DATAMATE_JWT_ENABLE:false}
secret: ${JWT_SECRET:default-insecure-key}
expiration: ${JWT_EXPIRATION:86400}
storage:
type: minio
endpoint: ${MINIO_ENDPOINT:minio:9000}
access-key: ${MINIO_ACCESS_KEY:minioadmin}
secret-key: ${MINIO_SECRET_KEY:minioadmin}
Spring Boot Config
spring:
datasource:
url: jdbc:postgresql://${DB_HOST:datamate-database}:${DB_PORT:5432}/${DB_NAME:datamate}
username: ${DB_USER:postgres}
password: ${DB_PASSWORD:password}
jpa:
hibernate:
ddl-auto: validate
show-sql: false
server:
port: 8092
Docker Compose Configuration
Environment Variables
services:
datamate-backend:
environment:
- DB_PASSWORD=${DB_PASSWORD:-password}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
Resource Limits
services:
datamate-backend:
deploy:
resources:
limits:
cpus: '2'
memory: 4G
Kubernetes Configuration
ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: datamate-config
data:
LOG_LEVEL: "INFO"
Secret
apiVersion: v1
kind: Secret
metadata:
name: datamate-secret
type: Opaque
data:
DB_PASSWORD: cGFzc3dvcmQ= # base64 encoded
Database Connection Pool
spring:
datasource:
hikari:
maximum-pool-size: 20
minimum-idle: 5
connection-timeout: 30000
JVM Parameters
JAVA_OPTS="-Xms2g -Xmx4g -XX:+UseG1GC"
2 - Troubleshooting
Common issues and solutions for DataMate
This document provides troubleshooting steps and solutions for common DataMate issues.
Service Startup Issues
Service Won’t Start
Symptoms
Service fails to start or exits immediately after running make install.
Troubleshooting Steps
- Check Port Conflicts
# Check port usage
lsof -i :8080 # API Gateway
lsof -i :30000 # Frontend
If port is occupied:
# Kill process
kill -9 <PID>
- View Container Logs
# View all containers
docker ps -a
# View specific container logs
docker logs datamate-backend
- Check Docker Resources
# View Docker system info
docker system df
# Clean unused resources
docker system prune -a
Common Causes and Solutions
| Cause | Solution |
|---|
| Port occupied | Kill process or modify port mapping |
| Insufficient memory | Increase Docker memory limit |
| Image not pulled | Run docker pull |
| Network issues | Check firewall and network config |
Troubleshooting
# View exit code
docker ps -a
# View detailed logs
docker logs <container-name> --tail 100
Database Connection Issues
Cannot Connect to Database
Troubleshooting Steps
- Check Database Container
docker ps | grep datamate-database
docker logs datamate-database
- Test Database Connection
# Enter database container
docker exec -it datamate-database psql -U postgres -d datamate
- Check Database Config
# Check environment variables
docker exec datamate-backend env | grep DB_
Frontend Issues
Frontend Not Accessible
Symptoms
Browser cannot access http://localhost:30000
Troubleshooting
- Check Frontend Container
docker ps | grep datamate-frontend
docker logs datamate-frontend
- Check Port Mapping
docker port datamate-frontend
API Request Failed
Troubleshooting
- Check Browser Console
Open browser DevTools → Network tab
- Check API Gateway
docker ps | grep datamate-gateway
docker logs datamate-gateway
- Test API
curl http://localhost:8080/actuator/health
Task Execution Issues
Task Stuck
Troubleshooting
- View Task Logs
docker logs datamate-backend --tail 100 | grep <task-id>
docker logs datamate-runtime --tail 100
- Check System Resources
Slow System Response
Troubleshooting
- Check System Resources
- Check Database Performance
-- View active queries
SELECT * FROM pg_stat_activity WHERE state = 'active';
Memory Overflow
Troubleshooting
# Check exit reason
docker inspect <container> | grep OOMKilled
Log Viewing
View Application Logs
# Backend logs
docker logs datamate-backend --tail 100 -f
# Frontend logs
docker logs datamate-frontend --tail 100 -f
Log File Locations
| Service | Log Path |
|---|
| Backend | /var/log/datamate/backend/app.log |
| Frontend | /var/log/datamate/frontend/ |
| Database | /var/log/datamate/database/ |
| Runtime | /var/log/datamate/runtime/ |
Getting Help
If issues persist:
Collect Information
- Error messages
- Log files
- System environment
- Reproduction steps
Search Existing Issues
Visit GitHub Issues
- Submit New Issue
Include:
- DataMate version
- OS version
- Docker version
- Detailed error messages
- Reproduction steps