User Guide on DataMate

Data Collection

Mon, 01 Jan 0001 00:00:00 +0000

Data collection module helps you collect data from multiple data sources (databases, file systems, APIs, etc.) into the DataMate platform.

Features Overview

Based on DataX, data collection module supports:

Multiple Data Sources: MySQL, PostgreSQL, Oracle, SQL Server, etc.
Heterogeneous Sync: Data sync between different sources
Batch Collection: Large-scale batch collection and sync
Scheduled Tasks: Support scheduled execution
Task Monitoring: Real-time monitoring of collection tasks

Supported Data Sources

Data Source Type	Reader	Writer	Description
General Relational Databases	✅	✅	Supports MySQL, PostgreSQL, OpenGauss, SQL Server, DM, DB2
MySQL	✅	✅	Relational database
PostgreSQL	✅	✅	Relational database
OpenGauss	✅	✅	Relational database
SQL Server	✅	✅	Microsoft database
DM (Dameng)	✅	✅	Domestic database
DB2	✅	✅	IBM database
StarRocks	✅	✅	Analytical database
NAS	✅	✅	Network storage
S3	✅	✅	Object storage
GlusterFS	✅	✅	Distributed file system
API Collection	✅	✅	API interface data
JSON Files	✅	✅	JSON format files
CSV Files	✅	✅	CSV format files
TXT Files	✅	✅	Text files
FTP	✅	✅	FTP servers
HDFS	✅	✅	Hadoop HDFS

Quick Start

1. Create Collection Task

Step 1: Enter Data Collection Page

Select Data Collection in the left navigation.

Data Management

Mon, 01 Jan 0001 00:00:00 +0000

Data management module provides unified dataset management capabilities, supporting multiple data types for storage, query, and operations.

Features Overview

Data management module provides:

Multiple data types: Image, text, audio, video, and multimodal support
File management: Upload, download, preview, delete operations
Directory structure: Support for hierarchical directory organization
Tag management: Use tags to categorize and retrieve data
Statistics: Dataset size, file count, and other statistics

Dataset Types

Type	Description	Supported Formats
Image	Image data	JPG, PNG, GIF, BMP, WebP
Text	Text data	TXT, MD, JSON, CSV
Audio	Audio data	MP3, WAV, FLAC, AAC
Video	Video data	MP4, AVI, MOV, MKV
Multimodal	Multimodal data	Mixed formats

Quick Start

1. Create Dataset

Step 1: Enter Data Management Page

In the left navigation, select Data Management.

Data Cleaning

Mon, 01 Jan 0001 00:00:00 +0000

Data cleaning module provides powerful data processing capabilities to help you clean, transform, and optimize data quality.

Features Overview

Data cleaning module provides:

Built-in Cleaning Operators: Rich pre-cleaning operator library
Visual Configuration: Drag-and-drop cleaning pipeline design
Template Management: Save and reuse cleaning templates
Batch Processing: Support large-scale data batch cleaning
Real-time Preview: Preview cleaning results

Cleaning Operator Types

Data Quality Operators

Operator	Function	Applicable Data Types
Deduplication	Remove duplicates	All types
Null Handling	Handle null values	All types
Outlier Detection	Detect outliers	Numerical
Format Validation	Validate format	All types

Text Cleaning Operators

Operator	Function
Remove Special Chars	Remove special characters
Case Conversion	Convert case
Remove Stopwords	Remove common stopwords
Text Segmentation	Chinese word segmentation
HTML Tag Cleaning	Clean HTML tags

Quick Start

1. Create Cleaning Task

Step 1: Enter Data Cleaning Page

Select Data Processing in the left navigation.

Data Annotation

Mon, 01 Jan 0001 00:00:00 +0000

Data annotation module integrates Label Studio to provide professional-grade data annotation capabilities.

Features Overview

Data annotation module provides:

Multiple Annotation Types: Image, text, audio, etc.
Annotation Templates: Rich annotation templates and configurations
Quality Control: Annotation review and consistency checks
Team Collaboration: Multi-person collaborative annotation
Annotation Export: Export annotation results

Annotation Types

Image Annotation

Type	Description	Use Cases
Image Classification	Classify entire image	Scene recognition
Object Detection	Annotate object locations	Object recognition
Semantic Segmentation	Pixel-level classification	Medical imaging
Key Point Annotation	Annotate key points	Pose estimation

Text Annotation

Type	Description	Use Cases
Text Classification	Classify text	Sentiment analysis
Named Entity Recognition	Annotate entities	Information extraction
Text Summarization	Generate summaries	Document understanding

Quick Start

1. Deploy Label Studio

make install-label-studio

Access: http://localhost:30001

Data Synthesis

Mon, 01 Jan 0001 00:00:00 +0000

Data synthesis module leverages large model capabilities to automatically generate high-quality training data, reducing data collection costs.

Features Overview

Data synthesis module provides:

Instruction template management: Create and manage synthesis instruction templates
Single task synthesis: Create individual synthesis tasks
Proportional synthesis task: Synthesize multi-category balanced data by specified ratios
Large model integration: Support for multiple LLM APIs
Quality evaluation: Automatic evaluation of synthesized data quality

Quick Start

1. Create Instruction Template

Step 1: Enter Data Synthesis Page

In the left navigation, select Data Synthesis → Synthesis Tasks.

Data Evaluation

Mon, 01 Jan 0001 00:00:00 +0000

Data evaluation module provides multi-dimensional data quality evaluation capabilities.

Features Overview

Data evaluation module provides:

Quality Metrics: Rich data quality evaluation metrics
Automatic Evaluation: Auto-execute evaluation tasks
Manual Evaluation: Manual sampling evaluation
Evaluation Reports: Generate detailed reports
Quality Tracking: Track data quality trends

Evaluation Dimensions

Data Completeness

Metric	Description	Calculation
Null Rate	Null value ratio	Null count / Total count
Missing Field Rate	Required field missing rate	Missing fields / Total fields
Record Complete Rate	Complete record ratio	Complete records / Total records

Data Accuracy

Metric	Description	Calculation
Format Correct Rate	Format compliance	Format correct / Total
Value Range Compliance	In valid range	In range / Total
Consistency Rate	Data consistency	Consistent records / Total

Quick Start

1. Create Evaluation Task

Step 1: Enter Data Evaluation Page

Select Data Evaluation in the left navigation.

Knowledge Base Management

Mon, 01 Jan 0001 00:00:00 +0000

Knowledge base management module helps you build enterprise knowledge bases for efficient vector retrieval and RAG applications.

Features Overview

Knowledge base management module provides:

Document upload: Support multiple document formats
Text chunking: Intelligent text splitting strategies
Vectorization: Automatic text-to-vector conversion
Vector search: Semantic similarity-based retrieval
Knowledge base Q&A: RAG-intelligent Q&A

Supported Document Formats

Format	Description	Recommended For
TXT	Plain text	General text
PDF	PDF documents	Documents, reports
Markdown	Markdown files	Technical docs
JSON	JSON data	Structured data
CSV	CSV tables	Tabular data
DOCX	Word documents	Office documents

Quick Start

1. Create Knowledge Base

Step 1: Enter Knowledge Base Page

In the left navigation, select Knowledge Generation.

Operator Market

Mon, 01 Jan 0001 00:00:00 +0000

Operator marketplace provides rich data processing operators and supports custom operator development.

Features Overview

Operator marketplace provides:

Built-in Operators: Rich built-in data processing operators
Operator Publishing: Publish and share custom operators
Operator Installation: Install third-party operators
Custom Development: Develop custom operators

Built-in Operators

Data Cleaning Operators

Operator	Function	Input	Output
Deduplication	Remove duplicates	Dataset	Deduplicated data
Null Handler	Handle nulls	Dataset	Filled data
Format Converter	Convert format	Original format	New format

Text Processing Operators

Operator	Function
Text Segmentation	Chinese word segmentation
Remove Stopwords	Remove common stopwords
Text Cleaning	Clean special characters

Quick Start

1. Browse Operators

Step 1: Enter Operator Market

Select Operator Market in the left navigation.

Pipeline Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

Pipeline orchestration module provides drag-and-drop visual interface for designing and managing complex data processing workflows.

Features Overview

Pipeline orchestration provides:

Visual Designer: Drag-and-drop workflow design
Rich Node Types: Data processing, conditions, loops, etc.
Flow Execution: Auto-execute and monitor workflows
Template Management: Save and reuse flow templates
Version Management: Flow version control

Node Types

Data Nodes

Node	Function	Config
Input Dataset	Read from dataset	Select dataset
Output Dataset	Write to dataset	Select dataset
Data Collection	Execute collection task	Select task
Data Cleaning	Execute cleaning task	Select task
Data Synthesis	Execute synthesis task	Select task

Logic Nodes

Node	Function	Config
Condition Branch	Execute different branches	Condition expression
Loop	Repeat execution	Loop count/condition
Parallel	Execute multiple branches in parallel	Branch count
Wait	Wait for specified time	Duration

Quick Start

1. Create Pipeline

Step 1: Enter Pipeline Orchestration Page

Select Pipeline Orchestration in left navigation.

Agent Chat

Mon, 01 Jan 0001 00:00:00 +0000

Agent chat module integrates large language models to provide intelligent Q&A and knowledge base retrieval capabilities.

Features Overview

Agent chat module provides:

Intelligent Chat: Natural language conversation based on LLMs
Knowledge Base Q&A: RAG-integrated knowledge base Q&A
Multi-turn Dialogue: Support context-aware multi-turn conversation
Dialogue Management: Conversation history and management
Quick Commands: Quick commands for common operations

Quick Start

1. Access Agent

Select Agent Chat in left navigation, or visit: