Merge commit 'd803bfe2b1fe7f5e219e50ac20d6801a0a58ac75' as 'vendor/ruvector'

This commit is contained in:
ruv
2026-02-28 14:39:40 -05:00
7854 changed files with 3522914 additions and 0 deletions

View File

@@ -0,0 +1,501 @@
# Agentic-Synth Examples - Progressive Tutorials
Complete, runnable tutorials for learning **agentic-synth** and **DSPy.ts** integration from beginner to advanced.
## 📚 Tutorial Structure
### 🟢 Beginner Level
Perfect for getting started with synthetic data generation and DSPy training.
### 🟡 Intermediate Level
Learn multi-model comparison, self-learning systems, and optimization.
### 🔴 Advanced Level
Build production-grade systems with custom learning and complete pipelines.
---
## 🚀 Quick Start
### Prerequisites
```bash
# Install dependencies
npm install dspy.ts @ruvector/agentic-synth
# Set up API keys
export GEMINI_API_KEY="your-gemini-api-key"
export ANTHROPIC_API_KEY="your-anthropic-key" # Optional, for multi-model
export OPENAI_API_KEY="your-openai-key" # Optional, for multi-model
```
### Running Tutorials
```bash
# From the package root
npx tsx examples/beginner/first-dspy-training.ts
npx tsx examples/intermediate/multi-model-comparison.ts
npx tsx examples/advanced/production-pipeline.ts
```
---
## 📖 Tutorial Catalog
### 🟢 Beginner Tutorials
#### 1. First DSPy Training (`beginner/first-dspy-training.ts`)
**Learn:** Basic DSPy.ts training with a single model
**Concepts:**
- Setting up DSPy language models
- Defining signatures for tasks
- Chain-of-Thought reasoning
- Simple evaluation metrics
- Training with examples
**Run:**
```bash
npx tsx examples/beginner/first-dspy-training.ts
```
**Output:**
```
🚀 Starting Your First DSPy Training Session
📊 Training with 3 examples...
✅ Training complete!
🧪 Testing the model with new products:
📦 Product: Smart Watch Pro
Quality Score: 85%
✅ Excellent
```
**What You'll Build:** A product description generator that learns from examples
---
#### 2. Simple Data Generation (`beginner/simple-data-generation.ts`)
**Learn:** Generate structured synthetic data with schemas
**Concepts:**
- Defining data schemas
- Structured data generation
- Working with different formats (JSON, CSV)
- Saving output to files
- Using constraints for realistic data
**Run:**
```bash
npx tsx examples/beginner/simple-data-generation.ts
```
**Output:**
```
🎯 Simple Data Generation Tutorial
📊 Generating 5 sample users...
✅ Generation Complete!
Generated 5 users in 1234ms
👥 Generated Users:
1. John Smith (admin)
📧 john.smith@example.com
🎂 Age: 34
🏠 San Francisco, USA
💾 Data saved to: examples/output/sample-users.json
```
**What You'll Build:** A user data generator for testing and prototyping
---
### 🟡 Intermediate Tutorials
#### 3. Multi-Model Comparison (`intermediate/multi-model-comparison.ts`)
**Learn:** Compare multiple AI models to find the best performer
**Concepts:**
- Running parallel model benchmarks
- Quality scoring across models
- Performance and speed metrics
- Cost tracking and optimization
- Selecting models for production
**Run:**
```bash
npx tsx examples/intermediate/multi-model-comparison.ts
```
**Output:**
```
🏆 Multi-Model Comparison Benchmark
📊 BENCHMARK RESULTS
┌─────────────────────┬──────────┬──────────┬──────────┬──────────┐
│ Model │ Quality │ Speed │ Cost │ Success │
├─────────────────────┼──────────┼──────────┼──────────┼──────────┤
│ 🥇 GPT-4 Turbo │ 94.5% │ 892ms │ $0.0023 │ 100% │
│ 🥈 Gemini Flash │ 89.2% │ 423ms │ $0.0004 │ 100% │
│ 🥉 Claude Sonnet 4 │ 91.8% │ 654ms │ $0.0012 │ 100% │
└─────────────────────┴──────────┴──────────┴──────────┴──────────┘
🎯 WINNER: GPT-4 Turbo
💡 RECOMMENDATIONS:
⚡ Fastest: Gemini Flash (423ms avg)
💰 Cheapest: Gemini Flash ($0.0004 total)
🎯 Most Reliable: All models (100% success)
```
**What You'll Build:** A comprehensive model benchmarking system
---
#### 4. Self-Learning System (`intermediate/self-learning-system.ts`)
**Learn:** Build AI systems that improve over time through feedback
**Concepts:**
- Feedback loops for quality improvement
- Adaptive prompt engineering
- Pattern recognition from successes
- Tracking improvement over iterations
- Learning from mistakes
**Run:**
```bash
npx tsx examples/intermediate/self-learning-system.ts
```
**Output:**
```
🧠 Starting Self-Learning Session
📊 Iteration 1/8
Quality: 65.0%
⚠️ Weaknesses: Description too short
🔧 Adapting strategy:
• Expand description with more details
📊 Iteration 5/8
Quality: 85.0%
✅ Target quality reached!
🎓 LEARNING SUMMARY
Quality Progression:
Iteration 1: ████████████████ 65.0%
Iteration 2: ████████████████████ 72.0%
Iteration 3: ██████████████████████ 78.0%
Iteration 4: ████████████████████████ 82.0%
Iteration 5: ██████████████████████████ 85.0%
Improvement: +20.0% (+30.8%)
```
**What You'll Build:** An adaptive generator that learns from feedback
---
### 🔴 Advanced Tutorials
#### 5. Custom Learning System (`advanced/custom-learning-system.ts`)
**Learn:** Extend self-learning with custom evaluation and domain-specific optimization
**Concepts:**
- Custom multi-objective evaluators
- Domain-specific learning strategies
- Progressive difficulty training
- Knowledge base management
- Transfer learning patterns
- Few-shot learning from examples
**Run:**
```bash
npx tsx examples/advanced/custom-learning-system.ts
```
**Output:**
```
🏋️ Starting Advanced Training Session
Domain: ecommerce
Strategy: adaptive
📚 Phase 1: Learning Basics (Easy Examples)
📚 Phase 2: Intermediate Concepts (Medium Examples)
📚 Phase 3: Advanced Patterns (Hard Examples)
🎓 TRAINING RESULTS
Knowledge Base: 8 high-quality examples
Average Quality: 87.3%
Learned Categories:
• electronics: 4 examples
• fitness: 2 examples
• photography: 2 examples
🧪 Testing Trained System
Test 1/3: Wireless Earbuds
📊 Metrics:
Overall: 89.2%
Accuracy: 92% | Creativity: 88%
Relevance: 90% | Engagement: 85%
📈 TEST SUMMARY
Overall Performance: 87.8%
```
**What You'll Build:** A sophisticated domain-specific learning system
---
#### 6. Production Pipeline (`advanced/production-pipeline.ts`)
**Learn:** Build production-ready data generation with monitoring and controls
**Concepts:**
- Error handling and retry logic
- Rate limiting and cost controls
- Batch processing with concurrency
- Quality validation
- Comprehensive metrics tracking
- Results persistence
- Performance monitoring
**Run:**
```bash
npx tsx examples/advanced/production-pipeline.ts
```
**Output:**
```
🏭 Starting Production Pipeline
Configuration:
Total Requests: 25
Batch Size: 5
Max Concurrency: 2
Cost Budget: $1.00
Rate Limit: 30/min
📦 Processing 5 batches...
Batch 1/5 (5 items)
✓ Batch complete: 5/5 successful
Cost so far: $0.0005
Cache hits: 0
📊 PIPELINE METRICS
Performance:
Total Time: 12.34s
Avg Request Time: 456ms
Throughput: 2.02 req/s
Reliability:
Total Requests: 25
Successful: 24 (96.0%)
Failed: 1
Retries: 2
Cost & Efficiency:
Total Cost: $0.0024
Avg Cost/Request: $0.000096
Cache Hit Rate: 32.0%
Cost Savings from Cache: $0.0008
💾 Results saved to: output/production/generation-2025-01-15T10-30-45.json
📊 Metrics saved to: output/production/metrics-2025-01-15T10-30-45.json
```
**What You'll Build:** An enterprise-grade data generation pipeline
---
## 🎯 Learning Path
### Recommended Order:
1. **Start Here:** `beginner/first-dspy-training.ts`
- Get comfortable with DSPy basics
- Understand training concepts
2. **Then:** `beginner/simple-data-generation.ts`
- Learn agentic-synth API
- Practice schema definition
3. **Next:** `intermediate/multi-model-comparison.ts`
- Compare model performance
- Understand cost/quality tradeoffs
4. **Continue:** `intermediate/self-learning-system.ts`
- Build adaptive systems
- Implement feedback loops
5. **Advanced:** `advanced/custom-learning-system.ts`
- Create domain-specific systems
- Multi-objective optimization
6. **Finally:** `advanced/production-pipeline.ts`
- Production patterns
- Monitoring and reliability
---
## 💡 Key Concepts
### DSPy Integration
All tutorials demonstrate DSPy.ts integration with agentic-synth:
- **Language Models:** Configure AI providers
- **Signatures:** Define input/output structures
- **Chain-of-Thought:** Step-by-step reasoning
- **Optimizers:** BootstrapFewShot, MIPROv2
### Quality Evaluation
Learn multiple evaluation approaches:
- **Basic Metrics:** Length, completeness
- **Advanced Metrics:** Creativity, relevance, engagement
- **Multi-Objective:** Balance multiple goals
- **Domain-Specific:** Custom validators
### Production Patterns
Essential patterns for real-world use:
- **Error Handling:** Retries, fallbacks, recovery
- **Rate Limiting:** API quota management
- **Cost Control:** Budget tracking, optimization
- **Monitoring:** Metrics, logging, alerting
- **Caching:** Performance optimization
---
## 🛠️ Customization
### Modify for Your Use Case
Each tutorial is designed to be customized:
```typescript
// Change the domain
const domain = 'healthcare'; // or 'finance', 'legal', etc.
// Adjust schemas
const schema = {
// Your custom fields
};
// Custom evaluation
class CustomEvaluator {
evaluate(output: any): number {
// Your logic
}
}
// Different models
const models = ['gemini', 'claude', 'gpt4', 'llama'];
```
---
## 📊 Expected Results
### Performance Benchmarks
| Tutorial | Runtime | API Calls | Est. Cost |
|----------|---------|-----------|-----------|
| First DSPy Training | 30-60s | 5-10 | $0.01 |
| Simple Data Generation | 10-30s | 2-5 | $0.005 |
| Multi-Model Comparison | 2-5min | 12-30 | $0.15 |
| Self-Learning System | 1-3min | 8-15 | $0.02 |
| Custom Learning | 3-6min | 15-30 | $0.05 |
| Production Pipeline | 1-2min | 20-50 | $0.10 |
*Costs are estimates and vary by model and usage*
---
## 🔧 Troubleshooting
### Common Issues
**API Key Not Set:**
```bash
# Error: API key not configured
export GEMINI_API_KEY="your-key-here"
```
**Module Not Found:**
```bash
# Run from package root
cd packages/agentic-synth-examples
npm install
```
**Rate Limit Errors:**
```typescript
// Adjust in pipeline config
rateLimitPerMinute: 10 // Lower the rate
```
**Cost Budget Exceeded:**
```typescript
// Increase budget or reduce requests
costBudget: 5.0 // Higher budget
```
---
## 📚 Additional Resources
### Documentation
- [Agentic-Synth Main Docs](../README.md)
- [DSPy.ts Documentation](https://github.com/XpressAI/dspy.ts)
- [API Reference](../docs/api.md)
### Related Examples
- [Production Use Cases](../examples/use-cases/)
- [Integration Patterns](../examples/integrations/)
- [Testing Strategies](../examples/testing/)
---
## 🤝 Contributing
Have an idea for a tutorial?
1. Create your example file
2. Add comprehensive comments
3. Include error handling
4. Test thoroughly
5. Submit a pull request
---
## 📞 Support
- **Issues:** [GitHub Issues](https://github.com/ruvnet/ruvector/issues)
- **Discussions:** [GitHub Discussions](https://github.com/ruvnet/ruvector/discussions)
- **Questions:** Tag us on Twitter [@ruvnet](https://twitter.com/ruvnet)
---
## 📄 License
MIT © [ruvnet](https://github.com/ruvnet)
---
**Ready to learn?** Start with the [First DSPy Training tutorial](beginner/first-dspy-training.ts)! 🚀

View File

@@ -0,0 +1,72 @@
/**
* ADVANCED TUTORIAL: Custom Learning System
*
* Extend the self-learning system with custom optimization strategies,
* domain-specific learning, and advanced evaluation metrics. Perfect for
* building production-grade adaptive AI systems.
*
* What you'll learn:
* - Creating custom evaluators
* - Domain-specific optimization
* - Advanced feedback loops
* - Multi-objective optimization
* - Transfer learning patterns
*
* Prerequisites:
* - Complete intermediate tutorials first
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/custom-learning-system.ts
*/
import { Prediction } from 'dspy.ts';
interface EvaluationMetrics {
accuracy: number;
creativity: number;
relevance: number;
engagement: number;
technicalQuality: number;
overall: number;
}
interface AdvancedLearningConfig {
domain: string;
objectives: string[];
weights: Record<string, number>;
learningStrategy: 'aggressive' | 'conservative' | 'adaptive';
convergenceThreshold: number;
diversityBonus: boolean;
transferLearning: boolean;
}
interface TrainingExample {
input: any;
expectedOutput: any;
quality: number;
metadata: {
domain: string;
difficulty: 'easy' | 'medium' | 'hard';
tags: string[];
};
}
interface Evaluator {
evaluate(output: Prediction, context: any): Promise<EvaluationMetrics>;
}
declare class EcommerceEvaluator implements Evaluator {
evaluate(output: Prediction, context: any): Promise<EvaluationMetrics>;
}
declare class AdvancedLearningSystem {
private lm;
private config;
private evaluator;
private knowledgeBase;
private promptStrategies;
constructor(config: AdvancedLearningConfig, evaluator: Evaluator);
private getTemperatureForStrategy;
learnFromExample(example: TrainingExample): Promise<void>;
train(examples: TrainingExample[]): Promise<void>;
private generate;
private findSimilarExamples;
private displayTrainingResults;
test(testCases: any[]): Promise<void>;
}
export { AdvancedLearningSystem, EcommerceEvaluator, AdvancedLearningConfig };
//# sourceMappingURL=custom-learning-system.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"custom-learning-system.d.ts","sourceRoot":"","sources":["custom-learning-system.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAsB,UAAU,EAAE,MAAM,SAAS,CAAC;AAIzD,UAAU,iBAAiB;IACzB,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,gBAAgB,EAAE,MAAM,CAAC;IACzB,OAAO,EAAE,MAAM,CAAC;CACjB;AAGD,UAAU,sBAAsB;IAC9B,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,EAAE,MAAM,EAAE,CAAC;IACrB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAChC,gBAAgB,EAAE,YAAY,GAAG,cAAc,GAAG,UAAU,CAAC;IAC7D,oBAAoB,EAAE,MAAM,CAAC;IAC7B,cAAc,EAAE,OAAO,CAAC;IACxB,gBAAgB,EAAE,OAAO,CAAC;CAC3B;AAGD,UAAU,eAAe;IACvB,KAAK,EAAE,GAAG,CAAC;IACX,cAAc,EAAE,GAAG,CAAC;IACpB,OAAO,EAAE,MAAM,CAAC;IAChB,QAAQ,EAAE;QACR,MAAM,EAAE,MAAM,CAAC;QACf,UAAU,EAAE,MAAM,GAAG,QAAQ,GAAG,MAAM,CAAC;QACvC,IAAI,EAAE,MAAM,EAAE,CAAC;KAChB,CAAC;CACH;AAGD,UAAU,SAAS;IACjB,QAAQ,CAAC,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,GAAG,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;CACxE;AAGD,cAAM,kBAAmB,YAAW,SAAS;IACrC,QAAQ,CAAC,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,GAAG,GAAG,OAAO,CAAC,iBAAiB,CAAC;CAsG7E;AAGD,cAAM,sBAAsB;IAC1B,OAAO,CAAC,EAAE,CAAK;IACf,OAAO,CAAC,MAAM,CAAyB;IACvC,OAAO,CAAC,SAAS,CAAY;IAC7B,OAAO,CAAC,aAAa,CAAyB;IAC9C,OAAO,CAAC,gBAAgB,CAAkC;gBAE9C,MAAM,EAAE,sBAAsB,EAAE,SAAS,EAAE,SAAS;IAYhE,OAAO,CAAC,yBAAyB;IAS3B,gBAAgB,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAqBzD,KAAK,CAAC,QAAQ,EAAE,eAAe,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;YAwCzC,QAAQ;IA0BtB,OAAO,CAAC,mBAAmB;IAW3B,OAAO,CAAC,sBAAsB;IA4BxB,IAAI,CAAC,SAAS,EAAE,GAAG,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;CAwD5C;AA+ED,OAAO,EAAE,sBAAsB,EAAE,kBAAkB,EAAE,sBAAsB,EAAE,CAAC"}

View File

@@ -0,0 +1,353 @@
"use strict";
/**
* ADVANCED TUTORIAL: Custom Learning System
*
* Extend the self-learning system with custom optimization strategies,
* domain-specific learning, and advanced evaluation metrics. Perfect for
* building production-grade adaptive AI systems.
*
* What you'll learn:
* - Creating custom evaluators
* - Domain-specific optimization
* - Advanced feedback loops
* - Multi-objective optimization
* - Transfer learning patterns
*
* Prerequisites:
* - Complete intermediate tutorials first
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/custom-learning-system.ts
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.EcommerceEvaluator = exports.AdvancedLearningSystem = void 0;
const dspy_ts_1 = require("dspy.ts");
// Domain-specific evaluator for e-commerce
class EcommerceEvaluator {
async evaluate(output, context) {
const metrics = {
accuracy: 0,
creativity: 0,
relevance: 0,
engagement: 0,
technicalQuality: 0,
overall: 0
};
// Accuracy: Check for required information
if (output.description && output.key_features) {
metrics.accuracy += 0.5;
// Check if key product attributes are mentioned
const desc = output.description.toLowerCase();
const productName = context.product_name.toLowerCase();
const category = context.category.toLowerCase();
if (desc.includes(productName.split(' ')[0])) {
metrics.accuracy += 0.25;
}
if (desc.includes(category)) {
metrics.accuracy += 0.25;
}
}
// Creativity: Check for unique, non-generic phrases
if (output.description) {
const genericPhrases = ['high quality', 'great product', 'best choice'];
const hasGenericPhrase = genericPhrases.some(phrase => output.description.toLowerCase().includes(phrase));
metrics.creativity = hasGenericPhrase ? 0.3 : 0.8;
// Bonus for specific details
const hasNumbers = /\d+/.test(output.description);
const hasSpecifics = /(\d+\s*(hours|days|years|gb|mb|kg|lbs))/i.test(output.description);
if (hasSpecifics)
metrics.creativity += 0.2;
}
// Relevance: Check alignment with category
const categoryKeywords = {
electronics: ['technology', 'device', 'digital', 'battery', 'power'],
fashion: ['style', 'design', 'material', 'comfort', 'wear'],
food: ['taste', 'flavor', 'nutrition', 'organic', 'fresh'],
fitness: ['workout', 'exercise', 'health', 'training', 'performance']
};
const category = context.category.toLowerCase();
const relevantKeywords = categoryKeywords[category] || [];
if (output.description) {
const desc = output.description.toLowerCase();
const matchedKeywords = relevantKeywords.filter(kw => desc.includes(kw));
metrics.relevance = Math.min(matchedKeywords.length / 3, 1.0);
}
// Engagement: Check for emotional appeal and calls to action
if (output.description) {
const desc = output.description.toLowerCase();
const emotionalWords = ['amazing', 'incredible', 'perfect', 'premium', 'exceptional', 'revolutionary'];
const actionWords = ['discover', 'experience', 'enjoy', 'upgrade', 'transform'];
const hasEmotion = emotionalWords.some(word => desc.includes(word));
const hasAction = actionWords.some(word => desc.includes(word));
metrics.engagement = (hasEmotion ? 0.5 : 0) + (hasAction ? 0.5 : 0);
}
// Technical Quality: Check structure and formatting
if (output.key_features && Array.isArray(output.key_features)) {
const features = output.key_features;
let techScore = 0;
// Optimal number of features
if (features.length >= 4 && features.length <= 6) {
techScore += 0.4;
}
// Feature formatting
const wellFormatted = features.filter(f => f.length >= 15 && f.length <= 60 && !f.endsWith('.'));
techScore += (wellFormatted.length / features.length) * 0.6;
metrics.technicalQuality = techScore;
}
// Calculate overall score with weights
metrics.overall = (metrics.accuracy * 0.25 +
metrics.creativity * 0.20 +
metrics.relevance * 0.25 +
metrics.engagement * 0.15 +
metrics.technicalQuality * 0.15);
return metrics;
}
}
exports.EcommerceEvaluator = EcommerceEvaluator;
// Advanced self-learning generator
class AdvancedLearningSystem {
constructor(config, evaluator) {
this.knowledgeBase = [];
this.promptStrategies = new Map();
this.config = config;
this.evaluator = evaluator;
this.lm = new dspy_ts_1.LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: this.getTemperatureForStrategy()
});
}
getTemperatureForStrategy() {
switch (this.config.learningStrategy) {
case 'aggressive': return 0.9;
case 'conservative': return 0.5;
case 'adaptive': return 0.7;
}
}
// Learn from a single example
async learnFromExample(example) {
console.log(`\n🎯 Learning from example (${example.metadata.difficulty})...`);
const output = await this.generate(example.input);
const metrics = await this.evaluator.evaluate(output, example.input);
console.log(` Overall Quality: ${(metrics.overall * 100).toFixed(1)}%`);
console.log(` Accuracy: ${(metrics.accuracy * 100).toFixed(0)}% | Creativity: ${(metrics.creativity * 100).toFixed(0)}%`);
console.log(` Relevance: ${(metrics.relevance * 100).toFixed(0)}% | Engagement: ${(metrics.engagement * 100).toFixed(0)}%`);
// Store high-quality examples
if (metrics.overall >= 0.7) {
this.knowledgeBase.push({
...example,
quality: metrics.overall
});
console.log(` ✓ Added to knowledge base`);
}
}
// Train on a dataset
async train(examples) {
console.log('🏋️ Starting Advanced Training Session\n');
console.log('='.repeat(70));
console.log(`\nDomain: ${this.config.domain}`);
console.log(`Strategy: ${this.config.learningStrategy}`);
console.log(`Examples: ${examples.length}`);
console.log(`\nObjectives:`);
this.config.objectives.forEach(obj => console.log(`${obj}`));
console.log('\n' + '='.repeat(70));
// Group by difficulty
const byDifficulty = {
easy: examples.filter(e => e.metadata.difficulty === 'easy'),
medium: examples.filter(e => e.metadata.difficulty === 'medium'),
hard: examples.filter(e => e.metadata.difficulty === 'hard')
};
// Progressive learning: start with easy, move to hard
console.log('\n📚 Phase 1: Learning Basics (Easy Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.easy) {
await this.learnFromExample(example);
}
console.log('\n📚 Phase 2: Intermediate Concepts (Medium Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.medium) {
await this.learnFromExample(example);
}
console.log('\n📚 Phase 3: Advanced Patterns (Hard Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.hard) {
await this.learnFromExample(example);
}
this.displayTrainingResults();
}
// Generate with learned knowledge
async generate(input) {
// Use knowledge base for few-shot learning
const similarExamples = this.findSimilarExamples(input, 3);
let enhancedDescription = 'Generate compelling product descriptions.';
if (similarExamples.length > 0) {
enhancedDescription += '\n\nLearn from these high-quality examples:\n';
similarExamples.forEach((ex, i) => {
enhancedDescription += `\nExample ${i + 1}:\n`;
enhancedDescription += `Input: ${JSON.stringify(ex.input)}\n`;
enhancedDescription += `Output: ${JSON.stringify(ex.expectedOutput)}`;
});
}
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]',
description: enhancedDescription
};
const generator = new dspy_ts_1.ChainOfThought(signature, { lm: this.lm });
return await generator.forward(input);
}
// Find similar examples from knowledge base
findSimilarExamples(input, count) {
// Simple similarity based on category match
const similar = this.knowledgeBase
.filter(ex => ex.input.category === input.category)
.sort((a, b) => b.quality - a.quality)
.slice(0, count);
return similar;
}
// Display training results
displayTrainingResults() {
console.log('\n\n' + '='.repeat(70));
console.log('\n🎓 TRAINING RESULTS\n');
console.log(`Knowledge Base: ${this.knowledgeBase.length} high-quality examples`);
if (this.knowledgeBase.length > 0) {
const avgQuality = this.knowledgeBase.reduce((sum, ex) => sum + ex.quality, 0) / this.knowledgeBase.length;
console.log(`Average Quality: ${(avgQuality * 100).toFixed(1)}%`);
// Group by category
const byCategory = {};
this.knowledgeBase.forEach(ex => {
const cat = ex.input.category;
byCategory[cat] = (byCategory[cat] || 0) + 1;
});
console.log(`\nLearned Categories:`);
Object.entries(byCategory).forEach(([cat, count]) => {
console.log(`${cat}: ${count} examples`);
});
}
console.log('\n✅ Training complete! System is ready for production.\n');
console.log('='.repeat(70) + '\n');
}
// Test the trained system
async test(testCases) {
console.log('\n🧪 Testing Trained System\n');
console.log('='.repeat(70) + '\n');
let totalMetrics = {
accuracy: 0,
creativity: 0,
relevance: 0,
engagement: 0,
technicalQuality: 0,
overall: 0
};
for (let i = 0; i < testCases.length; i++) {
const testCase = testCases[i];
console.log(`\nTest ${i + 1}/${testCases.length}: ${testCase.product_name}`);
console.log('─'.repeat(70));
const output = await this.generate(testCase);
const metrics = await this.evaluator.evaluate(output, testCase);
console.log(`\n📝 Generated:`);
console.log(` ${output.description}`);
console.log(`\n Features:`);
if (output.key_features) {
output.key_features.forEach((f) => console.log(`${f}`));
}
console.log(`\n📊 Metrics:`);
console.log(` Overall: ${(metrics.overall * 100).toFixed(1)}%`);
console.log(` Accuracy: ${(metrics.accuracy * 100).toFixed(0)}% | Creativity: ${(metrics.creativity * 100).toFixed(0)}%`);
console.log(` Relevance: ${(metrics.relevance * 100).toFixed(0)}% | Engagement: ${(metrics.engagement * 100).toFixed(0)}%`);
console.log(` Technical: ${(metrics.technicalQuality * 100).toFixed(0)}%`);
// Aggregate metrics
Object.keys(totalMetrics).forEach(key => {
totalMetrics[key] += metrics[key];
});
}
// Average metrics
Object.keys(totalMetrics).forEach(key => {
totalMetrics[key] /= testCases.length;
});
console.log('\n\n' + '='.repeat(70));
console.log('\n📈 TEST SUMMARY\n');
console.log(`Overall Performance: ${(totalMetrics.overall * 100).toFixed(1)}%`);
console.log(`\nDetailed Metrics:`);
console.log(` Accuracy: ${(totalMetrics.accuracy * 100).toFixed(1)}%`);
console.log(` Creativity: ${(totalMetrics.creativity * 100).toFixed(1)}%`);
console.log(` Relevance: ${(totalMetrics.relevance * 100).toFixed(1)}%`);
console.log(` Engagement: ${(totalMetrics.engagement * 100).toFixed(1)}%`);
console.log(` Technical Quality: ${(totalMetrics.technicalQuality * 100).toFixed(1)}%`);
console.log('\n' + '='.repeat(70) + '\n');
}
}
exports.AdvancedLearningSystem = AdvancedLearningSystem;
// Main execution
async function runAdvancedLearning() {
const config = {
domain: 'ecommerce',
objectives: [
'Generate accurate product descriptions',
'Maintain high creativity and engagement',
'Ensure category-specific relevance'
],
weights: {
accuracy: 0.25,
creativity: 0.20,
relevance: 0.25,
engagement: 0.15,
technical: 0.15
},
learningStrategy: 'adaptive',
convergenceThreshold: 0.85,
diversityBonus: true,
transferLearning: true
};
const evaluator = new EcommerceEvaluator();
const system = new AdvancedLearningSystem(config, evaluator);
// Training examples
const trainingExamples = [
{
input: { product_name: 'Smart Watch', category: 'electronics', price: 299 },
expectedOutput: {
description: 'Advanced fitness tracking meets elegant design in this premium smartwatch',
key_features: ['Heart rate monitoring', '7-day battery', 'Water resistant', 'GPS tracking']
},
quality: 0.9,
metadata: { domain: 'ecommerce', difficulty: 'easy', tags: ['electronics', 'wearable'] }
},
{
input: { product_name: 'Yoga Mat', category: 'fitness', price: 49 },
expectedOutput: {
description: 'Professional-grade yoga mat with superior grip and cushioning for all practice levels',
key_features: ['6mm thickness', 'Non-slip surface', 'Eco-friendly material', 'Easy to clean']
},
quality: 0.85,
metadata: { domain: 'ecommerce', difficulty: 'easy', tags: ['fitness', 'yoga'] }
},
{
input: { product_name: 'Mechanical Keyboard', category: 'electronics', price: 159 },
expectedOutput: {
description: 'Tactile perfection for enthusiasts with customizable RGB and premium switches',
key_features: ['Cherry MX switches', 'RGB backlighting', 'Programmable keys', 'Aluminum frame']
},
quality: 0.92,
metadata: { domain: 'ecommerce', difficulty: 'medium', tags: ['electronics', 'gaming'] }
}
];
// Train the system
await system.train(trainingExamples);
// Test the system
const testCases = [
{ product_name: 'Wireless Earbuds', category: 'electronics', price: 129 },
{ product_name: 'Resistance Bands Set', category: 'fitness', price: 29 },
{ product_name: 'Laptop Stand', category: 'electronics', price: 59 }
];
await system.test(testCases);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runAdvancedLearning().catch(error => {
console.error('❌ Advanced learning failed:', error);
process.exit(1);
});
}
//# sourceMappingURL=custom-learning-system.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,460 @@
/**
* ADVANCED TUTORIAL: Custom Learning System
*
* Extend the self-learning system with custom optimization strategies,
* domain-specific learning, and advanced evaluation metrics. Perfect for
* building production-grade adaptive AI systems.
*
* What you'll learn:
* - Creating custom evaluators
* - Domain-specific optimization
* - Advanced feedback loops
* - Multi-objective optimization
* - Transfer learning patterns
*
* Prerequisites:
* - Complete intermediate tutorials first
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/custom-learning-system.ts
*/
import { LM, ChainOfThought, Prediction } from 'dspy.ts';
import { AgenticSynth } from '@ruvector/agentic-synth';
// Multi-objective evaluation metrics
interface EvaluationMetrics {
accuracy: number;
creativity: number;
relevance: number;
engagement: number;
technicalQuality: number;
overall: number;
}
// Advanced learning configuration
interface AdvancedLearningConfig {
domain: string;
objectives: string[];
weights: Record<string, number>;
learningStrategy: 'aggressive' | 'conservative' | 'adaptive';
convergenceThreshold: number;
diversityBonus: boolean;
transferLearning: boolean;
}
// Training example with rich metadata
interface TrainingExample {
input: any;
expectedOutput: any;
quality: number;
metadata: {
domain: string;
difficulty: 'easy' | 'medium' | 'hard';
tags: string[];
};
}
// Custom evaluator interface
interface Evaluator {
evaluate(output: Prediction, context: any): Promise<EvaluationMetrics>;
}
// Domain-specific evaluator for e-commerce
class EcommerceEvaluator implements Evaluator {
async evaluate(output: Prediction, context: any): Promise<EvaluationMetrics> {
const metrics: EvaluationMetrics = {
accuracy: 0,
creativity: 0,
relevance: 0,
engagement: 0,
technicalQuality: 0,
overall: 0
};
// Accuracy: Check for required information
if (output.description && output.key_features) {
metrics.accuracy += 0.5;
// Check if key product attributes are mentioned
const desc = output.description.toLowerCase();
const productName = context.product_name.toLowerCase();
const category = context.category.toLowerCase();
if (desc.includes(productName.split(' ')[0])) {
metrics.accuracy += 0.25;
}
if (desc.includes(category)) {
metrics.accuracy += 0.25;
}
}
// Creativity: Check for unique, non-generic phrases
if (output.description) {
const genericPhrases = ['high quality', 'great product', 'best choice'];
const hasGenericPhrase = genericPhrases.some(phrase =>
output.description.toLowerCase().includes(phrase)
);
metrics.creativity = hasGenericPhrase ? 0.3 : 0.8;
// Bonus for specific details
const hasNumbers = /\d+/.test(output.description);
const hasSpecifics = /(\d+\s*(hours|days|years|gb|mb|kg|lbs))/i.test(output.description);
if (hasSpecifics) metrics.creativity += 0.2;
}
// Relevance: Check alignment with category
const categoryKeywords: Record<string, string[]> = {
electronics: ['technology', 'device', 'digital', 'battery', 'power'],
fashion: ['style', 'design', 'material', 'comfort', 'wear'],
food: ['taste', 'flavor', 'nutrition', 'organic', 'fresh'],
fitness: ['workout', 'exercise', 'health', 'training', 'performance']
};
const category = context.category.toLowerCase();
const relevantKeywords = categoryKeywords[category] || [];
if (output.description) {
const desc = output.description.toLowerCase();
const matchedKeywords = relevantKeywords.filter(kw => desc.includes(kw));
metrics.relevance = Math.min(matchedKeywords.length / 3, 1.0);
}
// Engagement: Check for emotional appeal and calls to action
if (output.description) {
const desc = output.description.toLowerCase();
const emotionalWords = ['amazing', 'incredible', 'perfect', 'premium', 'exceptional', 'revolutionary'];
const actionWords = ['discover', 'experience', 'enjoy', 'upgrade', 'transform'];
const hasEmotion = emotionalWords.some(word => desc.includes(word));
const hasAction = actionWords.some(word => desc.includes(word));
metrics.engagement = (hasEmotion ? 0.5 : 0) + (hasAction ? 0.5 : 0);
}
// Technical Quality: Check structure and formatting
if (output.key_features && Array.isArray(output.key_features)) {
const features = output.key_features;
let techScore = 0;
// Optimal number of features
if (features.length >= 4 && features.length <= 6) {
techScore += 0.4;
}
// Feature formatting
const wellFormatted = features.filter(f =>
f.length >= 15 && f.length <= 60 && !f.endsWith('.')
);
techScore += (wellFormatted.length / features.length) * 0.6;
metrics.technicalQuality = techScore;
}
// Calculate overall score with weights
metrics.overall = (
metrics.accuracy * 0.25 +
metrics.creativity * 0.20 +
metrics.relevance * 0.25 +
metrics.engagement * 0.15 +
metrics.technicalQuality * 0.15
);
return metrics;
}
}
// Advanced self-learning generator
class AdvancedLearningSystem {
private lm: LM;
private config: AdvancedLearningConfig;
private evaluator: Evaluator;
private knowledgeBase: TrainingExample[] = [];
private promptStrategies: Map<string, number> = new Map();
constructor(config: AdvancedLearningConfig, evaluator: Evaluator) {
this.config = config;
this.evaluator = evaluator;
this.lm = new LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: this.getTemperatureForStrategy()
});
}
private getTemperatureForStrategy(): number {
switch (this.config.learningStrategy) {
case 'aggressive': return 0.9;
case 'conservative': return 0.5;
case 'adaptive': return 0.7;
}
}
// Learn from a single example
async learnFromExample(example: TrainingExample): Promise<void> {
console.log(`\n🎯 Learning from example (${example.metadata.difficulty})...`);
const output = await this.generate(example.input);
const metrics = await this.evaluator.evaluate(output, example.input);
console.log(` Overall Quality: ${(metrics.overall * 100).toFixed(1)}%`);
console.log(` Accuracy: ${(metrics.accuracy * 100).toFixed(0)}% | Creativity: ${(metrics.creativity * 100).toFixed(0)}%`);
console.log(` Relevance: ${(metrics.relevance * 100).toFixed(0)}% | Engagement: ${(metrics.engagement * 100).toFixed(0)}%`);
// Store high-quality examples
if (metrics.overall >= 0.7) {
this.knowledgeBase.push({
...example,
quality: metrics.overall
});
console.log(` ✓ Added to knowledge base`);
}
}
// Train on a dataset
async train(examples: TrainingExample[]): Promise<void> {
console.log('🏋️ Starting Advanced Training Session\n');
console.log('=' .repeat(70));
console.log(`\nDomain: ${this.config.domain}`);
console.log(`Strategy: ${this.config.learningStrategy}`);
console.log(`Examples: ${examples.length}`);
console.log(`\nObjectives:`);
this.config.objectives.forEach(obj => console.log(`${obj}`));
console.log('\n' + '=' .repeat(70));
// Group by difficulty
const byDifficulty = {
easy: examples.filter(e => e.metadata.difficulty === 'easy'),
medium: examples.filter(e => e.metadata.difficulty === 'medium'),
hard: examples.filter(e => e.metadata.difficulty === 'hard')
};
// Progressive learning: start with easy, move to hard
console.log('\n📚 Phase 1: Learning Basics (Easy Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.easy) {
await this.learnFromExample(example);
}
console.log('\n📚 Phase 2: Intermediate Concepts (Medium Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.medium) {
await this.learnFromExample(example);
}
console.log('\n📚 Phase 3: Advanced Patterns (Hard Examples)');
console.log('─'.repeat(70));
for (const example of byDifficulty.hard) {
await this.learnFromExample(example);
}
this.displayTrainingResults();
}
// Generate with learned knowledge
private async generate(input: any): Promise<Prediction> {
// Use knowledge base for few-shot learning
const similarExamples = this.findSimilarExamples(input, 3);
let enhancedDescription = 'Generate compelling product descriptions.';
if (similarExamples.length > 0) {
enhancedDescription += '\n\nLearn from these high-quality examples:\n';
similarExamples.forEach((ex, i) => {
enhancedDescription += `\nExample ${i + 1}:\n`;
enhancedDescription += `Input: ${JSON.stringify(ex.input)}\n`;
enhancedDescription += `Output: ${JSON.stringify(ex.expectedOutput)}`;
});
}
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]',
description: enhancedDescription
};
const generator = new ChainOfThought(signature, { lm: this.lm });
return await generator.forward(input);
}
// Find similar examples from knowledge base
private findSimilarExamples(input: any, count: number): TrainingExample[] {
// Simple similarity based on category match
const similar = this.knowledgeBase
.filter(ex => ex.input.category === input.category)
.sort((a, b) => b.quality - a.quality)
.slice(0, count);
return similar;
}
// Display training results
private displayTrainingResults(): void {
console.log('\n\n' + '=' .repeat(70));
console.log('\n🎓 TRAINING RESULTS\n');
console.log(`Knowledge Base: ${this.knowledgeBase.length} high-quality examples`);
if (this.knowledgeBase.length > 0) {
const avgQuality = this.knowledgeBase.reduce((sum, ex) => sum + ex.quality, 0) / this.knowledgeBase.length;
console.log(`Average Quality: ${(avgQuality * 100).toFixed(1)}%`);
// Group by category
const byCategory: Record<string, number> = {};
this.knowledgeBase.forEach(ex => {
const cat = ex.input.category;
byCategory[cat] = (byCategory[cat] || 0) + 1;
});
console.log(`\nLearned Categories:`);
Object.entries(byCategory).forEach(([cat, count]) => {
console.log(`${cat}: ${count} examples`);
});
}
console.log('\n✅ Training complete! System is ready for production.\n');
console.log('=' .repeat(70) + '\n');
}
// Test the trained system
async test(testCases: any[]): Promise<void> {
console.log('\n🧪 Testing Trained System\n');
console.log('=' .repeat(70) + '\n');
let totalMetrics: EvaluationMetrics = {
accuracy: 0,
creativity: 0,
relevance: 0,
engagement: 0,
technicalQuality: 0,
overall: 0
};
for (let i = 0; i < testCases.length; i++) {
const testCase = testCases[i];
console.log(`\nTest ${i + 1}/${testCases.length}: ${testCase.product_name}`);
console.log('─'.repeat(70));
const output = await this.generate(testCase);
const metrics = await this.evaluator.evaluate(output, testCase);
console.log(`\n📝 Generated:`);
console.log(` ${output.description}`);
console.log(`\n Features:`);
if (output.key_features) {
output.key_features.forEach((f: string) => console.log(`${f}`));
}
console.log(`\n📊 Metrics:`);
console.log(` Overall: ${(metrics.overall * 100).toFixed(1)}%`);
console.log(` Accuracy: ${(metrics.accuracy * 100).toFixed(0)}% | Creativity: ${(metrics.creativity * 100).toFixed(0)}%`);
console.log(` Relevance: ${(metrics.relevance * 100).toFixed(0)}% | Engagement: ${(metrics.engagement * 100).toFixed(0)}%`);
console.log(` Technical: ${(metrics.technicalQuality * 100).toFixed(0)}%`);
// Aggregate metrics
Object.keys(totalMetrics).forEach(key => {
totalMetrics[key as keyof EvaluationMetrics] += metrics[key as keyof EvaluationMetrics];
});
}
// Average metrics
Object.keys(totalMetrics).forEach(key => {
totalMetrics[key as keyof EvaluationMetrics] /= testCases.length;
});
console.log('\n\n' + '=' .repeat(70));
console.log('\n📈 TEST SUMMARY\n');
console.log(`Overall Performance: ${(totalMetrics.overall * 100).toFixed(1)}%`);
console.log(`\nDetailed Metrics:`);
console.log(` Accuracy: ${(totalMetrics.accuracy * 100).toFixed(1)}%`);
console.log(` Creativity: ${(totalMetrics.creativity * 100).toFixed(1)}%`);
console.log(` Relevance: ${(totalMetrics.relevance * 100).toFixed(1)}%`);
console.log(` Engagement: ${(totalMetrics.engagement * 100).toFixed(1)}%`);
console.log(` Technical Quality: ${(totalMetrics.technicalQuality * 100).toFixed(1)}%`);
console.log('\n' + '=' .repeat(70) + '\n');
}
}
// Main execution
async function runAdvancedLearning() {
const config: AdvancedLearningConfig = {
domain: 'ecommerce',
objectives: [
'Generate accurate product descriptions',
'Maintain high creativity and engagement',
'Ensure category-specific relevance'
],
weights: {
accuracy: 0.25,
creativity: 0.20,
relevance: 0.25,
engagement: 0.15,
technical: 0.15
},
learningStrategy: 'adaptive',
convergenceThreshold: 0.85,
diversityBonus: true,
transferLearning: true
};
const evaluator = new EcommerceEvaluator();
const system = new AdvancedLearningSystem(config, evaluator);
// Training examples
const trainingExamples: TrainingExample[] = [
{
input: { product_name: 'Smart Watch', category: 'electronics', price: 299 },
expectedOutput: {
description: 'Advanced fitness tracking meets elegant design in this premium smartwatch',
key_features: ['Heart rate monitoring', '7-day battery', 'Water resistant', 'GPS tracking']
},
quality: 0.9,
metadata: { domain: 'ecommerce', difficulty: 'easy', tags: ['electronics', 'wearable'] }
},
{
input: { product_name: 'Yoga Mat', category: 'fitness', price: 49 },
expectedOutput: {
description: 'Professional-grade yoga mat with superior grip and cushioning for all practice levels',
key_features: ['6mm thickness', 'Non-slip surface', 'Eco-friendly material', 'Easy to clean']
},
quality: 0.85,
metadata: { domain: 'ecommerce', difficulty: 'easy', tags: ['fitness', 'yoga'] }
},
{
input: { product_name: 'Mechanical Keyboard', category: 'electronics', price: 159 },
expectedOutput: {
description: 'Tactile perfection for enthusiasts with customizable RGB and premium switches',
key_features: ['Cherry MX switches', 'RGB backlighting', 'Programmable keys', 'Aluminum frame']
},
quality: 0.92,
metadata: { domain: 'ecommerce', difficulty: 'medium', tags: ['electronics', 'gaming'] }
}
];
// Train the system
await system.train(trainingExamples);
// Test the system
const testCases = [
{ product_name: 'Wireless Earbuds', category: 'electronics', price: 129 },
{ product_name: 'Resistance Bands Set', category: 'fitness', price: 29 },
{ product_name: 'Laptop Stand', category: 'electronics', price: 59 }
];
await system.test(testCases);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runAdvancedLearning().catch(error => {
console.error('❌ Advanced learning failed:', error);
process.exit(1);
});
}
export { AdvancedLearningSystem, EcommerceEvaluator, AdvancedLearningConfig };

View File

@@ -0,0 +1,83 @@
/**
* ADVANCED TUTORIAL: Production Pipeline
*
* Build a complete production-ready data generation pipeline with:
* - Error handling and retry logic
* - Monitoring and metrics
* - Rate limiting and cost controls
* - Batch processing and caching
* - Quality validation
*
* What you'll learn:
* - Production-grade error handling
* - Performance monitoring
* - Cost optimization
* - Scalability patterns
* - Deployment best practices
*
* Prerequisites:
* - Complete previous tutorials
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/production-pipeline.ts
*/
import { GenerationResult } from '@ruvector/agentic-synth';
interface PipelineConfig {
maxRetries: number;
retryDelay: number;
batchSize: number;
maxConcurrency: number;
qualityThreshold: number;
costBudget: number;
rateLimitPerMinute: number;
enableCaching: boolean;
outputDirectory: string;
}
interface PipelineMetrics {
totalRequests: number;
successfulRequests: number;
failedRequests: number;
totalDuration: number;
totalCost: number;
averageQuality: number;
cacheHits: number;
retries: number;
errors: Array<{
timestamp: Date;
error: string;
context: any;
}>;
}
interface QualityValidator {
validate(data: any): {
valid: boolean;
score: number;
issues: string[];
};
}
declare class ProductionPipeline {
private config;
private synth;
private metrics;
private requestsThisMinute;
private minuteStartTime;
constructor(config?: Partial<PipelineConfig>);
private checkRateLimit;
private checkCostBudget;
private generateWithRetry;
private processBatch;
run(requests: any[], validator?: QualityValidator): Promise<GenerationResult[]>;
private saveResults;
private displayMetrics;
getMetrics(): PipelineMetrics;
}
declare class ProductQualityValidator implements QualityValidator {
validate(data: any[]): {
valid: boolean;
score: number;
issues: string[];
};
}
export { ProductionPipeline, ProductQualityValidator, PipelineConfig, PipelineMetrics };
//# sourceMappingURL=production-pipeline.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"production-pipeline.d.ts","sourceRoot":"","sources":["production-pipeline.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AAEH,OAAO,EAAgB,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAKzE,UAAU,cAAc;IACtB,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,gBAAgB,EAAE,MAAM,CAAC;IACzB,UAAU,EAAE,MAAM,CAAC;IACnB,kBAAkB,EAAE,MAAM,CAAC;IAC3B,aAAa,EAAE,OAAO,CAAC;IACvB,eAAe,EAAE,MAAM,CAAC;CACzB;AAGD,UAAU,eAAe;IACvB,aAAa,EAAE,MAAM,CAAC;IACtB,kBAAkB,EAAE,MAAM,CAAC;IAC3B,cAAc,EAAE,MAAM,CAAC;IACvB,aAAa,EAAE,MAAM,CAAC;IACtB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,KAAK,CAAC;QAAE,SAAS,EAAE,IAAI,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,GAAG,CAAA;KAAE,CAAC,CAAC;CACjE;AAGD,UAAU,gBAAgB;IACxB,QAAQ,CAAC,IAAI,EAAE,GAAG,GAAG;QAAE,KAAK,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE,CAAC;CAC1E;AAGD,cAAM,kBAAkB;IACtB,OAAO,CAAC,MAAM,CAAiB;IAC/B,OAAO,CAAC,KAAK,CAAe;IAC5B,OAAO,CAAC,OAAO,CAAkB;IACjC,OAAO,CAAC,kBAAkB,CAAa;IACvC,OAAO,CAAC,eAAe,CAAsB;gBAEjC,MAAM,GAAE,OAAO,CAAC,cAAc,CAAM;YA0ClC,cAAc;IAoB5B,OAAO,CAAC,eAAe;YAOT,iBAAiB;YAqDjB,YAAY;IAyCpB,GAAG,CACP,QAAQ,EAAE,GAAG,EAAE,EACf,SAAS,CAAC,EAAE,gBAAgB,GAC3B,OAAO,CAAC,gBAAgB,EAAE,CAAC;YA4DhB,WAAW;IA6BzB,OAAO,CAAC,cAAc;IAuCtB,UAAU,IAAI,eAAe;CAG9B;AAGD,cAAM,uBAAwB,YAAW,gBAAgB;IACvD,QAAQ,CAAC,IAAI,EAAE,GAAG,EAAE,GAAG;QAAE,KAAK,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,EAAE,CAAA;KAAE;CAyB3E;AAiDD,OAAO,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,cAAc,EAAE,eAAe,EAAE,CAAC"}

View File

@@ -0,0 +1,341 @@
"use strict";
/**
* ADVANCED TUTORIAL: Production Pipeline
*
* Build a complete production-ready data generation pipeline with:
* - Error handling and retry logic
* - Monitoring and metrics
* - Rate limiting and cost controls
* - Batch processing and caching
* - Quality validation
*
* What you'll learn:
* - Production-grade error handling
* - Performance monitoring
* - Cost optimization
* - Scalability patterns
* - Deployment best practices
*
* Prerequisites:
* - Complete previous tutorials
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/production-pipeline.ts
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.ProductQualityValidator = exports.ProductionPipeline = void 0;
const agentic_synth_1 = require("@ruvector/agentic-synth");
const fs_1 = require("fs");
const path_1 = require("path");
// Production-grade pipeline
class ProductionPipeline {
constructor(config = {}) {
this.requestsThisMinute = 0;
this.minuteStartTime = Date.now();
this.config = {
maxRetries: config.maxRetries || 3,
retryDelay: config.retryDelay || 1000,
batchSize: config.batchSize || 10,
maxConcurrency: config.maxConcurrency || 3,
qualityThreshold: config.qualityThreshold || 0.7,
costBudget: config.costBudget || 10.0,
rateLimitPerMinute: config.rateLimitPerMinute || 60,
enableCaching: config.enableCaching !== false,
outputDirectory: config.outputDirectory || './output'
};
this.synth = new agentic_synth_1.AgenticSynth({
provider: 'gemini',
apiKey: process.env.GEMINI_API_KEY,
model: 'gemini-2.0-flash-exp',
cacheStrategy: this.config.enableCaching ? 'memory' : 'none',
cacheTTL: 3600,
maxRetries: this.config.maxRetries,
timeout: 30000
});
this.metrics = {
totalRequests: 0,
successfulRequests: 0,
failedRequests: 0,
totalDuration: 0,
totalCost: 0,
averageQuality: 0,
cacheHits: 0,
retries: 0,
errors: []
};
// Ensure output directory exists
if (!(0, fs_1.existsSync)(this.config.outputDirectory)) {
(0, fs_1.mkdirSync)(this.config.outputDirectory, { recursive: true });
}
}
// Rate limiting check
async checkRateLimit() {
const now = Date.now();
const elapsedMinutes = (now - this.minuteStartTime) / 60000;
if (elapsedMinutes >= 1) {
// Reset counter for new minute
this.requestsThisMinute = 0;
this.minuteStartTime = now;
}
if (this.requestsThisMinute >= this.config.rateLimitPerMinute) {
const waitTime = 60000 - (now - this.minuteStartTime);
console.log(`⏳ Rate limit reached, waiting ${Math.ceil(waitTime / 1000)}s...`);
await new Promise(resolve => setTimeout(resolve, waitTime));
this.requestsThisMinute = 0;
this.minuteStartTime = Date.now();
}
}
// Cost check
checkCostBudget() {
if (this.metrics.totalCost >= this.config.costBudget) {
throw new Error(`Cost budget exceeded: $${this.metrics.totalCost.toFixed(4)} >= $${this.config.costBudget}`);
}
}
// Generate with retry logic
async generateWithRetry(options, attempt = 1) {
try {
await this.checkRateLimit();
this.checkCostBudget();
this.requestsThisMinute++;
this.metrics.totalRequests++;
const startTime = Date.now();
const result = await this.synth.generateStructured(options);
const duration = Date.now() - startTime;
this.metrics.totalDuration += duration;
this.metrics.successfulRequests++;
if (result.metadata.cached) {
this.metrics.cacheHits++;
}
// Estimate cost (rough approximation)
const estimatedCost = result.metadata.cached ? 0 : 0.0001;
this.metrics.totalCost += estimatedCost;
return result;
}
catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
if (attempt < this.config.maxRetries) {
this.metrics.retries++;
console.log(`⚠️ Attempt ${attempt} failed, retrying... (${errorMsg})`);
await new Promise(resolve => setTimeout(resolve, this.config.retryDelay * attempt));
return this.generateWithRetry(options, attempt + 1);
}
else {
this.metrics.failedRequests++;
this.metrics.errors.push({
timestamp: new Date(),
error: errorMsg,
context: options
});
throw error;
}
}
}
// Process a single batch
async processBatch(requests, validator) {
const results = [];
// Process with concurrency control
for (let i = 0; i < requests.length; i += this.config.maxConcurrency) {
const batch = requests.slice(i, i + this.config.maxConcurrency);
const batchResults = await Promise.allSettled(batch.map(req => this.generateWithRetry(req)));
batchResults.forEach((result, idx) => {
if (result.status === 'fulfilled') {
const genResult = result.value;
// Validate quality if validator provided
if (validator) {
const validation = validator.validate(genResult.data);
if (validation.valid) {
results.push(genResult);
}
else {
console.log(`⚠️ Quality validation failed (score: ${validation.score.toFixed(2)})`);
console.log(` Issues: ${validation.issues.join(', ')}`);
}
}
else {
results.push(genResult);
}
}
else {
console.error(`❌ Batch item ${i + idx} failed:`, result.reason);
}
});
}
return results;
}
// Main pipeline execution
async run(requests, validator) {
console.log('🏭 Starting Production Pipeline\n');
console.log('='.repeat(70));
console.log(`\nConfiguration:`);
console.log(` Total Requests: ${requests.length}`);
console.log(` Batch Size: ${this.config.batchSize}`);
console.log(` Max Concurrency: ${this.config.maxConcurrency}`);
console.log(` Max Retries: ${this.config.maxRetries}`);
console.log(` Cost Budget: $${this.config.costBudget}`);
console.log(` Rate Limit: ${this.config.rateLimitPerMinute}/min`);
console.log(` Caching: ${this.config.enableCaching ? 'Enabled' : 'Disabled'}`);
console.log(` Output: ${this.config.outputDirectory}`);
console.log('\n' + '='.repeat(70) + '\n');
const startTime = Date.now();
const allResults = [];
// Split into batches
const batches = [];
for (let i = 0; i < requests.length; i += this.config.batchSize) {
batches.push(requests.slice(i, i + this.config.batchSize));
}
console.log(`📦 Processing ${batches.length} batches...\n`);
// Process each batch
for (let i = 0; i < batches.length; i++) {
console.log(`\nBatch ${i + 1}/${batches.length} (${batches[i].length} items)`);
console.log('─'.repeat(70));
try {
const batchResults = await this.processBatch(batches[i], validator);
allResults.push(...batchResults);
console.log(`✓ Batch complete: ${batchResults.length}/${batches[i].length} successful`);
console.log(` Cost so far: $${this.metrics.totalCost.toFixed(4)}`);
console.log(` Cache hits: ${this.metrics.cacheHits}`);
}
catch (error) {
console.error(`✗ Batch failed:`, error instanceof Error ? error.message : 'Unknown error');
if (error instanceof Error && error.message.includes('budget')) {
console.log('\n⚠ Cost budget exceeded, stopping pipeline...');
break;
}
}
}
const totalTime = Date.now() - startTime;
// Save results
await this.saveResults(allResults);
// Display metrics
this.displayMetrics(totalTime);
return allResults;
}
// Save results to disk
async saveResults(results) {
try {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `generation-${timestamp}.json`;
const filepath = (0, path_1.join)(this.config.outputDirectory, filename);
const output = {
timestamp: new Date(),
results: results.map(r => r.data),
metadata: {
count: results.length,
metrics: this.metrics
}
};
(0, fs_1.writeFileSync)(filepath, JSON.stringify(output, null, 2));
console.log(`\n💾 Results saved to: ${filepath}`);
// Save metrics separately
const metricsFile = (0, path_1.join)(this.config.outputDirectory, `metrics-${timestamp}.json`);
(0, fs_1.writeFileSync)(metricsFile, JSON.stringify(this.metrics, null, 2));
console.log(`📊 Metrics saved to: ${metricsFile}`);
}
catch (error) {
console.error('⚠️ Failed to save results:', error instanceof Error ? error.message : 'Unknown error');
}
}
// Display comprehensive metrics
displayMetrics(totalTime) {
console.log('\n\n' + '='.repeat(70));
console.log('\n📊 PIPELINE METRICS\n');
const successRate = (this.metrics.successfulRequests / this.metrics.totalRequests) * 100;
const avgDuration = this.metrics.totalDuration / this.metrics.successfulRequests;
const cacheHitRate = (this.metrics.cacheHits / this.metrics.totalRequests) * 100;
console.log('Performance:');
console.log(` Total Time: ${(totalTime / 1000).toFixed(2)}s`);
console.log(` Avg Request Time: ${avgDuration.toFixed(0)}ms`);
console.log(` Throughput: ${(this.metrics.successfulRequests / (totalTime / 1000)).toFixed(2)} req/s`);
console.log('\nReliability:');
console.log(` Total Requests: ${this.metrics.totalRequests}`);
console.log(` Successful: ${this.metrics.successfulRequests} (${successRate.toFixed(1)}%)`);
console.log(` Failed: ${this.metrics.failedRequests}`);
console.log(` Retries: ${this.metrics.retries}`);
console.log('\nCost & Efficiency:');
console.log(` Total Cost: $${this.metrics.totalCost.toFixed(4)}`);
console.log(` Avg Cost/Request: $${(this.metrics.totalCost / this.metrics.totalRequests).toFixed(6)}`);
console.log(` Cache Hit Rate: ${cacheHitRate.toFixed(1)}%`);
console.log(` Cost Savings from Cache: $${(this.metrics.cacheHits * 0.0001).toFixed(4)}`);
if (this.metrics.errors.length > 0) {
console.log(`\n⚠️ Errors (${this.metrics.errors.length}):`);
this.metrics.errors.slice(0, 5).forEach((err, i) => {
console.log(` ${i + 1}. ${err.error}`);
});
if (this.metrics.errors.length > 5) {
console.log(` ... and ${this.metrics.errors.length - 5} more`);
}
}
console.log('\n' + '='.repeat(70) + '\n');
}
// Get metrics
getMetrics() {
return { ...this.metrics };
}
}
exports.ProductionPipeline = ProductionPipeline;
// Example quality validator
class ProductQualityValidator {
validate(data) {
const issues = [];
let score = 1.0;
if (!Array.isArray(data) || data.length === 0) {
return { valid: false, score: 0, issues: ['No data generated'] };
}
data.forEach((item, idx) => {
if (!item.description || item.description.length < 50) {
issues.push(`Item ${idx}: Description too short`);
score -= 0.1;
}
if (!item.key_features || !Array.isArray(item.key_features) || item.key_features.length < 3) {
issues.push(`Item ${idx}: Insufficient features`);
score -= 0.1;
}
});
score = Math.max(0, score);
const valid = score >= 0.7;
return { valid, score, issues };
}
}
exports.ProductQualityValidator = ProductQualityValidator;
// Main execution
async function runProductionPipeline() {
const pipeline = new ProductionPipeline({
maxRetries: 3,
retryDelay: 2000,
batchSize: 5,
maxConcurrency: 2,
qualityThreshold: 0.7,
costBudget: 1.0,
rateLimitPerMinute: 30,
enableCaching: true,
outputDirectory: (0, path_1.join)(process.cwd(), 'examples', 'output', 'production')
});
const validator = new ProductQualityValidator();
// Generate product data for e-commerce catalog
const requests = [
{
count: 2,
schema: {
id: { type: 'string', required: true },
name: { type: 'string', required: true },
description: { type: 'string', required: true },
key_features: { type: 'array', items: { type: 'string' }, required: true },
price: { type: 'number', required: true, minimum: 10, maximum: 1000 },
category: { type: 'string', enum: ['Electronics', 'Clothing', 'Home', 'Sports'] }
}
}
];
// Duplicate requests to test batching
const allRequests = Array(5).fill(null).map(() => requests[0]);
const results = await pipeline.run(allRequests, validator);
console.log(`\n✅ Pipeline complete! Generated ${results.length} batches of products.\n`);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runProductionPipeline().catch(error => {
console.error('❌ Pipeline failed:', error);
process.exit(1);
});
}
//# sourceMappingURL=production-pipeline.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,444 @@
/**
* ADVANCED TUTORIAL: Production Pipeline
*
* Build a complete production-ready data generation pipeline with:
* - Error handling and retry logic
* - Monitoring and metrics
* - Rate limiting and cost controls
* - Batch processing and caching
* - Quality validation
*
* What you'll learn:
* - Production-grade error handling
* - Performance monitoring
* - Cost optimization
* - Scalability patterns
* - Deployment best practices
*
* Prerequisites:
* - Complete previous tutorials
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/advanced/production-pipeline.ts
*/
import { AgenticSynth, GenerationResult } from '@ruvector/agentic-synth';
import { writeFileSync, existsSync, mkdirSync } from 'fs';
import { join } from 'path';
// Pipeline configuration
interface PipelineConfig {
maxRetries: number;
retryDelay: number;
batchSize: number;
maxConcurrency: number;
qualityThreshold: number;
costBudget: number;
rateLimitPerMinute: number;
enableCaching: boolean;
outputDirectory: string;
}
// Metrics tracking
interface PipelineMetrics {
totalRequests: number;
successfulRequests: number;
failedRequests: number;
totalDuration: number;
totalCost: number;
averageQuality: number;
cacheHits: number;
retries: number;
errors: Array<{ timestamp: Date; error: string; context: any }>;
}
// Quality validator
interface QualityValidator {
validate(data: any): { valid: boolean; score: number; issues: string[] };
}
// Production-grade pipeline
class ProductionPipeline {
private config: PipelineConfig;
private synth: AgenticSynth;
private metrics: PipelineMetrics;
private requestsThisMinute: number = 0;
private minuteStartTime: number = Date.now();
constructor(config: Partial<PipelineConfig> = {}) {
this.config = {
maxRetries: config.maxRetries || 3,
retryDelay: config.retryDelay || 1000,
batchSize: config.batchSize || 10,
maxConcurrency: config.maxConcurrency || 3,
qualityThreshold: config.qualityThreshold || 0.7,
costBudget: config.costBudget || 10.0,
rateLimitPerMinute: config.rateLimitPerMinute || 60,
enableCaching: config.enableCaching !== false,
outputDirectory: config.outputDirectory || './output'
};
this.synth = new AgenticSynth({
provider: 'gemini',
apiKey: process.env.GEMINI_API_KEY,
model: 'gemini-2.0-flash-exp',
cacheStrategy: this.config.enableCaching ? 'memory' : 'none',
cacheTTL: 3600,
maxRetries: this.config.maxRetries,
timeout: 30000
});
this.metrics = {
totalRequests: 0,
successfulRequests: 0,
failedRequests: 0,
totalDuration: 0,
totalCost: 0,
averageQuality: 0,
cacheHits: 0,
retries: 0,
errors: []
};
// Ensure output directory exists
if (!existsSync(this.config.outputDirectory)) {
mkdirSync(this.config.outputDirectory, { recursive: true });
}
}
// Rate limiting check
private async checkRateLimit(): Promise<void> {
const now = Date.now();
const elapsedMinutes = (now - this.minuteStartTime) / 60000;
if (elapsedMinutes >= 1) {
// Reset counter for new minute
this.requestsThisMinute = 0;
this.minuteStartTime = now;
}
if (this.requestsThisMinute >= this.config.rateLimitPerMinute) {
const waitTime = 60000 - (now - this.minuteStartTime);
console.log(`⏳ Rate limit reached, waiting ${Math.ceil(waitTime / 1000)}s...`);
await new Promise(resolve => setTimeout(resolve, waitTime));
this.requestsThisMinute = 0;
this.minuteStartTime = Date.now();
}
}
// Cost check
private checkCostBudget(): void {
if (this.metrics.totalCost >= this.config.costBudget) {
throw new Error(`Cost budget exceeded: $${this.metrics.totalCost.toFixed(4)} >= $${this.config.costBudget}`);
}
}
// Generate with retry logic
private async generateWithRetry(
options: any,
attempt: number = 1
): Promise<GenerationResult> {
try {
await this.checkRateLimit();
this.checkCostBudget();
this.requestsThisMinute++;
this.metrics.totalRequests++;
const startTime = Date.now();
const result = await this.synth.generateStructured(options);
const duration = Date.now() - startTime;
this.metrics.totalDuration += duration;
this.metrics.successfulRequests++;
if (result.metadata.cached) {
this.metrics.cacheHits++;
}
// Estimate cost (rough approximation)
const estimatedCost = result.metadata.cached ? 0 : 0.0001;
this.metrics.totalCost += estimatedCost;
return result;
} catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
if (attempt < this.config.maxRetries) {
this.metrics.retries++;
console.log(`⚠️ Attempt ${attempt} failed, retrying... (${errorMsg})`);
await new Promise(resolve =>
setTimeout(resolve, this.config.retryDelay * attempt)
);
return this.generateWithRetry(options, attempt + 1);
} else {
this.metrics.failedRequests++;
this.metrics.errors.push({
timestamp: new Date(),
error: errorMsg,
context: options
});
throw error;
}
}
}
// Process a single batch
private async processBatch(
requests: any[],
validator?: QualityValidator
): Promise<GenerationResult[]> {
const results: GenerationResult[] = [];
// Process with concurrency control
for (let i = 0; i < requests.length; i += this.config.maxConcurrency) {
const batch = requests.slice(i, i + this.config.maxConcurrency);
const batchResults = await Promise.allSettled(
batch.map(req => this.generateWithRetry(req))
);
batchResults.forEach((result, idx) => {
if (result.status === 'fulfilled') {
const genResult = result.value;
// Validate quality if validator provided
if (validator) {
const validation = validator.validate(genResult.data);
if (validation.valid) {
results.push(genResult);
} else {
console.log(`⚠️ Quality validation failed (score: ${validation.score.toFixed(2)})`);
console.log(` Issues: ${validation.issues.join(', ')}`);
}
} else {
results.push(genResult);
}
} else {
console.error(`❌ Batch item ${i + idx} failed:`, result.reason);
}
});
}
return results;
}
// Main pipeline execution
async run(
requests: any[],
validator?: QualityValidator
): Promise<GenerationResult[]> {
console.log('🏭 Starting Production Pipeline\n');
console.log('=' .repeat(70));
console.log(`\nConfiguration:`);
console.log(` Total Requests: ${requests.length}`);
console.log(` Batch Size: ${this.config.batchSize}`);
console.log(` Max Concurrency: ${this.config.maxConcurrency}`);
console.log(` Max Retries: ${this.config.maxRetries}`);
console.log(` Cost Budget: $${this.config.costBudget}`);
console.log(` Rate Limit: ${this.config.rateLimitPerMinute}/min`);
console.log(` Caching: ${this.config.enableCaching ? 'Enabled' : 'Disabled'}`);
console.log(` Output: ${this.config.outputDirectory}`);
console.log('\n' + '=' .repeat(70) + '\n');
const startTime = Date.now();
const allResults: GenerationResult[] = [];
// Split into batches
const batches = [];
for (let i = 0; i < requests.length; i += this.config.batchSize) {
batches.push(requests.slice(i, i + this.config.batchSize));
}
console.log(`📦 Processing ${batches.length} batches...\n`);
// Process each batch
for (let i = 0; i < batches.length; i++) {
console.log(`\nBatch ${i + 1}/${batches.length} (${batches[i].length} items)`);
console.log('─'.repeat(70));
try {
const batchResults = await this.processBatch(batches[i], validator);
allResults.push(...batchResults);
console.log(`✓ Batch complete: ${batchResults.length}/${batches[i].length} successful`);
console.log(` Cost so far: $${this.metrics.totalCost.toFixed(4)}`);
console.log(` Cache hits: ${this.metrics.cacheHits}`);
} catch (error) {
console.error(`✗ Batch failed:`, error instanceof Error ? error.message : 'Unknown error');
if (error instanceof Error && error.message.includes('budget')) {
console.log('\n⚠ Cost budget exceeded, stopping pipeline...');
break;
}
}
}
const totalTime = Date.now() - startTime;
// Save results
await this.saveResults(allResults);
// Display metrics
this.displayMetrics(totalTime);
return allResults;
}
// Save results to disk
private async saveResults(results: GenerationResult[]): Promise<void> {
try {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `generation-${timestamp}.json`;
const filepath = join(this.config.outputDirectory, filename);
const output = {
timestamp: new Date(),
results: results.map(r => r.data),
metadata: {
count: results.length,
metrics: this.metrics
}
};
writeFileSync(filepath, JSON.stringify(output, null, 2));
console.log(`\n💾 Results saved to: ${filepath}`);
// Save metrics separately
const metricsFile = join(this.config.outputDirectory, `metrics-${timestamp}.json`);
writeFileSync(metricsFile, JSON.stringify(this.metrics, null, 2));
console.log(`📊 Metrics saved to: ${metricsFile}`);
} catch (error) {
console.error('⚠️ Failed to save results:', error instanceof Error ? error.message : 'Unknown error');
}
}
// Display comprehensive metrics
private displayMetrics(totalTime: number): void {
console.log('\n\n' + '=' .repeat(70));
console.log('\n📊 PIPELINE METRICS\n');
const successRate = (this.metrics.successfulRequests / this.metrics.totalRequests) * 100;
const avgDuration = this.metrics.totalDuration / this.metrics.successfulRequests;
const cacheHitRate = (this.metrics.cacheHits / this.metrics.totalRequests) * 100;
console.log('Performance:');
console.log(` Total Time: ${(totalTime / 1000).toFixed(2)}s`);
console.log(` Avg Request Time: ${avgDuration.toFixed(0)}ms`);
console.log(` Throughput: ${(this.metrics.successfulRequests / (totalTime / 1000)).toFixed(2)} req/s`);
console.log('\nReliability:');
console.log(` Total Requests: ${this.metrics.totalRequests}`);
console.log(` Successful: ${this.metrics.successfulRequests} (${successRate.toFixed(1)}%)`);
console.log(` Failed: ${this.metrics.failedRequests}`);
console.log(` Retries: ${this.metrics.retries}`);
console.log('\nCost & Efficiency:');
console.log(` Total Cost: $${this.metrics.totalCost.toFixed(4)}`);
console.log(` Avg Cost/Request: $${(this.metrics.totalCost / this.metrics.totalRequests).toFixed(6)}`);
console.log(` Cache Hit Rate: ${cacheHitRate.toFixed(1)}%`);
console.log(` Cost Savings from Cache: $${(this.metrics.cacheHits * 0.0001).toFixed(4)}`);
if (this.metrics.errors.length > 0) {
console.log(`\n⚠ Errors (${this.metrics.errors.length}):`);
this.metrics.errors.slice(0, 5).forEach((err, i) => {
console.log(` ${i + 1}. ${err.error}`);
});
if (this.metrics.errors.length > 5) {
console.log(` ... and ${this.metrics.errors.length - 5} more`);
}
}
console.log('\n' + '=' .repeat(70) + '\n');
}
// Get metrics
getMetrics(): PipelineMetrics {
return { ...this.metrics };
}
}
// Example quality validator
class ProductQualityValidator implements QualityValidator {
validate(data: any[]): { valid: boolean; score: number; issues: string[] } {
const issues: string[] = [];
let score = 1.0;
if (!Array.isArray(data) || data.length === 0) {
return { valid: false, score: 0, issues: ['No data generated'] };
}
data.forEach((item, idx) => {
if (!item.description || item.description.length < 50) {
issues.push(`Item ${idx}: Description too short`);
score -= 0.1;
}
if (!item.key_features || !Array.isArray(item.key_features) || item.key_features.length < 3) {
issues.push(`Item ${idx}: Insufficient features`);
score -= 0.1;
}
});
score = Math.max(0, score);
const valid = score >= 0.7;
return { valid, score, issues };
}
}
// Main execution
async function runProductionPipeline() {
const pipeline = new ProductionPipeline({
maxRetries: 3,
retryDelay: 2000,
batchSize: 5,
maxConcurrency: 2,
qualityThreshold: 0.7,
costBudget: 1.0,
rateLimitPerMinute: 30,
enableCaching: true,
outputDirectory: join(process.cwd(), 'examples', 'output', 'production')
});
const validator = new ProductQualityValidator();
// Generate product data for e-commerce catalog
const requests = [
{
count: 2,
schema: {
id: { type: 'string', required: true },
name: { type: 'string', required: true },
description: { type: 'string', required: true },
key_features: { type: 'array', items: { type: 'string' }, required: true },
price: { type: 'number', required: true, minimum: 10, maximum: 1000 },
category: { type: 'string', enum: ['Electronics', 'Clothing', 'Home', 'Sports'] }
}
}
];
// Duplicate requests to test batching
const allRequests = Array(5).fill(null).map(() => requests[0]);
const results = await pipeline.run(allRequests, validator);
console.log(`\n✅ Pipeline complete! Generated ${results.length} batches of products.\n`);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runProductionPipeline().catch(error => {
console.error('❌ Pipeline failed:', error);
process.exit(1);
});
}
export { ProductionPipeline, ProductQualityValidator, PipelineConfig, PipelineMetrics };

View File

@@ -0,0 +1,25 @@
/**
* BEGINNER TUTORIAL: First DSPy Training
*
* This tutorial demonstrates the basics of training a single model using DSPy.ts
* with agentic-synth for synthetic data generation.
*
* What you'll learn:
* - How to set up a DSPy module
* - Basic configuration options
* - Training a model with examples
* - Evaluating output quality
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/first-dspy-training.ts
*/
import { ChainOfThought } from 'dspy.ts';
declare class ProductDescriptionGenerator extends ChainOfThought {
constructor();
}
declare function runTraining(): Promise<void>;
export { runTraining, ProductDescriptionGenerator };
//# sourceMappingURL=first-dspy-training.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"first-dspy-training.d.ts","sourceRoot":"","sources":["first-dspy-training.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAEH,OAAO,EAAE,cAAc,EAAkB,MAAM,SAAS,CAAC;AAqBzD,cAAM,2BAA4B,SAAQ,cAAc;;CAIvD;AAgDD,iBAAe,WAAW,kBA2EzB;AAUD,OAAO,EAAE,WAAW,EAAE,2BAA2B,EAAE,CAAC"}

View File

@@ -0,0 +1,158 @@
"use strict";
/**
* BEGINNER TUTORIAL: First DSPy Training
*
* This tutorial demonstrates the basics of training a single model using DSPy.ts
* with agentic-synth for synthetic data generation.
*
* What you'll learn:
* - How to set up a DSPy module
* - Basic configuration options
* - Training a model with examples
* - Evaluating output quality
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/first-dspy-training.ts
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.ProductDescriptionGenerator = void 0;
exports.runTraining = runTraining;
const dspy_ts_1 = require("dspy.ts");
// Step 1: Configure the language model
// We'll use Gemini as it's fast and cost-effective for learning
const lm = new dspy_ts_1.LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: 0.7, // Controls randomness (0 = deterministic, 1 = creative)
});
// Step 2: Define the signature for our task
// This tells DSPy what inputs we expect and what outputs we want
const productDescriptionSignature = {
input: 'product_name: string, category: string',
output: 'description: string, key_features: string[]',
description: 'Generate compelling product descriptions for e-commerce'
};
// Step 3: Create a DSPy module using Chain of Thought
// CoT helps the model reason through the task step-by-step
class ProductDescriptionGenerator extends dspy_ts_1.ChainOfThought {
constructor() {
super(productDescriptionSignature, { lm });
}
}
exports.ProductDescriptionGenerator = ProductDescriptionGenerator;
// Step 4: Prepare training examples
// These examples teach the model what good output looks like
const trainingExamples = [
{
product_name: 'Wireless Bluetooth Headphones',
category: 'Electronics',
description: 'Premium wireless headphones with active noise cancellation and 30-hour battery life',
key_features: ['ANC Technology', '30h Battery', 'Bluetooth 5.0', 'Comfortable Design']
},
{
product_name: 'Organic Green Tea',
category: 'Beverages',
description: 'Hand-picked organic green tea leaves from high-altitude gardens, rich in antioxidants',
key_features: ['100% Organic', 'High Antioxidants', 'Mountain Grown', 'Fair Trade']
},
{
product_name: 'Leather Laptop Bag',
category: 'Accessories',
description: 'Handcrafted genuine leather laptop bag with padded compartment for 15-inch laptops',
key_features: ['Genuine Leather', 'Padded Protection', '15" Laptop Fit', 'Professional Style']
}
];
// Step 5: Simple evaluation function
// This measures how good the generated descriptions are
function evaluateDescription(prediction) {
let score = 0;
// Check if description exists and has good length (50-200 chars)
if (prediction.description &&
prediction.description.length >= 50 &&
prediction.description.length <= 200) {
score += 0.5;
}
// Check if key features are provided (at least 3)
if (prediction.key_features &&
Array.isArray(prediction.key_features) &&
prediction.key_features.length >= 3) {
score += 0.5;
}
return score;
}
// Step 6: Main training function
async function runTraining() {
console.log('🚀 Starting Your First DSPy Training Session\n');
console.log('='.repeat(60));
// Initialize the generator
const generator = new ProductDescriptionGenerator();
console.log('\n📊 Training with', trainingExamples.length, 'examples...\n');
// Train the model by showing it examples
// In a real scenario, you'd use DSPy's optimizers like BootstrapFewShot
for (let i = 0; i < trainingExamples.length; i++) {
const example = trainingExamples[i];
console.log(`Example ${i + 1}/${trainingExamples.length}:`);
console.log(` Product: ${example.product_name}`);
console.log(` Category: ${example.category}`);
console.log(` ✓ Learned pattern\n`);
}
console.log('✅ Training complete!\n');
console.log('='.repeat(60));
// Step 7: Test the trained model
console.log('\n🧪 Testing the model with new products:\n');
const testCases = [
{ product_name: 'Smart Watch Pro', category: 'Wearables' },
{ product_name: 'Yoga Mat', category: 'Fitness' },
{ product_name: 'Coffee Maker', category: 'Kitchen Appliances' }
];
let totalScore = 0;
for (const testCase of testCases) {
try {
console.log(`\n📦 Product: ${testCase.product_name}`);
console.log(` Category: ${testCase.category}`);
// Generate description
const result = await generator.forward(testCase);
// Evaluate quality
const score = evaluateDescription(result);
totalScore += score;
console.log(`\n Generated Description:`);
console.log(` ${result.description}`);
console.log(`\n Key Features:`);
if (Array.isArray(result.key_features)) {
result.key_features.forEach(feature => {
console.log(`${feature}`);
});
}
console.log(`\n Quality Score: ${(score * 100).toFixed(0)}%`);
console.log(` ${score >= 0.8 ? '✅' : score >= 0.5 ? '⚠️' : '❌'} ${score >= 0.8 ? 'Excellent' : score >= 0.5 ? 'Good' : 'Needs Improvement'}`);
}
catch (error) {
console.error(` ❌ Error: ${error instanceof Error ? error.message : 'Unknown error'}`);
}
}
// Step 8: Summary
const avgScore = totalScore / testCases.length;
console.log('\n' + '='.repeat(60));
console.log('\n📈 Training Summary:');
console.log(` Average Quality: ${(avgScore * 100).toFixed(1)}%`);
console.log(` Tests Passed: ${testCases.length}`);
console.log(` Model: ${lm.model}`);
console.log(` Provider: ${lm.provider}`);
console.log('\n💡 Next Steps:');
console.log(' 1. Try the multi-model comparison example');
console.log(' 2. Experiment with different temperatures');
console.log(' 3. Add more training examples');
console.log(' 4. Customize the evaluation function\n');
}
// Run the training
if (import.meta.url === `file://${process.argv[1]}`) {
runTraining().catch(error => {
console.error('❌ Training failed:', error);
process.exit(1);
});
}
//# sourceMappingURL=first-dspy-training.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"first-dspy-training.js","sourceRoot":"","sources":["first-dspy-training.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;;;;GAiBG;;;AAgKM,kCAAW;AA9JpB,qCAAyD;AAEzD,uCAAuC;AACvC,gEAAgE;AAChE,MAAM,EAAE,GAAG,IAAI,YAAE,CAAC;IAChB,QAAQ,EAAE,cAAc;IACxB,KAAK,EAAE,sBAAsB;IAC7B,MAAM,EAAE,OAAO,CAAC,GAAG,CAAC,cAAc,IAAI,EAAE;IACxC,WAAW,EAAE,GAAG,EAAE,wDAAwD;CAC3E,CAAC,CAAC;AAEH,4CAA4C;AAC5C,iEAAiE;AACjE,MAAM,2BAA2B,GAAG;IAClC,KAAK,EAAE,wCAAwC;IAC/C,MAAM,EAAE,6CAA6C;IACrD,WAAW,EAAE,yDAAyD;CACvE,CAAC;AAEF,sDAAsD;AACtD,2DAA2D;AAC3D,MAAM,2BAA4B,SAAQ,wBAAc;IACtD;QACE,KAAK,CAAC,2BAA2B,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC;IAC7C,CAAC;CACF;AAqIqB,kEAA2B;AAnIjD,oCAAoC;AACpC,6DAA6D;AAC7D,MAAM,gBAAgB,GAAG;IACvB;QACE,YAAY,EAAE,+BAA+B;QAC7C,QAAQ,EAAE,aAAa;QACvB,WAAW,EAAE,qFAAqF;QAClG,YAAY,EAAE,CAAC,gBAAgB,EAAE,aAAa,EAAE,eAAe,EAAE,oBAAoB,CAAC;KACvF;IACD;QACE,YAAY,EAAE,mBAAmB;QACjC,QAAQ,EAAE,WAAW;QACrB,WAAW,EAAE,uFAAuF;QACpG,YAAY,EAAE,CAAC,cAAc,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,YAAY,CAAC;KACpF;IACD;QACE,YAAY,EAAE,oBAAoB;QAClC,QAAQ,EAAE,aAAa;QACvB,WAAW,EAAE,oFAAoF;QACjG,YAAY,EAAE,CAAC,iBAAiB,EAAE,mBAAmB,EAAE,gBAAgB,EAAE,oBAAoB,CAAC;KAC/F;CACF,CAAC;AAEF,qCAAqC;AACrC,wDAAwD;AACxD,SAAS,mBAAmB,CAAC,UAAsB;IACjD,IAAI,KAAK,GAAG,CAAC,CAAC;IAEd,iEAAiE;IACjE,IAAI,UAAU,CAAC,WAAW;QACtB,UAAU,CAAC,WAAW,CAAC,MAAM,IAAI,EAAE;QACnC,UAAU,CAAC,WAAW,CAAC,MAAM,IAAI,GAAG,EAAE,CAAC;QACzC,KAAK,IAAI,GAAG,CAAC;IACf,CAAC;IAED,kDAAkD;IAClD,IAAI,UAAU,CAAC,YAAY;QACvB,KAAK,CAAC,OAAO,CAAC,UAAU,CAAC,YAAY,CAAC;QACtC,UAAU,CAAC,YAAY,CAAC,MAAM,IAAI,CAAC,EAAE,CAAC;QACxC,KAAK,IAAI,GAAG,CAAC;IACf,CAAC;IAED,OAAO,KAAK,CAAC;AACf,CAAC;AAED,iCAAiC;AACjC,KAAK,UAAU,WAAW;IACxB,OAAO,CAAC,GAAG,CAAC,gDAAgD,CAAC,CAAC;IAC9D,OAAO,CAAC,GAAG,CAAC,GAAG,CAAE,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAE7B,2BAA2B;IAC3B,MAAM,SAAS,GAAG,IAAI,2BAA2B,EAAE,CAAC;IAEpD,OAAO,CAAC,GAAG,CAAC,oBAAoB,EAAE,gBAAgB,CAAC,MAAM,EAAE,eAAe,CAAC,CAAC;IAE5E,yCAAyC;IACzC,wEAAwE;IACxE,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,gBAAgB,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACjD,MAAM,OAAO,GAAG,gBAAgB,CAAC,CAAC,CAAC,CAAC;QACpC,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,GAAG,CAAC,IAAI,gBAAgB,CAAC,MAAM,GAAG,CAAC,CAAC;QAC5D,OAAO,CAAC,GAAG,CAAC,cAAc,OAAO,CAAC,YAAY,EAAE,CAAC,CAAC;QAClD,OAAO,CAAC,GAAG,CAAC,eAAe,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC;QAC/C,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,CAAC;IACvC,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,wBAAwB,CAAC,CAAC;IACtC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAE,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAE7B,iCAAiC;IACjC,OAAO,CAAC,GAAG,CAAC,6CAA6C,CAAC,CAAC;IAE3D,MAAM,SAAS,GAAG;QAChB,EAAE,YAAY,EAAE,iBAAiB,EAAE,QAAQ,EAAE,WAAW,EAAE;QAC1D,EAAE,YAAY,EAAE,UAAU,EAAE,QAAQ,EAAE,SAAS,EAAE;QACjD,EAAE,YAAY,EAAE,cAAc,EAAE,QAAQ,EAAE,oBAAoB,EAAE;KACjE,CAAC;IAEF,IAAI,UAAU,GAAG,CAAC,CAAC;IAEnB,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE,CAAC;QACjC,IAAI,CAAC;YACH,OAAO,CAAC,GAAG,CAAC,iBAAiB,QAAQ,CAAC,YAAY,EAAE,CAAC,CAAC;YACtD,OAAO,CAAC,GAAG,CAAC,gBAAgB,QAAQ,CAAC,QAAQ,EAAE,CAAC,CAAC;YAEjD,uBAAuB;YACvB,MAAM,MAAM,GAAG,MAAM,SAAS,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;YAEjD,mBAAmB;YACnB,MAAM,KAAK,GAAG,mBAAmB,CAAC,MAAM,CAAC,CAAC;YAC1C,UAAU,IAAI,KAAK,CAAC;YAEpB,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,CAAC;YAC3C,OAAO,CAAC,GAAG,CAAC,MAAM,MAAM,CAAC,WAAW,EAAE,CAAC,CAAC;YACxC,OAAO,CAAC,GAAG,CAAC,oBAAoB,CAAC,CAAC;YAClC,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,CAAC;gBACvC,MAAM,CAAC,YAAY,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE;oBACpC,OAAO,CAAC,GAAG,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;gBACjC,CAAC,CAAC,CAAC;YACL,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,KAAK,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;YAChE,OAAO,CAAC,GAAG,CAAC,MAAM,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,mBAAmB,EAAE,CAAC,CAAC;QAElJ,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,KAAK,CAAC,eAAe,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,eAAe,EAAE,CAAC,CAAC;QAC3F,CAAC;IACH,CAAC;IAED,kBAAkB;IAClB,MAAM,QAAQ,GAAG,UAAU,GAAG,SAAS,CAAC,MAAM,CAAC;IAC/C,OAAO,CAAC,GAAG,CAAC,IAAI,GAAG,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IACnC,OAAO,CAAC,GAAG,CAAC,wBAAwB,CAAC,CAAC;IACtC,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,QAAQ,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;IACnE,OAAO,CAAC,GAAG,CAAC,oBAAoB,SAAS,CAAC,MAAM,EAAE,CAAC,CAAC;IACpD,OAAO,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC,KAAK,EAAE,CAAC,CAAC;IACrC,OAAO,CAAC,GAAG,CAAC,gBAAgB,EAAE,CAAC,QAAQ,EAAE,CAAC,CAAC;IAE3C,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC,CAAC;IAChC,OAAO,CAAC,GAAG,CAAC,8CAA8C,CAAC,CAAC;IAC5D,OAAO,CAAC,GAAG,CAAC,8CAA8C,CAAC,CAAC;IAC5D,OAAO,CAAC,GAAG,CAAC,kCAAkC,CAAC,CAAC;IAChD,OAAO,CAAC,GAAG,CAAC,2CAA2C,CAAC,CAAC;AAC3D,CAAC;AAED,mBAAmB;AACnB,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,KAAK,UAAU,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC;IACpD,WAAW,EAAE,CAAC,KAAK,CAAC,KAAK,CAAC,EAAE;QAC1B,OAAO,CAAC,KAAK,CAAC,oBAAoB,EAAE,KAAK,CAAC,CAAC;QAC3C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC,CAAC,CAAC;AACL,CAAC"}

View File

@@ -0,0 +1,178 @@
/**
* BEGINNER TUTORIAL: First DSPy Training
*
* This tutorial demonstrates the basics of training a single model using DSPy.ts
* with agentic-synth for synthetic data generation.
*
* What you'll learn:
* - How to set up a DSPy module
* - Basic configuration options
* - Training a model with examples
* - Evaluating output quality
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/first-dspy-training.ts
*/
import { ChainOfThought, LM, Prediction } from 'dspy.ts';
// Step 1: Configure the language model
// We'll use Gemini as it's fast and cost-effective for learning
const lm = new LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: 0.7, // Controls randomness (0 = deterministic, 1 = creative)
});
// Step 2: Define the signature for our task
// This tells DSPy what inputs we expect and what outputs we want
const productDescriptionSignature = {
input: 'product_name: string, category: string',
output: 'description: string, key_features: string[]',
description: 'Generate compelling product descriptions for e-commerce'
};
// Step 3: Create a DSPy module using Chain of Thought
// CoT helps the model reason through the task step-by-step
class ProductDescriptionGenerator extends ChainOfThought {
constructor() {
super(productDescriptionSignature, { lm });
}
}
// Step 4: Prepare training examples
// These examples teach the model what good output looks like
const trainingExamples = [
{
product_name: 'Wireless Bluetooth Headphones',
category: 'Electronics',
description: 'Premium wireless headphones with active noise cancellation and 30-hour battery life',
key_features: ['ANC Technology', '30h Battery', 'Bluetooth 5.0', 'Comfortable Design']
},
{
product_name: 'Organic Green Tea',
category: 'Beverages',
description: 'Hand-picked organic green tea leaves from high-altitude gardens, rich in antioxidants',
key_features: ['100% Organic', 'High Antioxidants', 'Mountain Grown', 'Fair Trade']
},
{
product_name: 'Leather Laptop Bag',
category: 'Accessories',
description: 'Handcrafted genuine leather laptop bag with padded compartment for 15-inch laptops',
key_features: ['Genuine Leather', 'Padded Protection', '15" Laptop Fit', 'Professional Style']
}
];
// Step 5: Simple evaluation function
// This measures how good the generated descriptions are
function evaluateDescription(prediction: Prediction): number {
let score = 0;
// Check if description exists and has good length (50-200 chars)
if (prediction.description &&
prediction.description.length >= 50 &&
prediction.description.length <= 200) {
score += 0.5;
}
// Check if key features are provided (at least 3)
if (prediction.key_features &&
Array.isArray(prediction.key_features) &&
prediction.key_features.length >= 3) {
score += 0.5;
}
return score;
}
// Step 6: Main training function
async function runTraining() {
console.log('🚀 Starting Your First DSPy Training Session\n');
console.log('=' .repeat(60));
// Initialize the generator
const generator = new ProductDescriptionGenerator();
console.log('\n📊 Training with', trainingExamples.length, 'examples...\n');
// Train the model by showing it examples
// In a real scenario, you'd use DSPy's optimizers like BootstrapFewShot
for (let i = 0; i < trainingExamples.length; i++) {
const example = trainingExamples[i];
console.log(`Example ${i + 1}/${trainingExamples.length}:`);
console.log(` Product: ${example.product_name}`);
console.log(` Category: ${example.category}`);
console.log(` ✓ Learned pattern\n`);
}
console.log('✅ Training complete!\n');
console.log('=' .repeat(60));
// Step 7: Test the trained model
console.log('\n🧪 Testing the model with new products:\n');
const testCases = [
{ product_name: 'Smart Watch Pro', category: 'Wearables' },
{ product_name: 'Yoga Mat', category: 'Fitness' },
{ product_name: 'Coffee Maker', category: 'Kitchen Appliances' }
];
let totalScore = 0;
for (const testCase of testCases) {
try {
console.log(`\n📦 Product: ${testCase.product_name}`);
console.log(` Category: ${testCase.category}`);
// Generate description
const result = await generator.forward(testCase);
// Evaluate quality
const score = evaluateDescription(result);
totalScore += score;
console.log(`\n Generated Description:`);
console.log(` ${result.description}`);
console.log(`\n Key Features:`);
if (Array.isArray(result.key_features)) {
result.key_features.forEach(feature => {
console.log(`${feature}`);
});
}
console.log(`\n Quality Score: ${(score * 100).toFixed(0)}%`);
console.log(` ${score >= 0.8 ? '✅' : score >= 0.5 ? '⚠️' : '❌'} ${score >= 0.8 ? 'Excellent' : score >= 0.5 ? 'Good' : 'Needs Improvement'}`);
} catch (error) {
console.error(` ❌ Error: ${error instanceof Error ? error.message : 'Unknown error'}`);
}
}
// Step 8: Summary
const avgScore = totalScore / testCases.length;
console.log('\n' + '='.repeat(60));
console.log('\n📈 Training Summary:');
console.log(` Average Quality: ${(avgScore * 100).toFixed(1)}%`);
console.log(` Tests Passed: ${testCases.length}`);
console.log(` Model: ${lm.model}`);
console.log(` Provider: ${lm.provider}`);
console.log('\n💡 Next Steps:');
console.log(' 1. Try the multi-model comparison example');
console.log(' 2. Experiment with different temperatures');
console.log(' 3. Add more training examples');
console.log(' 4. Customize the evaluation function\n');
}
// Run the training
if (import.meta.url === `file://${process.argv[1]}`) {
runTraining().catch(error => {
console.error('❌ Training failed:', error);
process.exit(1);
});
}
export { runTraining, ProductDescriptionGenerator };

View File

@@ -0,0 +1,24 @@
/**
* BEGINNER TUTORIAL: Simple Data Generation
*
* Learn how to generate structured synthetic data with agentic-synth.
* Perfect for creating test data, mock APIs, or prototyping.
*
* What you'll learn:
* - Defining data schemas
* - Generating structured data
* - Saving output to files
* - Working with different formats
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/simple-data-generation.ts
*/
import { AgenticSynth } from '@ruvector/agentic-synth';
declare const synth: AgenticSynth;
declare function generateUserData(): Promise<void>;
declare function generateWithConstraints(): Promise<void>;
export { generateUserData, generateWithConstraints, synth };
//# sourceMappingURL=simple-data-generation.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"simple-data-generation.d.ts","sourceRoot":"","sources":["simple-data-generation.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAEH,OAAO,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AA0CvD,QAAA,MAAM,KAAK,cAMT,CAAC;AAGH,iBAAe,gBAAgB,kBA0H9B;AAGD,iBAAe,uBAAuB,kBAsBrC;AAUD,OAAO,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,KAAK,EAAE,CAAC"}

View File

@@ -0,0 +1,240 @@
"use strict";
/**
* BEGINNER TUTORIAL: Simple Data Generation
*
* Learn how to generate structured synthetic data with agentic-synth.
* Perfect for creating test data, mock APIs, or prototyping.
*
* What you'll learn:
* - Defining data schemas
* - Generating structured data
* - Saving output to files
* - Working with different formats
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/simple-data-generation.ts
*/
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
var desc = Object.getOwnPropertyDescriptor(m, k);
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
desc = { enumerable: true, get: function() { return m[k]; } };
}
Object.defineProperty(o, k2, desc);
}) : (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
o[k2] = m[k];
}));
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
Object.defineProperty(o, "default", { enumerable: true, value: v });
}) : function(o, v) {
o["default"] = v;
});
var __importStar = (this && this.__importStar) || (function () {
var ownKeys = function(o) {
ownKeys = Object.getOwnPropertyNames || function (o) {
var ar = [];
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
return ar;
};
return ownKeys(o);
};
return function (mod) {
if (mod && mod.__esModule) return mod;
var result = {};
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
__setModuleDefault(result, mod);
return result;
};
})();
Object.defineProperty(exports, "__esModule", { value: true });
exports.synth = void 0;
exports.generateUserData = generateUserData;
exports.generateWithConstraints = generateWithConstraints;
const agentic_synth_1 = require("@ruvector/agentic-synth");
const fs_1 = require("fs");
const path_1 = require("path");
// Step 1: Define your data schema
// This is like a blueprint for the data you want to generate
const userSchema = {
// Basic fields with types
id: { type: 'string', required: true },
name: { type: 'string', required: true },
email: { type: 'string', required: true },
age: { type: 'number', required: true, minimum: 18, maximum: 80 },
// Enum fields (restricted choices)
role: {
type: 'string',
required: true,
enum: ['user', 'admin', 'moderator']
},
// Nested object
address: {
type: 'object',
required: false,
properties: {
street: { type: 'string' },
city: { type: 'string' },
country: { type: 'string' },
postalCode: { type: 'string' }
}
},
// Array field
interests: {
type: 'array',
required: false,
items: { type: 'string' }
}
};
// Step 2: Initialize AgenticSynth
// We're using Gemini because it's fast and cost-effective
const synth = new agentic_synth_1.AgenticSynth({
provider: 'gemini',
apiKey: process.env.GEMINI_API_KEY,
model: 'gemini-2.0-flash-exp',
cacheStrategy: 'memory', // Cache results to save API calls
cacheTTL: 3600 // Cache for 1 hour
});
exports.synth = synth;
// Step 3: Main generation function
async function generateUserData() {
console.log('🎯 Simple Data Generation Tutorial\n');
console.log('='.repeat(60));
// Step 3a: Generate a small batch first (5 users)
console.log('\n📊 Generating 5 sample users...\n');
try {
const result = await synth.generateStructured({
count: 5,
schema: userSchema,
format: 'json', // Can also be 'csv' or 'array'
constraints: {
// Additional constraints for more realistic data
emailDomain: '@example.com',
nameFormat: 'FirstName LastName',
countryList: ['USA', 'UK', 'Canada', 'Australia']
}
});
// Step 4: Display the results
console.log('✅ Generation Complete!\n');
console.log(`Generated ${result.metadata.count} users in ${result.metadata.duration}ms`);
console.log(`Provider: ${result.metadata.provider}`);
console.log(`Model: ${result.metadata.model}`);
console.log(`Cached: ${result.metadata.cached ? 'Yes ⚡' : 'No'}\n`);
// Show the generated data
console.log('👥 Generated Users:\n');
result.data.forEach((user, index) => {
console.log(`${index + 1}. ${user.name} (${user.role})`);
console.log(` 📧 ${user.email}`);
console.log(` 🎂 Age: ${user.age}`);
if (user.address) {
console.log(` 🏠 ${user.address.city}, ${user.address.country}`);
}
if (user.interests && user.interests.length > 0) {
console.log(` ❤️ Interests: ${user.interests.join(', ')}`);
}
console.log('');
});
// Step 5: Save to file
const outputDir = (0, path_1.join)(process.cwd(), 'examples', 'output');
const outputFile = (0, path_1.join)(outputDir, 'sample-users.json');
try {
// Create output directory if it doesn't exist
const { mkdirSync } = await Promise.resolve().then(() => __importStar(require('fs')));
mkdirSync(outputDir, { recursive: true });
// Save the data
(0, fs_1.writeFileSync)(outputFile, JSON.stringify(result.data, null, 2));
console.log(`💾 Data saved to: ${outputFile}\n`);
}
catch (error) {
console.warn('⚠️ Could not save file:', error instanceof Error ? error.message : 'Unknown error');
}
// Step 6: Generate a larger batch
console.log('='.repeat(60));
console.log('\n📈 Now generating 20 users (to demonstrate scaling)...\n');
const largeResult = await synth.generateStructured({
count: 20,
schema: userSchema,
format: 'json'
});
console.log('✅ Large batch complete!');
console.log(` Generated: ${largeResult.metadata.count} users`);
console.log(` Time: ${largeResult.metadata.duration}ms`);
console.log(` Cached: ${largeResult.metadata.cached ? 'Yes ⚡' : 'No'}\n`);
// Step 7: Demonstrate CSV format
console.log('='.repeat(60));
console.log('\n📄 Generating data in CSV format...\n');
const csvResult = await synth.generateStructured({
count: 3,
schema: {
id: { type: 'string', required: true },
name: { type: 'string', required: true },
email: { type: 'string', required: true },
role: { type: 'string', required: true }
},
format: 'csv'
});
console.log('CSV Output (first 3 users):');
console.log('─'.repeat(60));
// Note: CSV format will be in the data array as strings
console.log('✅ CSV generation successful\n');
// Step 8: Show statistics
console.log('='.repeat(60));
console.log('\n📊 Session Statistics:');
console.log(` Total users generated: ${result.data.length + largeResult.data.length + csvResult.data.length}`);
console.log(` Total API calls: ${result.metadata.cached ? '1 (cached)' : '2'}`);
console.log(` Total time: ${result.metadata.duration + largeResult.metadata.duration}ms`);
// Step 9: Next steps
console.log('\n💡 What You Can Do Next:');
console.log(' 1. Modify the schema to match your use case');
console.log(' 2. Try different data types (timeseries, events)');
console.log(' 3. Experiment with constraints for more realistic data');
console.log(' 4. Generate thousands of records for load testing');
console.log(' 5. Integrate with your test suite or mock API\n');
}
catch (error) {
console.error('❌ Generation failed:', error instanceof Error ? error.message : 'Unknown error');
// Helpful error messages
if (error instanceof Error) {
if (error.message.includes('API key')) {
console.error('\n💡 Tip: Make sure GEMINI_API_KEY is set in your environment');
}
else if (error.message.includes('schema')) {
console.error('\n💡 Tip: Check your schema definition for errors');
}
}
process.exit(1);
}
}
// Additional helper: Generate with custom constraints
async function generateWithConstraints() {
console.log('\n🎨 Example: Custom Constraints\n');
const result = await synth.generateStructured({
count: 3,
schema: {
productName: { type: 'string', required: true },
price: { type: 'number', required: true, minimum: 10, maximum: 1000 },
category: {
type: 'string',
enum: ['Electronics', 'Clothing', 'Books', 'Food']
},
inStock: { type: 'boolean', required: true }
},
constraints: {
priceFormat: 'USD',
includeDiscounts: true,
realistic: true
}
});
console.log('Generated products:', result.data);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
generateUserData().catch(error => {
console.error('Fatal error:', error);
process.exit(1);
});
}
//# sourceMappingURL=simple-data-generation.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,228 @@
/**
* BEGINNER TUTORIAL: Simple Data Generation
*
* Learn how to generate structured synthetic data with agentic-synth.
* Perfect for creating test data, mock APIs, or prototyping.
*
* What you'll learn:
* - Defining data schemas
* - Generating structured data
* - Saving output to files
* - Working with different formats
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install @ruvector/agentic-synth
*
* Run: npx tsx examples/beginner/simple-data-generation.ts
*/
import { AgenticSynth } from '@ruvector/agentic-synth';
import { writeFileSync } from 'fs';
import { join } from 'path';
// Step 1: Define your data schema
// This is like a blueprint for the data you want to generate
const userSchema = {
// Basic fields with types
id: { type: 'string', required: true },
name: { type: 'string', required: true },
email: { type: 'string', required: true },
age: { type: 'number', required: true, minimum: 18, maximum: 80 },
// Enum fields (restricted choices)
role: {
type: 'string',
required: true,
enum: ['user', 'admin', 'moderator']
},
// Nested object
address: {
type: 'object',
required: false,
properties: {
street: { type: 'string' },
city: { type: 'string' },
country: { type: 'string' },
postalCode: { type: 'string' }
}
},
// Array field
interests: {
type: 'array',
required: false,
items: { type: 'string' }
}
};
// Step 2: Initialize AgenticSynth
// We're using Gemini because it's fast and cost-effective
const synth = new AgenticSynth({
provider: 'gemini',
apiKey: process.env.GEMINI_API_KEY,
model: 'gemini-2.0-flash-exp',
cacheStrategy: 'memory', // Cache results to save API calls
cacheTTL: 3600 // Cache for 1 hour
});
// Step 3: Main generation function
async function generateUserData() {
console.log('🎯 Simple Data Generation Tutorial\n');
console.log('=' .repeat(60));
// Step 3a: Generate a small batch first (5 users)
console.log('\n📊 Generating 5 sample users...\n');
try {
const result = await synth.generateStructured({
count: 5,
schema: userSchema,
format: 'json', // Can also be 'csv' or 'array'
constraints: {
// Additional constraints for more realistic data
emailDomain: '@example.com',
nameFormat: 'FirstName LastName',
countryList: ['USA', 'UK', 'Canada', 'Australia']
}
});
// Step 4: Display the results
console.log('✅ Generation Complete!\n');
console.log(`Generated ${result.metadata.count} users in ${result.metadata.duration}ms`);
console.log(`Provider: ${result.metadata.provider}`);
console.log(`Model: ${result.metadata.model}`);
console.log(`Cached: ${result.metadata.cached ? 'Yes ⚡' : 'No'}\n`);
// Show the generated data
console.log('👥 Generated Users:\n');
result.data.forEach((user: any, index: number) => {
console.log(`${index + 1}. ${user.name} (${user.role})`);
console.log(` 📧 ${user.email}`);
console.log(` 🎂 Age: ${user.age}`);
if (user.address) {
console.log(` 🏠 ${user.address.city}, ${user.address.country}`);
}
if (user.interests && user.interests.length > 0) {
console.log(` ❤️ Interests: ${user.interests.join(', ')}`);
}
console.log('');
});
// Step 5: Save to file
const outputDir = join(process.cwd(), 'examples', 'output');
const outputFile = join(outputDir, 'sample-users.json');
try {
// Create output directory if it doesn't exist
const { mkdirSync } = await import('fs');
mkdirSync(outputDir, { recursive: true });
// Save the data
writeFileSync(outputFile, JSON.stringify(result.data, null, 2));
console.log(`💾 Data saved to: ${outputFile}\n`);
} catch (error) {
console.warn('⚠️ Could not save file:', error instanceof Error ? error.message : 'Unknown error');
}
// Step 6: Generate a larger batch
console.log('=' .repeat(60));
console.log('\n📈 Now generating 20 users (to demonstrate scaling)...\n');
const largeResult = await synth.generateStructured({
count: 20,
schema: userSchema,
format: 'json'
});
console.log('✅ Large batch complete!');
console.log(` Generated: ${largeResult.metadata.count} users`);
console.log(` Time: ${largeResult.metadata.duration}ms`);
console.log(` Cached: ${largeResult.metadata.cached ? 'Yes ⚡' : 'No'}\n`);
// Step 7: Demonstrate CSV format
console.log('=' .repeat(60));
console.log('\n📄 Generating data in CSV format...\n');
const csvResult = await synth.generateStructured({
count: 3,
schema: {
id: { type: 'string', required: true },
name: { type: 'string', required: true },
email: { type: 'string', required: true },
role: { type: 'string', required: true }
},
format: 'csv'
});
console.log('CSV Output (first 3 users):');
console.log('─'.repeat(60));
// Note: CSV format will be in the data array as strings
console.log('✅ CSV generation successful\n');
// Step 8: Show statistics
console.log('=' .repeat(60));
console.log('\n📊 Session Statistics:');
console.log(` Total users generated: ${result.data.length + largeResult.data.length + csvResult.data.length}`);
console.log(` Total API calls: ${result.metadata.cached ? '1 (cached)' : '2'}`);
console.log(` Total time: ${result.metadata.duration + largeResult.metadata.duration}ms`);
// Step 9: Next steps
console.log('\n💡 What You Can Do Next:');
console.log(' 1. Modify the schema to match your use case');
console.log(' 2. Try different data types (timeseries, events)');
console.log(' 3. Experiment with constraints for more realistic data');
console.log(' 4. Generate thousands of records for load testing');
console.log(' 5. Integrate with your test suite or mock API\n');
} catch (error) {
console.error('❌ Generation failed:', error instanceof Error ? error.message : 'Unknown error');
// Helpful error messages
if (error instanceof Error) {
if (error.message.includes('API key')) {
console.error('\n💡 Tip: Make sure GEMINI_API_KEY is set in your environment');
} else if (error.message.includes('schema')) {
console.error('\n💡 Tip: Check your schema definition for errors');
}
}
process.exit(1);
}
}
// Additional helper: Generate with custom constraints
async function generateWithConstraints() {
console.log('\n🎨 Example: Custom Constraints\n');
const result = await synth.generateStructured({
count: 3,
schema: {
productName: { type: 'string', required: true },
price: { type: 'number', required: true, minimum: 10, maximum: 1000 },
category: {
type: 'string',
enum: ['Electronics', 'Clothing', 'Books', 'Food']
},
inStock: { type: 'boolean', required: true }
},
constraints: {
priceFormat: 'USD',
includeDiscounts: true,
realistic: true
}
});
console.log('Generated products:', result.data);
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
generateUserData().catch(error => {
console.error('Fatal error:', error);
process.exit(1);
});
}
export { generateUserData, generateWithConstraints, synth };

View File

@@ -0,0 +1,42 @@
/**
* INTERMEDIATE TUTORIAL: Multi-Model Comparison
*
* Compare multiple AI models (Gemini, Claude, GPT-4) to find the best
* performer for your specific task. Includes benchmarking, cost tracking,
* and performance metrics.
*
* What you'll learn:
* - Running parallel model comparisons
* - Benchmarking quality and speed
* - Tracking costs per model
* - Selecting the best model for production
*
* Prerequisites:
* - Set API keys: GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/multi-model-comparison.ts
*/
import { Prediction } from 'dspy.ts';
interface ModelConfig {
name: string;
provider: string;
model: string;
apiKey: string;
costPer1kTokens: number;
capabilities: string[];
}
declare const models: ModelConfig[];
interface BenchmarkResult {
modelName: string;
qualityScore: number;
avgResponseTime: number;
estimatedCost: number;
successRate: number;
outputs: Prediction[];
errors: string[];
}
declare function benchmarkModel(config: ModelConfig): Promise<BenchmarkResult>;
declare function runComparison(): Promise<BenchmarkResult[]>;
export { runComparison, benchmarkModel, models };
//# sourceMappingURL=multi-model-comparison.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"multi-model-comparison.d.ts","sourceRoot":"","sources":["multi-model-comparison.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AAEH,OAAO,EAAsB,UAAU,EAAE,MAAM,SAAS,CAAC;AAIzD,UAAU,WAAW;IACnB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,eAAe,EAAE,MAAM,CAAC;IACxB,YAAY,EAAE,MAAM,EAAE,CAAC;CACxB;AAGD,QAAA,MAAM,MAAM,EAAE,WAAW,EAyBxB,CAAC;AAGF,UAAU,eAAe;IACvB,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;IACrB,eAAe,EAAE,MAAM,CAAC;IACxB,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,CAAC;IACpB,OAAO,EAAE,UAAU,EAAE,CAAC;IACtB,MAAM,EAAE,MAAM,EAAE,CAAC;CAClB;AAwFD,iBAAe,cAAc,CAAC,MAAM,EAAE,WAAW,GAAG,OAAO,CAAC,eAAe,CAAC,CA0E3E;AAGD,iBAAe,aAAa,+BA4F3B;AAUD,OAAO,EAAE,aAAa,EAAE,cAAc,EAAE,MAAM,EAAE,CAAC"}

View File

@@ -0,0 +1,274 @@
"use strict";
/**
* INTERMEDIATE TUTORIAL: Multi-Model Comparison
*
* Compare multiple AI models (Gemini, Claude, GPT-4) to find the best
* performer for your specific task. Includes benchmarking, cost tracking,
* and performance metrics.
*
* What you'll learn:
* - Running parallel model comparisons
* - Benchmarking quality and speed
* - Tracking costs per model
* - Selecting the best model for production
*
* Prerequisites:
* - Set API keys: GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/multi-model-comparison.ts
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.models = void 0;
exports.runComparison = runComparison;
exports.benchmarkModel = benchmarkModel;
const dspy_ts_1 = require("dspy.ts");
// Available models to compare
const models = [
{
name: 'Gemini Flash',
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
costPer1kTokens: 0.001, // Very cheap
capabilities: ['fast', 'cost-effective', 'reasoning']
},
{
name: 'Claude Sonnet 4',
provider: 'anthropic',
model: 'claude-sonnet-4-20250514',
apiKey: process.env.ANTHROPIC_API_KEY || '',
costPer1kTokens: 0.003, // Medium cost
capabilities: ['high-quality', 'reasoning', 'code']
},
{
name: 'GPT-4 Turbo',
provider: 'openai',
model: 'gpt-4-turbo-preview',
apiKey: process.env.OPENAI_API_KEY || '',
costPer1kTokens: 0.01, // More expensive
capabilities: ['versatile', 'high-quality', 'creative']
}
];
exports.models = models;
// Test cases for comparison
const testCases = [
{
task: 'product_description',
input: {
product_name: 'Wireless Noise-Cancelling Headphones',
category: 'Electronics',
price: 299
},
expectedFeatures: ['noise cancellation', 'wireless', 'battery life']
},
{
task: 'product_description',
input: {
product_name: 'Organic Herbal Tea Collection',
category: 'Beverages',
price: 24
},
expectedFeatures: ['organic', 'herbal', 'health benefits']
},
{
task: 'product_description',
input: {
product_name: 'Professional Camera Tripod',
category: 'Photography',
price: 149
},
expectedFeatures: ['stability', 'adjustable', 'professional']
},
{
task: 'product_description',
input: {
product_name: 'Smart Fitness Tracker',
category: 'Wearables',
price: 79
},
expectedFeatures: ['fitness tracking', 'smart features', 'health monitoring']
}
];
// Quality evaluation function
function evaluateQuality(prediction, testCase) {
let score = 0;
const weights = {
hasDescription: 0.3,
descriptionLength: 0.2,
hasFeatures: 0.2,
featureCount: 0.15,
relevance: 0.15
};
// Check if description exists and is well-formed
if (prediction.description && typeof prediction.description === 'string') {
score += weights.hasDescription;
// Optimal length is 80-200 characters
const length = prediction.description.length;
if (length >= 80 && length <= 200) {
score += weights.descriptionLength;
}
else if (length >= 50 && length <= 250) {
score += weights.descriptionLength * 0.5;
}
}
// Check features
if (prediction.key_features && Array.isArray(prediction.key_features)) {
score += weights.hasFeatures;
// More features is better (up to 5)
const featureCount = Math.min(prediction.key_features.length, 5);
score += weights.featureCount * (featureCount / 5);
}
// Check relevance to expected features
if (prediction.description) {
const descLower = prediction.description.toLowerCase();
const relevantFeatures = testCase.expectedFeatures.filter(feature => descLower.includes(feature.toLowerCase()));
score += weights.relevance * (relevantFeatures.length / testCase.expectedFeatures.length);
}
return score;
}
// Run benchmark for a single model
async function benchmarkModel(config) {
console.log(`\n🔄 Testing ${config.name}...`);
const result = {
modelName: config.name,
qualityScore: 0,
avgResponseTime: 0,
estimatedCost: 0,
successRate: 0,
outputs: [],
errors: []
};
if (!config.apiKey) {
console.log(` ⚠️ API key not found, skipping...`);
result.errors.push('API key not configured');
return result;
}
const lm = new dspy_ts_1.LM({
provider: config.provider,
model: config.model,
apiKey: config.apiKey,
temperature: 0.7
});
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]'
};
const generator = new dspy_ts_1.ChainOfThought(signature, { lm });
const times = [];
let totalScore = 0;
let successCount = 0;
// Run all test cases
for (let i = 0; i < testCases.length; i++) {
const testCase = testCases[i];
try {
const startTime = Date.now();
const prediction = await generator.forward(testCase.input);
const duration = Date.now() - startTime;
times.push(duration);
result.outputs.push(prediction);
const score = evaluateQuality(prediction, testCase);
totalScore += score;
successCount++;
console.log(` ✓ Test ${i + 1}/${testCases.length} - Score: ${(score * 100).toFixed(0)}% - ${duration}ms`);
}
catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
result.errors.push(`Test ${i + 1}: ${errorMsg}`);
console.log(` ✗ Test ${i + 1}/${testCases.length} - Failed: ${errorMsg}`);
}
}
// Calculate metrics
result.avgResponseTime = times.length > 0
? times.reduce((a, b) => a + b, 0) / times.length
: 0;
result.qualityScore = successCount > 0 ? totalScore / testCases.length : 0;
result.successRate = successCount / testCases.length;
// Estimate cost (rough approximation based on avg tokens)
const avgTokens = 500; // Rough estimate
result.estimatedCost = (avgTokens / 1000) * config.costPer1kTokens * testCases.length;
return result;
}
// Main comparison function
async function runComparison() {
console.log('🏆 Multi-Model Comparison Benchmark\n');
console.log('='.repeat(70));
console.log('\nComparing models:');
models.forEach((m, i) => {
console.log(`${i + 1}. ${m.name} - $${m.costPer1kTokens}/1K tokens`);
console.log(` Capabilities: ${m.capabilities.join(', ')}`);
});
console.log(`\nRunning ${testCases.length} test cases per model...\n`);
console.log('='.repeat(70));
// Run all benchmarks in parallel
const results = await Promise.all(models.map(config => benchmarkModel(config)));
// Display results
console.log('\n' + '='.repeat(70));
console.log('\n📊 BENCHMARK RESULTS\n');
// Sort by quality score
const sortedResults = [...results].sort((a, b) => b.qualityScore - a.qualityScore);
console.log('┌─────────────────────┬──────────┬──────────┬──────────┬──────────┐');
console.log('│ Model │ Quality │ Speed │ Cost │ Success │');
console.log('├─────────────────────┼──────────┼──────────┼──────────┼──────────┤');
sortedResults.forEach((result, index) => {
const quality = `${(result.qualityScore * 100).toFixed(1)}%`;
const speed = `${result.avgResponseTime.toFixed(0)}ms`;
const cost = `$${result.estimatedCost.toFixed(4)}`;
const success = `${(result.successRate * 100).toFixed(0)}%`;
const modelName = result.modelName.padEnd(19);
const qualityPad = quality.padStart(8);
const speedPad = speed.padStart(8);
const costPad = cost.padStart(8);
const successPad = success.padStart(8);
const medal = index === 0 ? '🥇' : index === 1 ? '🥈' : index === 2 ? '🥉' : ' ';
console.log(`${medal} ${modelName}${qualityPad}${speedPad}${costPad}${successPad}`);
});
console.log('└─────────────────────┴──────────┴──────────┴──────────┴──────────┘\n');
// Winner analysis
const winner = sortedResults[0];
console.log('🎯 WINNER: ' + winner.modelName);
console.log(` Quality Score: ${(winner.qualityScore * 100).toFixed(1)}%`);
console.log(` Avg Response: ${winner.avgResponseTime.toFixed(0)}ms`);
console.log(` Total Cost: $${winner.estimatedCost.toFixed(4)}`);
console.log(` Success Rate: ${(winner.successRate * 100).toFixed(0)}%\n`);
// Recommendations
console.log('💡 RECOMMENDATIONS:\n');
const fastest = [...results].sort((a, b) => a.avgResponseTime - b.avgResponseTime)[0];
const cheapest = [...results].sort((a, b) => a.estimatedCost - b.estimatedCost)[0];
const mostReliable = [...results].sort((a, b) => b.successRate - a.successRate)[0];
console.log(`⚡ Fastest: ${fastest.modelName} (${fastest.avgResponseTime.toFixed(0)}ms avg)`);
console.log(`💰 Cheapest: ${cheapest.modelName} ($${cheapest.estimatedCost.toFixed(4)} total)`);
console.log(`🎯 Most Reliable: ${mostReliable.modelName} (${(mostReliable.successRate * 100).toFixed(0)}% success)\n`);
console.log('Use case suggestions:');
console.log(' • High-volume/cost-sensitive → ' + cheapest.modelName);
console.log(' • Latency-critical/real-time → ' + fastest.modelName);
console.log(' • Quality-critical/production → ' + winner.modelName + '\n');
// Error report
const errorsExist = results.some(r => r.errors.length > 0);
if (errorsExist) {
console.log('⚠️ ERRORS:\n');
results.forEach(result => {
if (result.errors.length > 0) {
console.log(`${result.modelName}:`);
result.errors.forEach(err => console.log(`${err}`));
console.log('');
}
});
}
console.log('='.repeat(70));
console.log('\n✅ Benchmark complete!\n');
console.log('Next steps:');
console.log(' 1. Configure your production app with the winning model');
console.log(' 2. Set up fallback chains for reliability');
console.log(' 3. Monitor performance in production');
console.log(' 4. Re-run benchmarks periodically as models improve\n');
return results;
}
// Run the comparison
if (import.meta.url === `file://${process.argv[1]}`) {
runComparison().catch(error => {
console.error('❌ Benchmark failed:', error);
process.exit(1);
});
}
//# sourceMappingURL=multi-model-comparison.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,338 @@
/**
* INTERMEDIATE TUTORIAL: Multi-Model Comparison
*
* Compare multiple AI models (Gemini, Claude, GPT-4) to find the best
* performer for your specific task. Includes benchmarking, cost tracking,
* and performance metrics.
*
* What you'll learn:
* - Running parallel model comparisons
* - Benchmarking quality and speed
* - Tracking costs per model
* - Selecting the best model for production
*
* Prerequisites:
* - Set API keys: GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/multi-model-comparison.ts
*/
import { LM, ChainOfThought, Prediction } from 'dspy.ts';
import { AgenticSynth } from '@ruvector/agentic-synth';
// Model configuration with pricing
interface ModelConfig {
name: string;
provider: string;
model: string;
apiKey: string;
costPer1kTokens: number; // Approximate pricing
capabilities: string[];
}
// Available models to compare
const models: ModelConfig[] = [
{
name: 'Gemini Flash',
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
costPer1kTokens: 0.001, // Very cheap
capabilities: ['fast', 'cost-effective', 'reasoning']
},
{
name: 'Claude Sonnet 4',
provider: 'anthropic',
model: 'claude-sonnet-4-20250514',
apiKey: process.env.ANTHROPIC_API_KEY || '',
costPer1kTokens: 0.003, // Medium cost
capabilities: ['high-quality', 'reasoning', 'code']
},
{
name: 'GPT-4 Turbo',
provider: 'openai',
model: 'gpt-4-turbo-preview',
apiKey: process.env.OPENAI_API_KEY || '',
costPer1kTokens: 0.01, // More expensive
capabilities: ['versatile', 'high-quality', 'creative']
}
];
// Benchmark results interface
interface BenchmarkResult {
modelName: string;
qualityScore: number;
avgResponseTime: number;
estimatedCost: number;
successRate: number;
outputs: Prediction[];
errors: string[];
}
// Test cases for comparison
const testCases = [
{
task: 'product_description',
input: {
product_name: 'Wireless Noise-Cancelling Headphones',
category: 'Electronics',
price: 299
},
expectedFeatures: ['noise cancellation', 'wireless', 'battery life']
},
{
task: 'product_description',
input: {
product_name: 'Organic Herbal Tea Collection',
category: 'Beverages',
price: 24
},
expectedFeatures: ['organic', 'herbal', 'health benefits']
},
{
task: 'product_description',
input: {
product_name: 'Professional Camera Tripod',
category: 'Photography',
price: 149
},
expectedFeatures: ['stability', 'adjustable', 'professional']
},
{
task: 'product_description',
input: {
product_name: 'Smart Fitness Tracker',
category: 'Wearables',
price: 79
},
expectedFeatures: ['fitness tracking', 'smart features', 'health monitoring']
}
];
// Quality evaluation function
function evaluateQuality(prediction: Prediction, testCase: typeof testCases[0]): number {
let score = 0;
const weights = {
hasDescription: 0.3,
descriptionLength: 0.2,
hasFeatures: 0.2,
featureCount: 0.15,
relevance: 0.15
};
// Check if description exists and is well-formed
if (prediction.description && typeof prediction.description === 'string') {
score += weights.hasDescription;
// Optimal length is 80-200 characters
const length = prediction.description.length;
if (length >= 80 && length <= 200) {
score += weights.descriptionLength;
} else if (length >= 50 && length <= 250) {
score += weights.descriptionLength * 0.5;
}
}
// Check features
if (prediction.key_features && Array.isArray(prediction.key_features)) {
score += weights.hasFeatures;
// More features is better (up to 5)
const featureCount = Math.min(prediction.key_features.length, 5);
score += weights.featureCount * (featureCount / 5);
}
// Check relevance to expected features
if (prediction.description) {
const descLower = prediction.description.toLowerCase();
const relevantFeatures = testCase.expectedFeatures.filter(feature =>
descLower.includes(feature.toLowerCase())
);
score += weights.relevance * (relevantFeatures.length / testCase.expectedFeatures.length);
}
return score;
}
// Run benchmark for a single model
async function benchmarkModel(config: ModelConfig): Promise<BenchmarkResult> {
console.log(`\n🔄 Testing ${config.name}...`);
const result: BenchmarkResult = {
modelName: config.name,
qualityScore: 0,
avgResponseTime: 0,
estimatedCost: 0,
successRate: 0,
outputs: [],
errors: []
};
if (!config.apiKey) {
console.log(` ⚠️ API key not found, skipping...`);
result.errors.push('API key not configured');
return result;
}
const lm = new LM({
provider: config.provider as any,
model: config.model,
apiKey: config.apiKey,
temperature: 0.7
});
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]'
};
const generator = new ChainOfThought(signature, { lm });
const times: number[] = [];
let totalScore = 0;
let successCount = 0;
// Run all test cases
for (let i = 0; i < testCases.length; i++) {
const testCase = testCases[i];
try {
const startTime = Date.now();
const prediction = await generator.forward(testCase.input);
const duration = Date.now() - startTime;
times.push(duration);
result.outputs.push(prediction);
const score = evaluateQuality(prediction, testCase);
totalScore += score;
successCount++;
console.log(` ✓ Test ${i + 1}/${testCases.length} - Score: ${(score * 100).toFixed(0)}% - ${duration}ms`);
} catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
result.errors.push(`Test ${i + 1}: ${errorMsg}`);
console.log(` ✗ Test ${i + 1}/${testCases.length} - Failed: ${errorMsg}`);
}
}
// Calculate metrics
result.avgResponseTime = times.length > 0
? times.reduce((a, b) => a + b, 0) / times.length
: 0;
result.qualityScore = successCount > 0 ? totalScore / testCases.length : 0;
result.successRate = successCount / testCases.length;
// Estimate cost (rough approximation based on avg tokens)
const avgTokens = 500; // Rough estimate
result.estimatedCost = (avgTokens / 1000) * config.costPer1kTokens * testCases.length;
return result;
}
// Main comparison function
async function runComparison() {
console.log('🏆 Multi-Model Comparison Benchmark\n');
console.log('=' .repeat(70));
console.log('\nComparing models:');
models.forEach((m, i) => {
console.log(`${i + 1}. ${m.name} - $${m.costPer1kTokens}/1K tokens`);
console.log(` Capabilities: ${m.capabilities.join(', ')}`);
});
console.log(`\nRunning ${testCases.length} test cases per model...\n`);
console.log('=' .repeat(70));
// Run all benchmarks in parallel
const results = await Promise.all(
models.map(config => benchmarkModel(config))
);
// Display results
console.log('\n' + '=' .repeat(70));
console.log('\n📊 BENCHMARK RESULTS\n');
// Sort by quality score
const sortedResults = [...results].sort((a, b) => b.qualityScore - a.qualityScore);
console.log('┌─────────────────────┬──────────┬──────────┬──────────┬──────────┐');
console.log('│ Model │ Quality │ Speed │ Cost │ Success │');
console.log('├─────────────────────┼──────────┼──────────┼──────────┼──────────┤');
sortedResults.forEach((result, index) => {
const quality = `${(result.qualityScore * 100).toFixed(1)}%`;
const speed = `${result.avgResponseTime.toFixed(0)}ms`;
const cost = `$${result.estimatedCost.toFixed(4)}`;
const success = `${(result.successRate * 100).toFixed(0)}%`;
const modelName = result.modelName.padEnd(19);
const qualityPad = quality.padStart(8);
const speedPad = speed.padStart(8);
const costPad = cost.padStart(8);
const successPad = success.padStart(8);
const medal = index === 0 ? '🥇' : index === 1 ? '🥈' : index === 2 ? '🥉' : ' ';
console.log(`${medal} ${modelName}${qualityPad}${speedPad}${costPad}${successPad}`);
});
console.log('└─────────────────────┴──────────┴──────────┴──────────┴──────────┘\n');
// Winner analysis
const winner = sortedResults[0];
console.log('🎯 WINNER: ' + winner.modelName);
console.log(` Quality Score: ${(winner.qualityScore * 100).toFixed(1)}%`);
console.log(` Avg Response: ${winner.avgResponseTime.toFixed(0)}ms`);
console.log(` Total Cost: $${winner.estimatedCost.toFixed(4)}`);
console.log(` Success Rate: ${(winner.successRate * 100).toFixed(0)}%\n`);
// Recommendations
console.log('💡 RECOMMENDATIONS:\n');
const fastest = [...results].sort((a, b) => a.avgResponseTime - b.avgResponseTime)[0];
const cheapest = [...results].sort((a, b) => a.estimatedCost - b.estimatedCost)[0];
const mostReliable = [...results].sort((a, b) => b.successRate - a.successRate)[0];
console.log(`⚡ Fastest: ${fastest.modelName} (${fastest.avgResponseTime.toFixed(0)}ms avg)`);
console.log(`💰 Cheapest: ${cheapest.modelName} ($${cheapest.estimatedCost.toFixed(4)} total)`);
console.log(`🎯 Most Reliable: ${mostReliable.modelName} (${(mostReliable.successRate * 100).toFixed(0)}% success)\n`);
console.log('Use case suggestions:');
console.log(' • High-volume/cost-sensitive → ' + cheapest.modelName);
console.log(' • Latency-critical/real-time → ' + fastest.modelName);
console.log(' • Quality-critical/production → ' + winner.modelName + '\n');
// Error report
const errorsExist = results.some(r => r.errors.length > 0);
if (errorsExist) {
console.log('⚠️ ERRORS:\n');
results.forEach(result => {
if (result.errors.length > 0) {
console.log(`${result.modelName}:`);
result.errors.forEach(err => console.log(`${err}`));
console.log('');
}
});
}
console.log('=' .repeat(70));
console.log('\n✅ Benchmark complete!\n');
console.log('Next steps:');
console.log(' 1. Configure your production app with the winning model');
console.log(' 2. Set up fallback chains for reliability');
console.log(' 3. Monitor performance in production');
console.log(' 4. Re-run benchmarks periodically as models improve\n');
return results;
}
// Run the comparison
if (import.meta.url === `file://${process.argv[1]}`) {
runComparison().catch(error => {
console.error('❌ Benchmark failed:', error);
process.exit(1);
});
}
export { runComparison, benchmarkModel, models };

View File

@@ -0,0 +1,57 @@
/**
* INTERMEDIATE TUTORIAL: Self-Learning System
*
* Build an adaptive AI system that improves its output quality over time
* through feedback loops and pattern recognition. This demonstrates how
* to create systems that learn from their mistakes and successes.
*
* What you'll learn:
* - Building feedback loops
* - Tracking quality improvements
* - Adaptive prompt engineering
* - Learning from examples
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/self-learning-system.ts
*/
import { Prediction } from 'dspy.ts';
interface LearningConfig {
targetQualityThreshold: number;
maxIterations: number;
improvementRate: number;
minImprovement: number;
}
interface Feedback {
quality: number;
strengths: string[];
weaknesses: string[];
suggestions: string[];
}
interface LearningEntry {
iteration: number;
quality: number;
output: Prediction;
feedback: Feedback;
promptModifications: string[];
timestamp: Date;
}
declare class SelfLearningGenerator {
private lm;
private history;
private config;
private basePrompt;
private currentPromptAdditions;
constructor(config?: Partial<LearningConfig>);
private evaluateOutput;
private adaptPrompt;
private generate;
learn(input: any, criteria?: any): Promise<void>;
private displaySummary;
getLearnedImprovements(): string[];
getHistory(): LearningEntry[];
}
export { SelfLearningGenerator, LearningConfig, LearningEntry };
//# sourceMappingURL=self-learning-system.d.ts.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"self-learning-system.d.ts","sourceRoot":"","sources":["self-learning-system.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AAEH,OAAO,EAAsB,UAAU,EAAE,MAAM,SAAS,CAAC;AAGzD,UAAU,cAAc;IACtB,sBAAsB,EAAE,MAAM,CAAC;IAC/B,aAAa,EAAE,MAAM,CAAC;IACtB,eAAe,EAAE,MAAM,CAAC;IACxB,cAAc,EAAE,MAAM,CAAC;CACxB;AAGD,UAAU,QAAQ;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,UAAU,EAAE,MAAM,EAAE,CAAC;IACrB,WAAW,EAAE,MAAM,EAAE,CAAC;CACvB;AAGD,UAAU,aAAa;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,UAAU,CAAC;IACnB,QAAQ,EAAE,QAAQ,CAAC;IACnB,mBAAmB,EAAE,MAAM,EAAE,CAAC;IAC9B,SAAS,EAAE,IAAI,CAAC;CACjB;AAGD,cAAM,qBAAqB;IACzB,OAAO,CAAC,EAAE,CAAK;IACf,OAAO,CAAC,OAAO,CAAuB;IACtC,OAAO,CAAC,MAAM,CAAiB;IAC/B,OAAO,CAAC,UAAU,CAAS;IAC3B,OAAO,CAAC,sBAAsB,CAAgB;gBAElC,MAAM,GAAE,OAAO,CAAC,cAAc,CAAM;IAmBhD,OAAO,CAAC,cAAc;IA6EtB,OAAO,CAAC,WAAW;YAuBL,QAAQ;IAiBhB,KAAK,CAAC,KAAK,EAAE,GAAG,EAAE,QAAQ,GAAE,GAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IA6F1D,OAAO,CAAC,cAAc;IA2CtB,sBAAsB,IAAI,MAAM,EAAE;IAKlC,UAAU,IAAI,aAAa,EAAE;CAG9B;AAiCD,OAAO,EAAE,qBAAqB,EAAE,cAAc,EAAE,aAAa,EAAE,CAAC"}

View File

@@ -0,0 +1,300 @@
"use strict";
/**
* INTERMEDIATE TUTORIAL: Self-Learning System
*
* Build an adaptive AI system that improves its output quality over time
* through feedback loops and pattern recognition. This demonstrates how
* to create systems that learn from their mistakes and successes.
*
* What you'll learn:
* - Building feedback loops
* - Tracking quality improvements
* - Adaptive prompt engineering
* - Learning from examples
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/self-learning-system.ts
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.SelfLearningGenerator = void 0;
const dspy_ts_1 = require("dspy.ts");
// Self-learning generator class
class SelfLearningGenerator {
constructor(config = {}) {
this.history = [];
this.currentPromptAdditions = [];
this.config = {
targetQualityThreshold: config.targetQualityThreshold || 0.9,
maxIterations: config.maxIterations || 10,
improvementRate: config.improvementRate || 0.15,
minImprovement: config.minImprovement || 0.02
};
this.lm = new dspy_ts_1.LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: 0.8 // Higher temperature for creativity during learning
});
this.basePrompt = '';
}
// Evaluate the quality of generated output
evaluateOutput(prediction, criteria) {
let quality = 0;
const strengths = [];
const weaknesses = [];
const suggestions = [];
// Check description quality
if (prediction.description) {
const desc = prediction.description;
const length = desc.length;
if (length >= 100 && length <= 200) {
quality += 0.3;
strengths.push('Description length is optimal');
}
else if (length < 50) {
weaknesses.push('Description too short');
suggestions.push('Expand description with more details');
}
else if (length > 250) {
weaknesses.push('Description too verbose');
suggestions.push('Make description more concise');
}
else {
quality += 0.15;
}
// Check for emotional/engaging language
const emotionalWords = ['amazing', 'powerful', 'innovative', 'premium', 'exceptional'];
const hasEmotionalLanguage = emotionalWords.some(word => desc.toLowerCase().includes(word));
if (hasEmotionalLanguage) {
quality += 0.2;
strengths.push('Uses engaging language');
}
else {
weaknesses.push('Could be more engaging');
suggestions.push('Add more descriptive and emotional words');
}
}
else {
weaknesses.push('Missing description');
suggestions.push('Generate a complete description');
}
// Check features
if (prediction.key_features && Array.isArray(prediction.key_features)) {
const features = prediction.key_features;
if (features.length >= 4 && features.length <= 6) {
quality += 0.3;
strengths.push('Optimal number of features');
}
else if (features.length < 3) {
weaknesses.push('Too few features');
suggestions.push('Include at least 4 key features');
}
else {
quality += 0.15;
}
// Check feature quality (should be concise)
const wellFormedFeatures = features.filter(f => f.length >= 10 && f.length <= 50);
if (wellFormedFeatures.length === features.length) {
quality += 0.2;
strengths.push('All features are well-formed');
}
else {
weaknesses.push('Some features need better formatting');
suggestions.push('Keep features concise (10-50 chars)');
}
}
else {
weaknesses.push('Missing features');
suggestions.push('Generate key features list');
}
return { quality, strengths, weaknesses, suggestions };
}
// Adapt prompt based on feedback
adaptPrompt(feedback) {
const modifications = [];
// Add specific instructions based on weaknesses
feedback.suggestions.forEach(suggestion => {
if (suggestion.includes('short')) {
modifications.push('Write detailed descriptions (100-200 characters)');
}
else if (suggestion.includes('verbose')) {
modifications.push('Keep descriptions concise and focused');
}
else if (suggestion.includes('engaging')) {
modifications.push('Use descriptive, engaging language');
}
else if (suggestion.includes('features')) {
modifications.push('Include 4-6 specific, measurable key features');
}
else if (suggestion.includes('concise')) {
modifications.push('Format features as short, punchy statements');
}
});
// Remove duplicates
return [...new Set(modifications)];
}
// Generate with current prompt
async generate(input) {
// Build enhanced signature with learned improvements
const enhancedInstructions = this.currentPromptAdditions.length > 0
? '\n\nImportant guidelines:\n' + this.currentPromptAdditions.map((s, i) => `${i + 1}. ${s}`).join('\n')
: '';
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]',
description: 'Generate compelling product descriptions' + enhancedInstructions
};
const generator = new dspy_ts_1.ChainOfThought(signature, { lm: this.lm });
return await generator.forward(input);
}
// Main learning loop
async learn(input, criteria = {}) {
console.log('🧠 Starting Self-Learning Session\n');
console.log('='.repeat(70));
console.log(`\nTarget Quality: ${(this.config.targetQualityThreshold * 100).toFixed(0)}%`);
console.log(`Max Iterations: ${this.config.maxIterations}`);
console.log(`Input: ${JSON.stringify(input, null, 2)}\n`);
console.log('='.repeat(70) + '\n');
let iteration = 0;
let previousQuality = 0;
while (iteration < this.config.maxIterations) {
iteration++;
console.log(`\n📊 Iteration ${iteration}/${this.config.maxIterations}`);
console.log('─'.repeat(70));
// Generate output
const startTime = Date.now();
const output = await this.generate(input);
const duration = Date.now() - startTime;
// Evaluate
const feedback = this.evaluateOutput(output, criteria);
// Store in history
this.history.push({
iteration,
quality: feedback.quality,
output,
feedback,
promptModifications: [...this.currentPromptAdditions],
timestamp: new Date()
});
// Display results
console.log(`\n⏱️ Generation time: ${duration}ms`);
console.log(`\n📝 Output:`);
console.log(` Description: ${output.description || 'N/A'}`);
if (output.key_features) {
console.log(` Features:`);
output.key_features.forEach((f) => console.log(`${f}`));
}
console.log(`\n📈 Quality: ${(feedback.quality * 100).toFixed(1)}%`);
if (feedback.strengths.length > 0) {
console.log(`\n✅ Strengths:`);
feedback.strengths.forEach(s => console.log(`${s}`));
}
if (feedback.weaknesses.length > 0) {
console.log(`\n⚠️ Weaknesses:`);
feedback.weaknesses.forEach(w => console.log(`${w}`));
}
// Check if target reached
if (feedback.quality >= this.config.targetQualityThreshold) {
console.log(`\n🎯 Target quality reached!`);
break;
}
// Check for improvement
const improvement = feedback.quality - previousQuality;
if (iteration > 1 && improvement < this.config.minImprovement) {
console.log(`\n⚠️ Improvement too small (${(improvement * 100).toFixed(1)}%), stopping...`);
break;
}
// Adapt for next iteration
const modifications = this.adaptPrompt(feedback);
if (modifications.length > 0) {
console.log(`\n🔧 Adapting strategy:`);
modifications.forEach(m => console.log(`${m}`));
// Add new modifications
modifications.forEach(m => {
if (!this.currentPromptAdditions.includes(m)) {
this.currentPromptAdditions.push(m);
}
});
}
previousQuality = feedback.quality;
// Brief pause between iterations
await new Promise(resolve => setTimeout(resolve, 1000));
}
// Final summary
this.displaySummary();
}
// Display learning summary
displaySummary() {
console.log('\n\n' + '='.repeat(70));
console.log('\n🎓 LEARNING SUMMARY\n');
if (this.history.length === 0) {
console.log('No learning history available.\n');
return;
}
const firstQuality = this.history[0].quality;
const lastQuality = this.history[this.history.length - 1].quality;
const improvement = lastQuality - firstQuality;
const improvementPercent = (improvement / firstQuality) * 100;
console.log(`Total Iterations: ${this.history.length}`);
console.log(`Starting Quality: ${(firstQuality * 100).toFixed(1)}%`);
console.log(`Final Quality: ${(lastQuality * 100).toFixed(1)}%`);
console.log(`Improvement: ${improvement >= 0 ? '+' : ''}${(improvement * 100).toFixed(1)}% (${improvementPercent >= 0 ? '+' : ''}${improvementPercent.toFixed(1)}%)`);
console.log(`\n📊 Quality Progression:`);
this.history.forEach(entry => {
const bar = '█'.repeat(Math.floor(entry.quality * 50));
const percent = (entry.quality * 100).toFixed(1);
console.log(` Iteration ${entry.iteration}: ${bar} ${percent}%`);
});
console.log(`\n🔧 Learned Improvements (${this.currentPromptAdditions.length}):`);
this.currentPromptAdditions.forEach((mod, i) => {
console.log(` ${i + 1}. ${mod}`);
});
console.log('\n💡 Key Insights:');
if (improvement > 0) {
console.log(` ✓ System successfully learned and improved`);
console.log(` ✓ Quality increased by ${(improvement * 100).toFixed(1)}%`);
}
console.log(` ✓ Discovered ${this.currentPromptAdditions.length} optimization strategies`);
console.log(` ✓ These improvements can be applied to future generations\n`);
console.log('='.repeat(70) + '\n');
}
// Get the learned prompt modifications
getLearnedImprovements() {
return [...this.currentPromptAdditions];
}
// Get learning history
getHistory() {
return [...this.history];
}
}
exports.SelfLearningGenerator = SelfLearningGenerator;
// Main execution
async function runSelfLearning() {
const generator = new SelfLearningGenerator({
targetQualityThreshold: 0.85,
maxIterations: 8,
improvementRate: 0.15,
minImprovement: 0.03
});
const testProduct = {
product_name: 'Professional DSLR Camera',
category: 'Photography',
price: 1299
};
await generator.learn(testProduct);
// Save learned improvements
const improvements = generator.getLearnedImprovements();
console.log('📝 Learned improvements can be reused:\n');
console.log(JSON.stringify(improvements, null, 2) + '\n');
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runSelfLearning().catch(error => {
console.error('❌ Learning failed:', error);
process.exit(1);
});
}
//# sourceMappingURL=self-learning-system.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,370 @@
/**
* INTERMEDIATE TUTORIAL: Self-Learning System
*
* Build an adaptive AI system that improves its output quality over time
* through feedback loops and pattern recognition. This demonstrates how
* to create systems that learn from their mistakes and successes.
*
* What you'll learn:
* - Building feedback loops
* - Tracking quality improvements
* - Adaptive prompt engineering
* - Learning from examples
*
* Prerequisites:
* - Set GEMINI_API_KEY environment variable
* - npm install dspy.ts @ruvector/agentic-synth
*
* Run: npx tsx examples/intermediate/self-learning-system.ts
*/
import { LM, ChainOfThought, Prediction } from 'dspy.ts';
// Learning session configuration
interface LearningConfig {
targetQualityThreshold: number; // Stop when this quality is reached
maxIterations: number; // Maximum learning iterations
improvementRate: number; // How aggressively to adjust (0.1 = 10% per iteration)
minImprovement: number; // Minimum improvement to continue
}
// Feedback from each iteration
interface Feedback {
quality: number;
strengths: string[];
weaknesses: string[];
suggestions: string[];
}
// Learning history entry
interface LearningEntry {
iteration: number;
quality: number;
output: Prediction;
feedback: Feedback;
promptModifications: string[];
timestamp: Date;
}
// Self-learning generator class
class SelfLearningGenerator {
private lm: LM;
private history: LearningEntry[] = [];
private config: LearningConfig;
private basePrompt: string;
private currentPromptAdditions: string[] = [];
constructor(config: Partial<LearningConfig> = {}) {
this.config = {
targetQualityThreshold: config.targetQualityThreshold || 0.9,
maxIterations: config.maxIterations || 10,
improvementRate: config.improvementRate || 0.15,
minImprovement: config.minImprovement || 0.02
};
this.lm = new LM({
provider: 'google-genai',
model: 'gemini-2.0-flash-exp',
apiKey: process.env.GEMINI_API_KEY || '',
temperature: 0.8 // Higher temperature for creativity during learning
});
this.basePrompt = '';
}
// Evaluate the quality of generated output
private evaluateOutput(prediction: Prediction, criteria: any): Feedback {
let quality = 0;
const strengths: string[] = [];
const weaknesses: string[] = [];
const suggestions: string[] = [];
// Check description quality
if (prediction.description) {
const desc = prediction.description;
const length = desc.length;
if (length >= 100 && length <= 200) {
quality += 0.3;
strengths.push('Description length is optimal');
} else if (length < 50) {
weaknesses.push('Description too short');
suggestions.push('Expand description with more details');
} else if (length > 250) {
weaknesses.push('Description too verbose');
suggestions.push('Make description more concise');
} else {
quality += 0.15;
}
// Check for emotional/engaging language
const emotionalWords = ['amazing', 'powerful', 'innovative', 'premium', 'exceptional'];
const hasEmotionalLanguage = emotionalWords.some(word =>
desc.toLowerCase().includes(word)
);
if (hasEmotionalLanguage) {
quality += 0.2;
strengths.push('Uses engaging language');
} else {
weaknesses.push('Could be more engaging');
suggestions.push('Add more descriptive and emotional words');
}
} else {
weaknesses.push('Missing description');
suggestions.push('Generate a complete description');
}
// Check features
if (prediction.key_features && Array.isArray(prediction.key_features)) {
const features = prediction.key_features;
if (features.length >= 4 && features.length <= 6) {
quality += 0.3;
strengths.push('Optimal number of features');
} else if (features.length < 3) {
weaknesses.push('Too few features');
suggestions.push('Include at least 4 key features');
} else {
quality += 0.15;
}
// Check feature quality (should be concise)
const wellFormedFeatures = features.filter(f =>
f.length >= 10 && f.length <= 50
);
if (wellFormedFeatures.length === features.length) {
quality += 0.2;
strengths.push('All features are well-formed');
} else {
weaknesses.push('Some features need better formatting');
suggestions.push('Keep features concise (10-50 chars)');
}
} else {
weaknesses.push('Missing features');
suggestions.push('Generate key features list');
}
return { quality, strengths, weaknesses, suggestions };
}
// Adapt prompt based on feedback
private adaptPrompt(feedback: Feedback): string[] {
const modifications: string[] = [];
// Add specific instructions based on weaknesses
feedback.suggestions.forEach(suggestion => {
if (suggestion.includes('short')) {
modifications.push('Write detailed descriptions (100-200 characters)');
} else if (suggestion.includes('verbose')) {
modifications.push('Keep descriptions concise and focused');
} else if (suggestion.includes('engaging')) {
modifications.push('Use descriptive, engaging language');
} else if (suggestion.includes('features')) {
modifications.push('Include 4-6 specific, measurable key features');
} else if (suggestion.includes('concise')) {
modifications.push('Format features as short, punchy statements');
}
});
// Remove duplicates
return [...new Set(modifications)];
}
// Generate with current prompt
private async generate(input: any): Promise<Prediction> {
// Build enhanced signature with learned improvements
const enhancedInstructions = this.currentPromptAdditions.length > 0
? '\n\nImportant guidelines:\n' + this.currentPromptAdditions.map((s, i) => `${i + 1}. ${s}`).join('\n')
: '';
const signature = {
input: 'product_name: string, category: string, price: number',
output: 'description: string, key_features: string[]',
description: 'Generate compelling product descriptions' + enhancedInstructions
};
const generator = new ChainOfThought(signature, { lm: this.lm });
return await generator.forward(input);
}
// Main learning loop
async learn(input: any, criteria: any = {}): Promise<void> {
console.log('🧠 Starting Self-Learning Session\n');
console.log('=' .repeat(70));
console.log(`\nTarget Quality: ${(this.config.targetQualityThreshold * 100).toFixed(0)}%`);
console.log(`Max Iterations: ${this.config.maxIterations}`);
console.log(`Input: ${JSON.stringify(input, null, 2)}\n`);
console.log('=' .repeat(70) + '\n');
let iteration = 0;
let previousQuality = 0;
while (iteration < this.config.maxIterations) {
iteration++;
console.log(`\n📊 Iteration ${iteration}/${this.config.maxIterations}`);
console.log('─'.repeat(70));
// Generate output
const startTime = Date.now();
const output = await this.generate(input);
const duration = Date.now() - startTime;
// Evaluate
const feedback = this.evaluateOutput(output, criteria);
// Store in history
this.history.push({
iteration,
quality: feedback.quality,
output,
feedback,
promptModifications: [...this.currentPromptAdditions],
timestamp: new Date()
});
// Display results
console.log(`\n⏱ Generation time: ${duration}ms`);
console.log(`\n📝 Output:`);
console.log(` Description: ${output.description || 'N/A'}`);
if (output.key_features) {
console.log(` Features:`);
output.key_features.forEach((f: string) => console.log(`${f}`));
}
console.log(`\n📈 Quality: ${(feedback.quality * 100).toFixed(1)}%`);
if (feedback.strengths.length > 0) {
console.log(`\n✅ Strengths:`);
feedback.strengths.forEach(s => console.log(`${s}`));
}
if (feedback.weaknesses.length > 0) {
console.log(`\n⚠ Weaknesses:`);
feedback.weaknesses.forEach(w => console.log(`${w}`));
}
// Check if target reached
if (feedback.quality >= this.config.targetQualityThreshold) {
console.log(`\n🎯 Target quality reached!`);
break;
}
// Check for improvement
const improvement = feedback.quality - previousQuality;
if (iteration > 1 && improvement < this.config.minImprovement) {
console.log(`\n⚠ Improvement too small (${(improvement * 100).toFixed(1)}%), stopping...`);
break;
}
// Adapt for next iteration
const modifications = this.adaptPrompt(feedback);
if (modifications.length > 0) {
console.log(`\n🔧 Adapting strategy:`);
modifications.forEach(m => console.log(`${m}`));
// Add new modifications
modifications.forEach(m => {
if (!this.currentPromptAdditions.includes(m)) {
this.currentPromptAdditions.push(m);
}
});
}
previousQuality = feedback.quality;
// Brief pause between iterations
await new Promise(resolve => setTimeout(resolve, 1000));
}
// Final summary
this.displaySummary();
}
// Display learning summary
private displaySummary(): void {
console.log('\n\n' + '=' .repeat(70));
console.log('\n🎓 LEARNING SUMMARY\n');
if (this.history.length === 0) {
console.log('No learning history available.\n');
return;
}
const firstQuality = this.history[0].quality;
const lastQuality = this.history[this.history.length - 1].quality;
const improvement = lastQuality - firstQuality;
const improvementPercent = (improvement / firstQuality) * 100;
console.log(`Total Iterations: ${this.history.length}`);
console.log(`Starting Quality: ${(firstQuality * 100).toFixed(1)}%`);
console.log(`Final Quality: ${(lastQuality * 100).toFixed(1)}%`);
console.log(`Improvement: ${improvement >= 0 ? '+' : ''}${(improvement * 100).toFixed(1)}% (${improvementPercent >= 0 ? '+' : ''}${improvementPercent.toFixed(1)}%)`);
console.log(`\n📊 Quality Progression:`);
this.history.forEach(entry => {
const bar = '█'.repeat(Math.floor(entry.quality * 50));
const percent = (entry.quality * 100).toFixed(1);
console.log(` Iteration ${entry.iteration}: ${bar} ${percent}%`);
});
console.log(`\n🔧 Learned Improvements (${this.currentPromptAdditions.length}):`);
this.currentPromptAdditions.forEach((mod, i) => {
console.log(` ${i + 1}. ${mod}`);
});
console.log('\n💡 Key Insights:');
if (improvement > 0) {
console.log(` ✓ System successfully learned and improved`);
console.log(` ✓ Quality increased by ${(improvement * 100).toFixed(1)}%`);
}
console.log(` ✓ Discovered ${this.currentPromptAdditions.length} optimization strategies`);
console.log(` ✓ These improvements can be applied to future generations\n`);
console.log('=' .repeat(70) + '\n');
}
// Get the learned prompt modifications
getLearnedImprovements(): string[] {
return [...this.currentPromptAdditions];
}
// Get learning history
getHistory(): LearningEntry[] {
return [...this.history];
}
}
// Main execution
async function runSelfLearning() {
const generator = new SelfLearningGenerator({
targetQualityThreshold: 0.85,
maxIterations: 8,
improvementRate: 0.15,
minImprovement: 0.03
});
const testProduct = {
product_name: 'Professional DSLR Camera',
category: 'Photography',
price: 1299
};
await generator.learn(testProduct);
// Save learned improvements
const improvements = generator.getLearnedImprovements();
console.log('📝 Learned improvements can be reused:\n');
console.log(JSON.stringify(improvements, null, 2) + '\n');
}
// Run the example
if (import.meta.url === `file://${process.argv[1]}`) {
runSelfLearning().catch(error => {
console.error('❌ Learning failed:', error);
process.exit(1);
});
}
export { SelfLearningGenerator, LearningConfig, LearningEntry };