Full parallel implementation of typesense and opensearch

2025-09-20 09:40:09 +02:00
parent 54df3c471e
commit f1773873d4
20 changed files with 2869 additions and 290 deletions
--- a/DUAL_WRITE_USAGE.md
+++ b/DUAL_WRITE_USAGE.md
@@ -0,0 +1,188 @@
+# Dual-Write Search Engine Usage Guide
+
+This guide explains how to use the dual-write functionality during the search engine migration.
+
+## 🎛️ Configuration Options
+
+### Environment Variables
+
+```bash
+# Search engine selection
+SEARCH_ENGINE=typesense          # or 'opensearch'
+SEARCH_DUAL_WRITE=false          # or 'true'
+
+# OpenSearch connection (required when using OpenSearch)
+OPENSEARCH_PASSWORD=your_password
+```
+
+### Migration Flow
+
+#### Phase 1: Initial Setup
+```bash
+SEARCH_ENGINE=typesense          # Keep using Typesense
+SEARCH_DUAL_WRITE=false          # Single-write to Typesense only
+```
+
+#### Phase 2: Enable Dual-Write
+```bash
+SEARCH_ENGINE=typesense          # Still reading from Typesense
+SEARCH_DUAL_WRITE=true           # Now writing to BOTH engines
+```
+
+#### Phase 3: Switch to OpenSearch
+```bash
+SEARCH_ENGINE=opensearch         # Now reading from OpenSearch
+SEARCH_DUAL_WRITE=true           # Still writing to both (safety)
+```
+
+#### Phase 4: Complete Migration
+```bash
+SEARCH_ENGINE=opensearch         # Reading from OpenSearch
+SEARCH_DUAL_WRITE=false          # Only writing to OpenSearch
+```
+
+## 🔧 Admin API Endpoints
+
+### Check Current Status
+```bash
+GET /api/admin/search/status
+
+Response:
+{
+  "primaryEngine": "typesense",
+  "dualWrite": false,
+  "typesenseAvailable": true,
+  "openSearchAvailable": true
+}
+```
+
+### Switch Configuration
+```bash
+POST /api/admin/search/configure
+Content-Type: application/json
+
+{
+  "engine": "opensearch",
+  "dualWrite": true
+}
+```
+
+### Quick Actions
+```bash
+# Enable dual-write
+POST /api/admin/search/dual-write/enable
+
+# Disable dual-write
+POST /api/admin/search/dual-write/disable
+
+# Switch to OpenSearch
+POST /api/admin/search/switch/opensearch
+
+# Switch back to Typesense
+POST /api/admin/search/switch/typesense
+
+# Emergency rollback (Typesense only, no dual-write)
+POST /api/admin/search/emergency-rollback
+```
+
+## 🚀 Migration Process Example
+
+### Step 1: Verify OpenSearch is Ready
+```bash
+# Check if OpenSearch is available
+curl http://localhost:8080/api/admin/search/status
+
+# Should show openSearchAvailable: true
+```
+
+### Step 2: Enable Dual-Write
+```bash
+# Enable writing to both engines
+curl -X POST http://localhost:8080/api/admin/search/dual-write/enable
+
+# Or via configuration
+curl -X POST http://localhost:8080/api/admin/search/configure \
+  -H "Content-Type: application/json" \
+  -d '{"engine": "typesense", "dualWrite": true}'
+```
+
+### Step 3: Populate OpenSearch
+At this point, any new/updated stories will be written to both engines.
+For existing data, you may want to trigger a reindex.
+
+### Step 4: Switch to OpenSearch
+```bash
+# Switch primary engine to OpenSearch
+curl -X POST http://localhost:8080/api/admin/search/switch/opensearch
+
+# Check it worked
+curl http://localhost:8080/api/admin/search/status
+# Should show: primaryEngine: "opensearch", dualWrite: true
+```
+
+### Step 5: Test & Validate
+- Test search functionality
+- Verify results match expectations
+- Monitor for any errors
+
+### Step 6: Complete Migration
+```bash
+# Disable dual-write (OpenSearch only)
+curl -X POST http://localhost:8080/api/admin/search/dual-write/disable
+```
+
+## 🚨 Emergency Procedures
+
+### Immediate Rollback
+If OpenSearch has issues:
+```bash
+curl -X POST http://localhost:8080/api/admin/search/emergency-rollback
+```
+This immediately switches to Typesense-only mode.
+
+### Partial Rollback
+Switch back to Typesense but keep dual-write:
+```bash
+curl -X POST http://localhost:8080/api/admin/search/switch/typesense
+```
+
+## 📊 Monitoring
+
+### Check Current Configuration
+```bash
+curl http://localhost:8080/api/admin/search/status
+```
+
+### Application Logs
+Watch for dual-write success/failure messages:
+```bash
+# Successful operations
+2025-09-18 08:00:00 DEBUG SearchMigrationManager - Successfully indexed story 123 in OpenSearch
+2025-09-18 08:00:00 DEBUG SearchMigrationManager - Successfully indexed story 123 in Typesense
+
+# Failed operations (non-critical in dual-write mode)
+2025-09-18 08:00:00 ERROR SearchMigrationManager - Failed to index story 123 in OpenSearch
+```
+
+## ⚠️ Important Notes
+
+1. **Dual-write errors are non-critical** - if one engine fails, the other continues
+2. **Read operations only use primary engine** - no dual-read
+3. **Configuration updates take effect immediately** - no restart required
+4. **Emergency rollback is always available** - safe to experiment
+5. **Both engines must be available** for dual-write to work optimally
+
+## 🔄 Typical Migration Timeline
+
+| Step | Duration | Configuration | Purpose |
+|------|----------|---------------|---------|
+| 1 | - | typesense + dual:false | Current state |
+| 2 | 1 day | typesense + dual:true | Populate OpenSearch |
+| 3 | 1 day | opensearch + dual:true | Test OpenSearch |
+| 4 | - | opensearch + dual:false | Complete migration |
+
+**Total migration time: ~2 days of gradual transition**
+
+---
+
+**Note:** This dual-write mechanism is temporary and will be removed once the migration is complete. See `CLEANUP_CHECKLIST.md` for removal instructions.