shardegger/storycove
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

replacing opensearch with solr

Browse Source
This commit is contained in:
Stefan Hardegger
2025-09-22 09:44:50 +02:00
parent 9e684a956b
commit 87f37567fb
40 changed files with 2000 additions and 3464 deletions
Download Patch File Download Diff File Expand all files Collapse all files

889
OPENSEARCH_MIGRATION_SPECIFICATION.md
View File

@@ -1,889 +0,0 @@
# StoryCove Search Migration Specification: Typesense to OpenSearch
## Executive Summary
This document specifies the migration from Typesense to OpenSearch for the StoryCove application. The migration will be implemented using a parallel approach, maintaining Typesense functionality while gradually transitioning to OpenSearch, ensuring zero downtime and the ability to rollback if needed.
**Migration Goals:**
- Solve random query reliability issues
- Improve complex filtering performance
- Maintain feature parity during transition
- Zero downtime migration
- Improved developer experience
---
## Current State Analysis
### Typesense Implementation Overview
**Service Architecture:**
- `TypesenseService.java` (~2000 lines) - Primary search service
- 3 search indexes: Stories, Authors, Collections
- Multi-library support with dynamic collection names
- Integration with Spring Boot backend
**Core Functionality:**
1. **Full-text Search**: Stories, Authors with complex query building
2. **Random Story Selection**: `_rand()` function with fallback logic
3. **Advanced Filtering**: 15+ filter conditions with boolean logic
4. **Faceting**: Tag aggregations and counts
5. **Autocomplete**: Search suggestions with typeahead
6. **CRUD Operations**: Index/update/delete for all entity types
**Current Issues Identified:**
- `_rand()` function unreliability requiring complex fallback logic
- Complex filter query building with escaping issues
- Limited aggregation capabilities
- Inconsistent API behavior across query patterns
- Multi-collection management complexity
### Data Models and Schema
**Story Index Fields:**
```java
// Core fields
UUID id, String title, String description, String sourceUrl
Integer wordCount, Integer rating, Integer volume
Boolean isRead, LocalDateTime lastReadAt, Integer readingPosition
// Relationships
UUID authorId, String authorName
UUID seriesId, String seriesName
List<String> tagNames
// Metadata
LocalDateTime createdAt, LocalDateTime updatedAt
String coverPath, String sourceDomain
```
**Author Index Fields:**
```java
UUID id, String name, String notes
Integer authorRating, Double averageStoryRating, Integer storyCount
List<String> urls, String avatarImagePath
LocalDateTime createdAt, LocalDateTime updatedAt
```
**Collection Index Fields:**
```java
UUID id, String name, String description
List<String> tagNames, Boolean archived
LocalDateTime createdAt, LocalDateTime updatedAt
Integer storyCount, Integer currentPosition
```
### API Endpoints Current State
**Search Endpoints Analysis:**
**✅ USED by Frontend (Must Implement):**
- `GET /api/stories/search` - Main story search with complex filtering (CRITICAL)
- `GET /api/stories/random` - Random story selection with filters (CRITICAL)
- `GET /api/authors/search-typesense` - Author search (HIGH)
- `GET /api/tags/autocomplete` - Tag suggestions (MEDIUM)
- `POST /api/stories/reindex-typesense` - Admin reindex operations (MEDIUM)
- `POST /api/authors/reindex-typesense` - Admin reindex operations (MEDIUM)
- `POST /api/stories/recreate-typesense-collection` - Admin recreate (MEDIUM)
- `POST /api/authors/recreate-typesense-collection` - Admin recreate (MEDIUM)
**❌ UNUSED by Frontend (Skip Implementation):**
- `GET /api/stories/search/suggestions` - Not used by frontend
- `GET /api/authors/search` - Superseded by typesense version
- `GET /api/series/search` - Not used by frontend
- `GET /api/tags/search` - Superseded by autocomplete
- `POST /api/search/reindex` - Not used by frontend
- `GET /api/search/health` - Not used by frontend
**Scope Reduction: ~40% fewer endpoints to implement**
**Search Parameters (Stories):**
```
query, page, size, authors[], tags[], minRating, maxRating
sortBy, sortDir, facetBy[]
minWordCount, maxWordCount, createdAfter, createdBefore
lastReadAfter, lastReadBefore, unratedOnly, readingStatus
hasReadingProgress, hasCoverImage, sourceDomain, seriesFilter
minTagCount, popularOnly, hiddenGemsOnly
```
---
## Target OpenSearch Architecture
### Service Layer Design
**New Components:**
```
OpenSearchService.java - Primary search service (mirrors TypesenseService API)
OpenSearchConfig.java - Configuration and client setup
SearchMigrationService.java - Handles parallel operation during migration
SearchServiceAdapter.java - Abstraction layer for service switching
```
**Index Strategy:**
- **Single-node deployment** for development/small installations
- **Index-per-library** approach: `stories-{libraryId}`, `authors-{libraryId}`, `collections-{libraryId}`
- **Index templates** for consistent mapping across libraries
- **Aliases** for easy switching and zero-downtime updates
### OpenSearch Index Mappings
**Stories Index Mapping:**
```json
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"story_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "snowball"]
}
}
}
},
"mappings": {
"properties": {
"id": {"type": "keyword"},
"title": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {"keyword": {"type": "keyword"}}
},
"description": {
"type": "text",
"analyzer": "story_analyzer"
},
"authorName": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {"keyword": {"type": "keyword"}}
},
"seriesName": {
"type": "text",
"fields": {"keyword": {"type": "keyword"}}
},
"tagNames": {"type": "keyword"},
"wordCount": {"type": "integer"},
"rating": {"type": "integer"},
"volume": {"type": "integer"},
"isRead": {"type": "boolean"},
"readingPosition": {"type": "integer"},
"lastReadAt": {"type": "date"},
"createdAt": {"type": "date"},
"updatedAt": {"type": "date"},
"coverPath": {"type": "keyword"},
"sourceUrl": {"type": "keyword"},
"sourceDomain": {"type": "keyword"}
}
}
}
```
**Authors Index Mapping:**
```json
{
"mappings": {
"properties": {
"id": {"type": "keyword"},
"name": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {"keyword": {"type": "keyword"}}
},
"notes": {"type": "text"},
"authorRating": {"type": "integer"},
"averageStoryRating": {"type": "float"},
"storyCount": {"type": "integer"},
"urls": {"type": "keyword"},
"avatarImagePath": {"type": "keyword"},
"createdAt": {"type": "date"},
"updatedAt": {"type": "date"}
}
}
}
```
**Collections Index Mapping:**
```json
{
"mappings": {
"properties": {
"id": {"type": "keyword"},
"name": {
"type": "text",
"fields": {"keyword": {"type": "keyword"}}
},
"description": {"type": "text"},
"tagNames": {"type": "keyword"},
"archived": {"type": "boolean"},
"storyCount": {"type": "integer"},
"currentPosition": {"type": "integer"},
"createdAt": {"type": "date"},
"updatedAt": {"type": "date"}
}
}
}
```
### Query Translation Strategy
**Random Story Queries:**
```java
// Typesense (problematic)
String sortBy = seed != null ? "_rand(" + seed + ")" : "_rand()";
// OpenSearch (reliable)
QueryBuilder randomQuery = QueryBuilders.functionScoreQuery(
QueryBuilders.boolQuery().must(filters),
ScoreFunctionBuilders.randomFunction(seed != null ? seed.intValue() : null)
);
```
**Complex Filtering:**
```java
// Build bool query with multiple filter conditions
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery()
.must(QueryBuilders.multiMatchQuery(query, "title", "description", "authorName"))
.filter(QueryBuilders.termsQuery("tagNames", tags))
.filter(QueryBuilders.rangeQuery("wordCount").gte(minWords).lte(maxWords))
.filter(QueryBuilders.rangeQuery("rating").gte(minRating).lte(maxRating));
```
**Faceting/Aggregations:**
```java
// Tags aggregation
AggregationBuilder tagsAgg = AggregationBuilders
.terms("tags")
.field("tagNames")
.size(100);
// Rating ranges
AggregationBuilder ratingRanges = AggregationBuilders
.range("rating_ranges")
.field("rating")
.addRange("unrated", 0, 1)
.addRange("low", 1, 3)
.addRange("high", 4, 6);
```
---
## Revised Implementation Phases (Scope Reduced by 40%)
### Phase 1: Infrastructure Setup (Week 1)
**Objectives:**
- Add OpenSearch to Docker Compose
- Create basic OpenSearch service
- Establish index templates and mappings
- **Focus**: Only stories, authors, and tags indexes (skip series, collections)
**Deliverables:**
1. **Docker Compose Updates:**
```yaml
opensearch:
image: opensearchproject/opensearch:2.11.0
environment:
- discovery.type=single-node
- DISABLE_SECURITY_PLUGIN=true
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx1g
ports:
- "9200:9200"
volumes:
- opensearch_data:/usr/share/opensearch/data
```
2. **OpenSearchConfig.java:**
```java
@Configuration
@ConditionalOnProperty(name = "storycove.opensearch.enabled", havingValue = "true")
public class OpenSearchConfig {
@Bean
public OpenSearchClient openSearchClient() {
// Client configuration
}
}
```
3. **Basic Index Creation:**
- Create index templates for stories, authors, collections
- Implement index creation with proper mappings
- Add health check endpoint
**Success Criteria:**
- OpenSearch container starts successfully
- Basic connectivity established
- Index templates created and validated
### Phase 2: Core Service Implementation (Week 2)
**Objectives:**
- Implement OpenSearchService with core functionality
- Create service abstraction layer
- Implement basic search operations
- **Focus**: Only critical endpoints (stories search, random, authors)
**Deliverables:**
1. **OpenSearchService.java** - Core service implementing:
- `indexStory()`, `updateStory()`, `deleteStory()`
- `searchStories()` with basic query support (CRITICAL)
- `getRandomStoryId()` with reliable seed support (CRITICAL)
- `indexAuthor()`, `updateAuthor()`, `deleteAuthor()`
- `searchAuthors()` for authors page (HIGH)
- `bulkIndexStories()`, `bulkIndexAuthors()` for initial data loading
2. **SearchServiceAdapter.java** - Abstraction layer:
```java
@Service
public class SearchServiceAdapter {
@Autowired(required = false)
private TypesenseService typesenseService;
@Autowired(required = false)
private OpenSearchService openSearchService;
@Value("${storycove.search.provider:typesense}")
private String searchProvider;
public SearchResultDto<StorySearchDto> searchStories(...) {
return "opensearch".equals(searchProvider)
? openSearchService.searchStories(...)
: typesenseService.searchStories(...);
}
}
```
3. **Basic Query Implementation:**
- Full-text search across title/description/author
- Basic filtering (tags, rating, word count)
- Pagination and sorting
**Success Criteria:**
- Basic search functionality working
- Service abstraction layer functional
- Can switch between Typesense and OpenSearch via configuration
### Phase 3: Advanced Features Implementation (Week 3)
**Objectives:**
- Implement complex filtering (all 15+ filter types)
- Add random story functionality
- Implement faceting/aggregations
- Add autocomplete/suggestions
**Deliverables:**
1. **Complex Query Builder:**
- All filter conditions from original implementation
- Date range filtering with proper timezone handling
- Boolean logic for reading status, coverage, series filters
2. **Random Story Implementation:**
```java
public Optional<UUID> getRandomStoryId(String searchQuery, List<String> tags, Long seed, ...) {
BoolQueryBuilder baseQuery = buildFilterQuery(searchQuery, tags, ...);
QueryBuilder randomQuery = QueryBuilders.functionScoreQuery(
baseQuery,
ScoreFunctionBuilders.randomFunction(seed != null ? seed.intValue() : null)
);
SearchRequest request = new SearchRequest("stories-" + getCurrentLibraryId())
.source(new SearchSourceBuilder()
.query(randomQuery)
.size(1)
.fetchSource(new String[]{"id"}, null));
// Execute and return result
}
```
3. **Faceting Implementation:**
- Tag aggregations with counts
- Rating range aggregations
- Author aggregations
- Custom facet builders
4. **Autocomplete Service:**
- Suggest-based implementation using completion fields
- Prefix matching for story titles and author names
**Success Criteria:**
- All filter conditions working correctly
- Random story selection reliable with seed support
- Faceting returns accurate counts
- Autocomplete responsive and accurate
### Phase 4: Data Migration & Parallel Operation (Week 4)
**Objectives:**
- Implement bulk data migration from database
- Enable parallel operation (write to both systems)
- Comprehensive testing of OpenSearch functionality
**Deliverables:**
1. **Migration Service:**
```java
@Service
public class SearchMigrationService {
public void performFullMigration() {
// Migrate all libraries
List<Library> libraries = libraryService.findAll();
for (Library library : libraries) {
migrateLibraryData(library);
}
}
private void migrateLibraryData(Library library) {
// Create indexes for library
// Bulk load stories, authors, collections
// Verify data integrity
}
}
```
2. **Dual-Write Implementation:**
- Modify all entity update operations to write to both systems
- Add configuration flag for dual-write mode
- Error handling for partial failures
3. **Data Validation Tools:**
- Compare search result counts between systems
- Validate random story selection consistency
- Check faceting accuracy
**Success Criteria:**
- Complete data migration with 100% accuracy
- Dual-write operations working without errors
- Search result parity between systems verified
### Phase 5: API Integration & Testing (Week 5)
**Objectives:**
- Update controller endpoints to use OpenSearch
- Comprehensive integration testing
- Performance testing and optimization
**Deliverables:**
1. **Controller Updates:**
- Modify controllers to use SearchServiceAdapter
- Add migration controls for gradual rollout
- Implement A/B testing capability
2. **Integration Tests:**
```java
@SpringBootTest
@TestMethodOrder(OrderAnnotation.class)
public class OpenSearchIntegrationTest {
@Test
@Order(1)
void testBasicSearch() {
// Test basic story search functionality
}
@Test
@Order(2)
void testComplexFiltering() {
// Test all 15+ filter conditions
}
@Test
@Order(3)
void testRandomStory() {
// Test random story with and without seed
}
@Test
@Order(4)
void testFaceting() {
// Test aggregation accuracy
}
}
```
3. **Performance Testing:**
- Load testing with realistic data volumes
- Query performance benchmarking
- Memory usage monitoring
**Success Criteria:**
- All integration tests passing
- Performance meets or exceeds Typesense baseline
- Memory usage within acceptable limits (< 2GB)
### Phase 6: Production Rollout & Monitoring (Week 6)
**Objectives:**
- Production deployment with feature flags
- Gradual user migration with monitoring
- Rollback capability testing
**Deliverables:**
1. **Feature Flag Implementation:**
```java
@Component
public class SearchFeatureFlags {
@Value("${storycove.search.opensearch.enabled:false}")
private boolean openSearchEnabled;
@Value("${storycove.search.opensearch.percentage:0}")
private int rolloutPercentage;
public boolean shouldUseOpenSearch(String userId) {
if (!openSearchEnabled) return false;
return userId.hashCode() % 100 < rolloutPercentage;
}
}
```
2. **Monitoring & Alerting:**
- Query performance metrics
- Error rate monitoring
- Search result accuracy validation
- User experience metrics
3. **Rollback Procedures:**
- Immediate rollback to Typesense capability
- Data consistency verification
- Performance rollback triggers
**Success Criteria:**
- Successful production deployment
- Zero user-facing issues during rollout
- Monitoring showing improved performance
- Rollback procedures validated
### Phase 7: Cleanup & Documentation (Week 7)
**Objectives:**
- Remove Typesense dependencies
- Update documentation
- Performance optimization
**Deliverables:**
1. **Code Cleanup:**
- Remove TypesenseService and related classes
- Clean up Docker Compose configuration
- Remove unused dependencies
2. **Documentation Updates:**
- Update deployment documentation
- Search API documentation
- Troubleshooting guides
3. **Performance Tuning:**
- Index optimization
- Query performance tuning
- Resource allocation optimization
**Success Criteria:**
- Typesense completely removed
- Documentation up to date
- Optimized performance in production
---
## Data Migration Strategy
### Pre-Migration Validation
**Data Integrity Checks:**
1. Count validation: Ensure all stories/authors/collections are present
2. Field validation: Verify all required fields are populated
3. Relationship validation: Check author-story and series-story relationships
4. Library separation: Ensure proper multi-library data isolation
**Migration Process:**
1. **Index Creation:**
```java
// Create indexes with proper mappings for each library
for (Library library : libraries) {
String storiesIndex = "stories-" + library.getId();
createIndexWithMapping(storiesIndex, getStoriesMapping());
createIndexWithMapping("authors-" + library.getId(), getAuthorsMapping());
createIndexWithMapping("collections-" + library.getId(), getCollectionsMapping());
}
```
2. **Bulk Data Loading:**
```java
// Load in batches to manage memory usage
int batchSize = 1000;
List<Story> allStories = storyService.findByLibraryId(libraryId);
for (int i = 0; i < allStories.size(); i += batchSize) {
List<Story> batch = allStories.subList(i, Math.min(i + batchSize, allStories.size()));
List<StoryDocument> documents = batch.stream()
.map(this::convertToSearchDocument)
.collect(Collectors.toList());
bulkIndexStories(documents, "stories-" + libraryId);
}
```
3. **Post-Migration Validation:**
- Count comparison between database and OpenSearch
- Spot-check random records for field accuracy
- Test search functionality with known queries
- Verify faceting counts match expected values
### Rollback Strategy
**Immediate Rollback Triggers:**
- Search error rate > 1%
- Query performance degradation > 50%
- Data inconsistency detected
- Memory usage > 4GB sustained
**Rollback Process:**
1. Update feature flag to disable OpenSearch
2. Verify Typesense still operational
3. Clear OpenSearch indexes to free resources
4. Investigate and document issues
**Data Consistency During Rollback:**
- Continue dual-write during investigation
- Re-sync any missed updates to OpenSearch
- Validate data integrity before retry
---
## Testing Strategy
### Unit Tests
**OpenSearchService Unit Tests:**
```java
@ExtendWith(MockitoExtension.class)
class OpenSearchServiceTest {
@Mock private OpenSearchClient client;
@InjectMocks private OpenSearchService service;
@Test
void testSearchStoriesBasicQuery() {
// Mock OpenSearch response
// Test basic search functionality
}
@Test
void testComplexFilterQuery() {
// Test complex boolean query building
}
@Test
void testRandomStorySelection() {
// Test random query with seed
}
}
```
**Query Builder Tests:**
- Test all 15+ filter conditions
- Validate query structure and parameters
- Test edge cases and null handling
### Integration Tests
**Full Search Integration:**
```java
@SpringBootTest
@Testcontainers
class OpenSearchIntegrationTest {
@Container
static OpenSearchContainer opensearch = new OpenSearchContainer("opensearchproject/opensearch:2.11.0");
@Test
void testEndToEndStorySearch() {
// Insert test data
// Perform search via controller
// Validate results
}
}
```
### Performance Tests
**Load Testing Scenarios:**
1. **Concurrent Search Load:**
- 50 concurrent users performing searches
- Mixed query complexity
- Duration: 10 minutes
2. **Bulk Indexing Performance:**
- Index 10,000 stories in batches
- Measure throughput and memory usage
3. **Random Query Performance:**
- 1000 random story requests with different seeds
- Compare with Typesense baseline
### Acceptance Tests
**Functional Requirements:**
- All existing search functionality preserved
- Random story selection improved reliability
- Faceting accuracy maintained
- Multi-library separation working
**Performance Requirements:**
- Search response time < 100ms for 95th percentile
- Random story selection < 50ms
- Index update operations < 10ms
- Memory usage < 2GB in production
---
## Risk Analysis & Mitigation
### Technical Risks
**Risk: OpenSearch Memory Usage**
- *Probability: Medium*
- *Impact: High*
- *Mitigation: Resource monitoring, index optimization, container limits*
**Risk: Query Performance Regression**
- *Probability: Low*
- *Impact: High*
- *Mitigation: Performance testing, query optimization, caching layer*
**Risk: Data Migration Accuracy**
- *Probability: Low*
- *Impact: Critical*
- *Mitigation: Comprehensive validation, dual-write verification, rollback procedures*
**Risk: Complex Filter Compatibility**
- *Probability: Medium*
- *Impact: Medium*
- *Mitigation: Extensive testing, gradual rollout, feature flags*
### Operational Risks
**Risk: Production Deployment Issues**
- *Probability: Medium*
- *Impact: High*
- *Mitigation: Staging environment testing, gradual rollout, immediate rollback capability*
**Risk: Team Learning Curve**
- *Probability: High*
- *Impact: Low*
- *Mitigation: Documentation, training, gradual responsibility transfer*
### Business Continuity
**Zero-Downtime Requirements:**
- Maintain Typesense during entire migration
- Feature flag-based switching
- Immediate rollback capability
- Health monitoring with automated alerts
---
## Success Criteria
### Functional Requirements ✅
- [ ] All search functionality migrated successfully
- [ ] Random story selection working reliably with seeds
- [ ] Complex filtering (15+ conditions) working accurately
- [ ] Faceting/aggregation results match expected values
- [ ] Multi-library support maintained
- [ ] Autocomplete functionality preserved
### Performance Requirements ✅
- [ ] Search response time ≤ 100ms (95th percentile)
- [ ] Random story selection ≤ 50ms
- [ ] Index operations ≤ 10ms
- [ ] Memory usage ≤ 2GB sustained
- [ ] Zero search downtime during migration
### Technical Requirements ✅
- [ ] Code quality maintained (test coverage ≥ 80%)
- [ ] Documentation updated and comprehensive
- [ ] Monitoring and alerting implemented
- [ ] Rollback procedures tested and validated
- [ ] Typesense dependencies cleanly removed
---
## Timeline Summary
| Phase | Duration | Key Deliverables | Risk Level |
|-------|----------|------------------|------------|
| 1. Infrastructure | 1 week | Docker setup, basic service | Low |
| 2. Core Service | 1 week | Basic search operations | Medium |
| 3. Advanced Features | 1 week | Complex filtering, random queries | High |
| 4. Data Migration | 1 week | Full data migration, dual-write | High |
| 5. API Integration | 1 week | Controller updates, testing | Medium |
| 6. Production Rollout | 1 week | Gradual deployment, monitoring | High |
| 7. Cleanup | 1 week | Remove Typesense, documentation | Low |
**Total Estimated Duration: 7 weeks**
---
## Configuration Management
### Environment Variables
```bash
# OpenSearch Configuration
OPENSEARCH_HOST=opensearch
OPENSEARCH_PORT=9200
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=${OPENSEARCH_PASSWORD}
# Feature Flags
STORYCOVE_OPENSEARCH_ENABLED=true
STORYCOVE_SEARCH_PROVIDER=opensearch
STORYCOVE_SEARCH_DUAL_WRITE=true
STORYCOVE_OPENSEARCH_ROLLOUT_PERCENTAGE=100
# Performance Tuning
OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx2g
STORYCOVE_SEARCH_BATCH_SIZE=1000
STORYCOVE_SEARCH_TIMEOUT=30s
```
### Docker Compose Updates
```yaml
# Add to docker-compose.yml
opensearch:
image: opensearchproject/opensearch:2.11.0
environment:
- discovery.type=single-node
- DISABLE_SECURITY_PLUGIN=true
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx2g
volumes:
- opensearch_data:/usr/share/opensearch/data
networks:
- storycove-network
volumes:
opensearch_data:
```
---
## Conclusion
This specification provides a comprehensive roadmap for migrating StoryCove from Typesense to OpenSearch. The phased approach ensures minimal risk while delivering improved reliability and performance, particularly for random story queries.
The parallel implementation strategy allows for thorough validation and provides confidence in the migration while maintaining the ability to rollback if issues arise. Upon successful completion, StoryCove will have a more robust and scalable search infrastructure that better supports its growth and feature requirements.
**Next Steps:**
1. Review and approve this specification
2. Set up development environment with OpenSearch
3. Begin Phase 1 implementation
4. Establish monitoring and success metrics
5. Execute migration according to timeline
---
*Document Version: 1.0*
*Last Updated: 2025-01-17*
*Author: Claude Code Assistant*

22
backend/pom.xml
View File

@@ -84,9 +84,25 @@
<artifactId>httpclient5</artifactId> <artifactId>httpclient5</artifactId>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.opensearch.client</groupId> <groupId>org.apache.solr</groupId>
<artifactId>opensearch-java</artifactId> <artifactId>solr-solrj</artifactId>
<version>3.2.0</version> <version>9.9.0</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-client</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-http</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-io</artifactId>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.httpcomponents.core5</groupId> <groupId>org.apache.httpcomponents.core5</groupId>

211
backend/src/main/java/com/storycove/config/OpenSearchConfig.java
View File

@@ -1,211 +0,0 @@
package com.storycove.config;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.UsernamePasswordCredentials;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager;
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManagerBuilder;
import org.apache.hc.client5.http.ssl.ClientTlsStrategyBuilder;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.util.Timeout;
import org.opensearch.client.json.jackson.JacksonJsonpMapper;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.transport.OpenSearchTransport;
import org.opensearch.client.transport.httpclient5.ApacheHttpClient5TransportBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.io.FileInputStream;
import java.security.KeyStore;
import java.security.cert.X509Certificate;
@Configuration
public class OpenSearchConfig {
private static final Logger logger = LoggerFactory.getLogger(OpenSearchConfig.class);
private final OpenSearchProperties properties;
public OpenSearchConfig(@Qualifier("openSearchProperties") OpenSearchProperties properties) {
this.properties = properties;
}
@Bean
public OpenSearchClient openSearchClient() throws Exception {
logger.info("Initializing OpenSearch client for profile: {}", properties.getProfile());
// Create credentials provider
BasicCredentialsProvider credentialsProvider = createCredentialsProvider();
// Create SSL context based on environment
SSLContext sslContext = createSSLContext();
// Create connection manager with pooling
PoolingAsyncClientConnectionManager connectionManager = createConnectionManager(sslContext);
// Create custom ObjectMapper for proper date serialization
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(new JavaTimeModule());
objectMapper.disable(com.fasterxml.jackson.databind.SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
// Create the transport with all configurations and custom Jackson mapper
OpenSearchTransport transport = ApacheHttpClient5TransportBuilder
.builder(new HttpHost(properties.getScheme(), properties.getHost(), properties.getPort()))
.setMapper(new JacksonJsonpMapper(objectMapper))
.setHttpClientConfigCallback(httpClientBuilder -> {
// Only set credentials provider if authentication is configured
if (properties.getUsername() != null && !properties.getUsername().isEmpty() &&
properties.getPassword() != null && !properties.getPassword().isEmpty()) {
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
}
httpClientBuilder.setConnectionManager(connectionManager);
// Set timeouts
httpClientBuilder.setDefaultRequestConfig(
org.apache.hc.client5.http.config.RequestConfig.custom()
.setConnectionRequestTimeout(Timeout.ofMilliseconds(properties.getConnection().getTimeout()))
.setResponseTimeout(Timeout.ofMilliseconds(properties.getConnection().getSocketTimeout()))
.build()
);
return httpClientBuilder;
})
.build();
OpenSearchClient client = new OpenSearchClient(transport);
// Test connection
testConnection(client);
return client;
}
private BasicCredentialsProvider createCredentialsProvider() {
BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
// Only set credentials if username and password are provided
if (properties.getUsername() != null && !properties.getUsername().isEmpty() &&
properties.getPassword() != null && !properties.getPassword().isEmpty()) {
credentialsProvider.setCredentials(
new AuthScope(properties.getHost(), properties.getPort()),
new UsernamePasswordCredentials(
properties.getUsername(),
properties.getPassword().toCharArray()
)
);
logger.info("OpenSearch credentials configured for user: {}", properties.getUsername());
} else {
logger.info("OpenSearch running without authentication (no credentials configured)");
}
return credentialsProvider;
}
private SSLContext createSSLContext() throws Exception {
SSLContext sslContext;
if (isProduction() && !properties.getSecurity().isTrustAllCertificates()) {
// Production SSL configuration with proper certificate validation
sslContext = createProductionSSLContext();
} else {
// Development SSL configuration (trust all certificates)
sslContext = createDevelopmentSSLContext();
}
return sslContext;
}
private SSLContext createProductionSSLContext() throws Exception {
logger.info("Configuring production SSL context with certificate validation");
SSLContext sslContext = SSLContext.getInstance("TLS");
// Load custom keystore/truststore if provided
if (properties.getSecurity().getTruststorePath() != null) {
KeyStore trustStore = KeyStore.getInstance("JKS");
try (FileInputStream fis = new FileInputStream(properties.getSecurity().getTruststorePath())) {
trustStore.load(fis, properties.getSecurity().getTruststorePassword().toCharArray());
}
javax.net.ssl.TrustManagerFactory tmf =
javax.net.ssl.TrustManagerFactory.getInstance(javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm());
tmf.init(trustStore);
sslContext.init(null, tmf.getTrustManagers(), null);
} else {
// Use default system SSL context for production
sslContext.init(null, null, null);
}
return sslContext;
}
private SSLContext createDevelopmentSSLContext() throws Exception {
logger.warn("Configuring development SSL context - TRUSTING ALL CERTIFICATES (not for production!)");
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, new TrustManager[] {
new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) {}
public void checkServerTrusted(X509Certificate[] certs, String authType) {}
}
}, null);
return sslContext;
}
private PoolingAsyncClientConnectionManager createConnectionManager(SSLContext sslContext) {
PoolingAsyncClientConnectionManagerBuilder builder = PoolingAsyncClientConnectionManagerBuilder.create();
// Configure TLS strategy
if (properties.getScheme().equals("https")) {
if (isProduction() && properties.getSecurity().isSslVerification()) {
// Production TLS with hostname verification
builder.setTlsStrategy(ClientTlsStrategyBuilder.create()
.setSslContext(sslContext)
.build());
} else {
// Development TLS without hostname verification
builder.setTlsStrategy(ClientTlsStrategyBuilder.create()
.setSslContext(sslContext)
.setHostnameVerifier((hostname, session) -> true)
.build());
}
}
PoolingAsyncClientConnectionManager connectionManager = builder.build();
// Configure connection pool settings
connectionManager.setMaxTotal(properties.getConnection().getMaxConnectionsTotal());
connectionManager.setDefaultMaxPerRoute(properties.getConnection().getMaxConnectionsPerRoute());
return connectionManager;
}
private boolean isProduction() {
return "production".equalsIgnoreCase(properties.getProfile());
}
private void testConnection(OpenSearchClient client) {
try {
var response = client.info();
logger.info("OpenSearch connection successful - Version: {}, Cluster: {}",
response.version().number(),
response.clusterName());
} catch (Exception e) {
logger.warn("OpenSearch connection test failed during initialization: {}", e.getMessage());
logger.debug("OpenSearch connection test full error", e);
// Don't throw exception here - let the client be created and handle failures in service methods
}
}
}

164
backend/src/main/java/com/storycove/config/OpenSearchProperties.java
View File

@@ -1,164 +0,0 @@
package com.storycove.config;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@Component
@ConfigurationProperties(prefix = "storycove.opensearch")
public class OpenSearchProperties {
private String host = "localhost";
private int port = 9200;
private String scheme = "https";
private String username = "admin";
private String password;
private String profile = "development";
private Security security = new Security();
private Connection connection = new Connection();
private Indices indices = new Indices();
private Bulk bulk = new Bulk();
private Health health = new Health();
// Getters and setters
public String getHost() { return host; }
public void setHost(String host) { this.host = host; }
public int getPort() { return port; }
public void setPort(int port) { this.port = port; }
public String getScheme() { return scheme; }
public void setScheme(String scheme) { this.scheme = scheme; }
public String getUsername() { return username; }
public void setUsername(String username) { this.username = username; }
public String getPassword() { return password; }
public void setPassword(String password) { this.password = password; }
public String getProfile() { return profile; }
public void setProfile(String profile) { this.profile = profile; }
public Security getSecurity() { return security; }
public void setSecurity(Security security) { this.security = security; }
public Connection getConnection() { return connection; }
public void setConnection(Connection connection) { this.connection = connection; }
public Indices getIndices() { return indices; }
public void setIndices(Indices indices) { this.indices = indices; }
public Bulk getBulk() { return bulk; }
public void setBulk(Bulk bulk) { this.bulk = bulk; }
public Health getHealth() { return health; }
public void setHealth(Health health) { this.health = health; }
public static class Security {
private boolean sslVerification = false;
private boolean trustAllCertificates = true;
private String keystorePath;
private String keystorePassword;
private String truststorePath;
private String truststorePassword;
// Getters and setters
public boolean isSslVerification() { return sslVerification; }
public void setSslVerification(boolean sslVerification) { this.sslVerification = sslVerification; }
public boolean isTrustAllCertificates() { return trustAllCertificates; }
public void setTrustAllCertificates(boolean trustAllCertificates) { this.trustAllCertificates = trustAllCertificates; }
public String getKeystorePath() { return keystorePath; }
public void setKeystorePath(String keystorePath) { this.keystorePath = keystorePath; }
public String getKeystorePassword() { return keystorePassword; }
public void setKeystorePassword(String keystorePassword) { this.keystorePassword = keystorePassword; }
public String getTruststorePath() { return truststorePath; }
public void setTruststorePath(String truststorePath) { this.truststorePath = truststorePath; }
public String getTruststorePassword() { return truststorePassword; }
public void setTruststorePassword(String truststorePassword) { this.truststorePassword = truststorePassword; }
}
public static class Connection {
private int timeout = 30000;
private int socketTimeout = 60000;
private int maxConnectionsPerRoute = 10;
private int maxConnectionsTotal = 30;
private boolean retryOnFailure = true;
private int maxRetries = 3;
// Getters and setters
public int getTimeout() { return timeout; }
public void setTimeout(int timeout) { this.timeout = timeout; }
public int getSocketTimeout() { return socketTimeout; }
public void setSocketTimeout(int socketTimeout) { this.socketTimeout = socketTimeout; }
public int getMaxConnectionsPerRoute() { return maxConnectionsPerRoute; }
public void setMaxConnectionsPerRoute(int maxConnectionsPerRoute) { this.maxConnectionsPerRoute = maxConnectionsPerRoute; }
public int getMaxConnectionsTotal() { return maxConnectionsTotal; }
public void setMaxConnectionsTotal(int maxConnectionsTotal) { this.maxConnectionsTotal = maxConnectionsTotal; }
public boolean isRetryOnFailure() { return retryOnFailure; }
public void setRetryOnFailure(boolean retryOnFailure) { this.retryOnFailure = retryOnFailure; }
public int getMaxRetries() { return maxRetries; }
public void setMaxRetries(int maxRetries) { this.maxRetries = maxRetries; }
}
public static class Indices {
private int defaultShards = 1;
private int defaultReplicas = 0;
private String refreshInterval = "1s";
// Getters and setters
public int getDefaultShards() { return defaultShards; }
public void setDefaultShards(int defaultShards) { this.defaultShards = defaultShards; }
public int getDefaultReplicas() { return defaultReplicas; }
public void setDefaultReplicas(int defaultReplicas) { this.defaultReplicas = defaultReplicas; }
public String getRefreshInterval() { return refreshInterval; }
public void setRefreshInterval(String refreshInterval) { this.refreshInterval = refreshInterval; }
}
public static class Bulk {
private int actions = 1000;
private long size = 5242880; // 5MB
private int timeout = 10000;
private int concurrentRequests = 1;
// Getters and setters
public int getActions() { return actions; }
public void setActions(int actions) { this.actions = actions; }
public long getSize() { return size; }
public void setSize(long size) { this.size = size; }
public int getTimeout() { return timeout; }
public void setTimeout(int timeout) { this.timeout = timeout; }
public int getConcurrentRequests() { return concurrentRequests; }
public void setConcurrentRequests(int concurrentRequests) { this.concurrentRequests = concurrentRequests; }
}
public static class Health {
private int checkInterval = 30000;
private int slowQueryThreshold = 5000;
private boolean enableMetrics = true;
// Getters and setters
public int getCheckInterval() { return checkInterval; }
public void setCheckInterval(int checkInterval) { this.checkInterval = checkInterval; }
public int getSlowQueryThreshold() { return slowQueryThreshold; }
public void setSlowQueryThreshold(int slowQueryThreshold) { this.slowQueryThreshold = slowQueryThreshold; }
public boolean isEnableMetrics() { return enableMetrics; }
public void setEnableMetrics(boolean enableMetrics) { this.enableMetrics = enableMetrics; }
}
}

57
backend/src/main/java/com/storycove/config/SolrConfig.java Normal file
View File

@@ -0,0 +1,57 @@
package com.storycove.config;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@ConditionalOnProperty(
value = "storycove.search.engine",
havingValue = "solr",
matchIfMissing = false
)
public class SolrConfig {
private static final Logger logger = LoggerFactory.getLogger(SolrConfig.class);
private final SolrProperties properties;
public SolrConfig(SolrProperties properties) {
this.properties = properties;
}
@Bean
public SolrClient solrClient() {
logger.info("Initializing Solr client with URL: {}", properties.getUrl());
HttpSolrClient.Builder builder = new HttpSolrClient.Builder(properties.getUrl())
.withConnectionTimeout(properties.getConnection().getTimeout())
.withSocketTimeout(properties.getConnection().getSocketTimeout());
SolrClient client = builder.build();
logger.info("Solr running without authentication");
// Test connection
testConnection(client);
return client;
}
private void testConnection(SolrClient client) {
try {
// Test connection by pinging the server
var response = client.ping();
logger.info("Solr connection successful - Response time: {}ms",
response.getElapsedTime());
} catch (Exception e) {
logger.warn("Solr connection test failed during initialization: {}", e.getMessage());
logger.debug("Solr connection test full error", e);
// Don't throw exception here - let the client be created and handle failures in service methods
}
}
}

140
backend/src/main/java/com/storycove/config/SolrProperties.java Normal file
View File

@@ -0,0 +1,140 @@
package com.storycove.config;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@Component
@ConfigurationProperties(prefix = "storycove.solr")
public class SolrProperties {
private String url = "http://localhost:8983/solr";
private String username;
private String password;
private Cores cores = new Cores();
private Connection connection = new Connection();
private Query query = new Query();
private Commit commit = new Commit();
private Health health = new Health();
// Getters and setters
public String getUrl() { return url; }
public void setUrl(String url) { this.url = url; }
public String getUsername() { return username; }
public void setUsername(String username) { this.username = username; }
public String getPassword() { return password; }
public void setPassword(String password) { this.password = password; }
public Cores getCores() { return cores; }
public void setCores(Cores cores) { this.cores = cores; }
public Connection getConnection() { return connection; }
public void setConnection(Connection connection) { this.connection = connection; }
public Query getQuery() { return query; }
public void setQuery(Query query) { this.query = query; }
public Commit getCommit() { return commit; }
public void setCommit(Commit commit) { this.commit = commit; }
public Health getHealth() { return health; }
public void setHealth(Health health) { this.health = health; }
public static class Cores {
private String stories = "storycove_stories";
private String authors = "storycove_authors";
// Getters and setters
public String getStories() { return stories; }
public void setStories(String stories) { this.stories = stories; }
public String getAuthors() { return authors; }
public void setAuthors(String authors) { this.authors = authors; }
}
public static class Connection {
private int timeout = 30000;
private int socketTimeout = 60000;
private int maxConnectionsPerRoute = 10;
private int maxConnectionsTotal = 30;
private boolean retryOnFailure = true;
private int maxRetries = 3;
// Getters and setters
public int getTimeout() { return timeout; }
public void setTimeout(int timeout) { this.timeout = timeout; }
public int getSocketTimeout() { return socketTimeout; }
public void setSocketTimeout(int socketTimeout) { this.socketTimeout = socketTimeout; }
public int getMaxConnectionsPerRoute() { return maxConnectionsPerRoute; }
public void setMaxConnectionsPerRoute(int maxConnectionsPerRoute) { this.maxConnectionsPerRoute = maxConnectionsPerRoute; }
public int getMaxConnectionsTotal() { return maxConnectionsTotal; }
public void setMaxConnectionsTotal(int maxConnectionsTotal) { this.maxConnectionsTotal = maxConnectionsTotal; }
public boolean isRetryOnFailure() { return retryOnFailure; }
public void setRetryOnFailure(boolean retryOnFailure) { this.retryOnFailure = retryOnFailure; }
public int getMaxRetries() { return maxRetries; }
public void setMaxRetries(int maxRetries) { this.maxRetries = maxRetries; }
}
public static class Query {
private int defaultRows = 10;
private int maxRows = 1000;
private String defaultOperator = "AND";
private boolean highlight = true;
private boolean facets = true;
// Getters and setters
public int getDefaultRows() { return defaultRows; }
public void setDefaultRows(int defaultRows) { this.defaultRows = defaultRows; }
public int getMaxRows() { return maxRows; }
public void setMaxRows(int maxRows) { this.maxRows = maxRows; }
public String getDefaultOperator() { return defaultOperator; }
public void setDefaultOperator(String defaultOperator) { this.defaultOperator = defaultOperator; }
public boolean isHighlight() { return highlight; }
public void setHighlight(boolean highlight) { this.highlight = highlight; }
public boolean isFacets() { return facets; }
public void setFacets(boolean facets) { this.facets = facets; }
}
public static class Commit {
private boolean softCommit = true;
private int commitWithin = 1000;
private boolean waitSearcher = false;
// Getters and setters
public boolean isSoftCommit() { return softCommit; }
public void setSoftCommit(boolean softCommit) { this.softCommit = softCommit; }
public int getCommitWithin() { return commitWithin; }
public void setCommitWithin(int commitWithin) { this.commitWithin = commitWithin; }
public boolean isWaitSearcher() { return waitSearcher; }
public void setWaitSearcher(boolean waitSearcher) { this.waitSearcher = waitSearcher; }
}
public static class Health {
private int checkInterval = 30000;
private int slowQueryThreshold = 5000;
private boolean enableMetrics = true;
// Getters and setters
public int getCheckInterval() { return checkInterval; }
public void setCheckInterval(int checkInterval) { this.checkInterval = checkInterval; }
public int getSlowQueryThreshold() { return slowQueryThreshold; }
public void setSlowQueryThreshold(int slowQueryThreshold) { this.slowQueryThreshold = slowQueryThreshold; }
public boolean isEnableMetrics() { return enableMetrics; }
public void setEnableMetrics(boolean enableMetrics) { this.enableMetrics = enableMetrics; }
}
}

60
backend/src/main/java/com/storycove/controller/AdminSearchController.java
View File

@@ -3,7 +3,7 @@ package com.storycove.controller;
import com.storycove.entity.Author; import com.storycove.entity.Author;
import com.storycove.entity.Story; import com.storycove.entity.Story;
import com.storycove.service.AuthorService; import com.storycove.service.AuthorService;
import com.storycove.service.OpenSearchService; import com.storycove.service.SolrService;
import com.storycove.service.SearchServiceAdapter; import com.storycove.service.SearchServiceAdapter;
import com.storycove.service.StoryService; import com.storycove.service.StoryService;
import org.slf4j.Logger; import org.slf4j.Logger;
@@ -16,7 +16,7 @@ import java.util.List;
import java.util.Map; import java.util.Map;
/** /**
* Admin controller for managing OpenSearch operations. * Admin controller for managing Solr operations.
* Provides endpoints for reindexing and index management. * Provides endpoints for reindexing and index management.
*/ */
@RestController @RestController
@@ -35,7 +35,7 @@ public class AdminSearchController {
private AuthorService authorService; private AuthorService authorService;
@Autowired(required = false) @Autowired(required = false)
private OpenSearchService openSearchService; private SolrService solrService;
/** /**
* Get current search status * Get current search status
@@ -48,7 +48,7 @@ public class AdminSearchController {
return ResponseEntity.ok(Map.of( return ResponseEntity.ok(Map.of(
"primaryEngine", status.getPrimaryEngine(), "primaryEngine", status.getPrimaryEngine(),
"dualWrite", status.isDualWrite(), "dualWrite", status.isDualWrite(),
"openSearchAvailable", status.isOpenSearchAvailable() "solrAvailable", status.isSolrAvailable()
)); ));
} catch (Exception e) { } catch (Exception e) {
logger.error("Error getting search status", e); logger.error("Error getting search status", e);
@@ -59,17 +59,17 @@ public class AdminSearchController {
} }
/** /**
* Reindex all data in OpenSearch * Reindex all data in Solr
*/ */
@PostMapping("/opensearch/reindex") @PostMapping("/solr/reindex")
public ResponseEntity<Map<String, Object>> reindexOpenSearch() { public ResponseEntity<Map<String, Object>> reindexSolr() {
try { try {
logger.info("Starting OpenSearch full reindex"); logger.info("Starting Solr full reindex");
if (!searchServiceAdapter.isSearchServiceAvailable()) { if (!searchServiceAdapter.isSearchServiceAvailable()) {
return ResponseEntity.badRequest().body(Map.of( return ResponseEntity.badRequest().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch is not available or healthy" "error", "Solr is not available or healthy"
)); ));
} }
@@ -77,14 +77,14 @@ public class AdminSearchController {
List<Story> allStories = storyService.findAllWithAssociations(); List<Story> allStories = storyService.findAllWithAssociations();
List<Author> allAuthors = authorService.findAllWithStories(); List<Author> allAuthors = authorService.findAllWithStories();
// Bulk index directly in OpenSearch // Bulk index directly in Solr
if (openSearchService != null) { if (solrService != null) {
openSearchService.bulkIndexStories(allStories); solrService.bulkIndexStories(allStories);
openSearchService.bulkIndexAuthors(allAuthors); solrService.bulkIndexAuthors(allAuthors);
} else { } else {
return ResponseEntity.badRequest().body(Map.of( return ResponseEntity.badRequest().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch service not available" "error", "Solr service not available"
)); ));
} }
@@ -92,7 +92,7 @@ public class AdminSearchController {
return ResponseEntity.ok(Map.of( return ResponseEntity.ok(Map.of(
"success", true, "success", true,
"message", String.format("Reindexed %d stories and %d authors in OpenSearch", "message", String.format("Reindexed %d stories and %d authors in Solr",
allStories.size(), allAuthors.size()), allStories.size(), allAuthors.size()),
"storiesCount", allStories.size(), "storiesCount", allStories.size(),
"authorsCount", allAuthors.size(), "authorsCount", allAuthors.size(),
@@ -100,36 +100,36 @@ public class AdminSearchController {
)); ));
} catch (Exception e) { } catch (Exception e) {
logger.error("Error during OpenSearch reindex", e); logger.error("Error during Solr reindex", e);
return ResponseEntity.internalServerError().body(Map.of( return ResponseEntity.internalServerError().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch reindex failed: " + e.getMessage() "error", "Solr reindex failed: " + e.getMessage()
)); ));
} }
} }
/** /**
* Recreate OpenSearch indices * Recreate Solr indices
*/ */
@PostMapping("/opensearch/recreate") @PostMapping("/solr/recreate")
public ResponseEntity<Map<String, Object>> recreateOpenSearchIndices() { public ResponseEntity<Map<String, Object>> recreateSolrIndices() {
try { try {
logger.info("Starting OpenSearch indices recreation"); logger.info("Starting Solr indices recreation");
if (!searchServiceAdapter.isSearchServiceAvailable()) { if (!searchServiceAdapter.isSearchServiceAvailable()) {
return ResponseEntity.badRequest().body(Map.of( return ResponseEntity.badRequest().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch is not available or healthy" "error", "Solr is not available or healthy"
)); ));
} }
// Recreate indices // Recreate indices
if (openSearchService != null) { if (solrService != null) {
openSearchService.recreateIndices(); solrService.recreateIndices();
} else { } else {
return ResponseEntity.badRequest().body(Map.of( return ResponseEntity.badRequest().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch service not available" "error", "Solr service not available"
)); ));
} }
@@ -138,14 +138,14 @@ public class AdminSearchController {
List<Author> allAuthors = authorService.findAllWithStories(); List<Author> allAuthors = authorService.findAllWithStories();
// Bulk index after recreation // Bulk index after recreation
openSearchService.bulkIndexStories(allStories); solrService.bulkIndexStories(allStories);
openSearchService.bulkIndexAuthors(allAuthors); solrService.bulkIndexAuthors(allAuthors);
int totalIndexed = allStories.size() + allAuthors.size(); int totalIndexed = allStories.size() + allAuthors.size();
return ResponseEntity.ok(Map.of( return ResponseEntity.ok(Map.of(
"success", true, "success", true,
"message", String.format("Recreated OpenSearch indices and indexed %d stories and %d authors", "message", String.format("Recreated Solr indices and indexed %d stories and %d authors",
allStories.size(), allAuthors.size()), allStories.size(), allAuthors.size()),
"storiesCount", allStories.size(), "storiesCount", allStories.size(),
"authorsCount", allAuthors.size(), "authorsCount", allAuthors.size(),
@@ -153,10 +153,10 @@ public class AdminSearchController {
)); ));
} catch (Exception e) { } catch (Exception e) {
logger.error("Error during OpenSearch indices recreation", e); logger.error("Error during Solr indices recreation", e);
return ResponseEntity.internalServerError().body(Map.of( return ResponseEntity.internalServerError().body(Map.of(
"success", false, "success", false,
"error", "OpenSearch indices recreation failed: " + e.getMessage() "error", "Solr indices recreation failed: " + e.getMessage()
)); ));
} }
} }

2
backend/src/main/java/com/storycove/controller/CollectionController.java
View File

@@ -291,7 +291,7 @@ public class CollectionController {
// Collections are not indexed in search engine yet // Collections are not indexed in search engine yet
return ResponseEntity.ok(Map.of( return ResponseEntity.ok(Map.of(
"success", true, "success", true,
"message", "Collections indexing not yet implemented in OpenSearch", "message", "Collections indexing not yet implemented in Solr",
"count", allCollections.size() "count", allCollections.size()
)); ));
} catch (Exception e) { } catch (Exception e) {

12
backend/src/main/java/com/storycove/dto/SearchResultDto.java
View File

@@ -34,6 +34,18 @@ public class SearchResultDto<T> {
this.facets = facets; this.facets = facets;
} }
// Simple constructor for basic search results with facet list
public SearchResultDto(List<T> results, long totalHits, int resultCount, List<FacetCountDto> facetsList) {
this.results = results;
this.totalHits = totalHits;
this.page = 0;
this.perPage = resultCount;
this.query = "";
this.searchTimeMs = 0;
// Convert list to map if needed - for now just set empty map
this.facets = java.util.Collections.emptyMap();
}
// Getters and Setters // Getters and Setters
public List<T> getResults() { public List<T> getResults() {
return results; return results;

16
backend/src/main/java/com/storycove/service/AuthorService.java
View File

@@ -132,7 +132,7 @@ public class AuthorService {
validateAuthorForCreate(author); validateAuthorForCreate(author);
Author savedAuthor = authorRepository.save(author); Author savedAuthor = authorRepository.save(author);
// Index in OpenSearch // Index in Solr
searchServiceAdapter.indexAuthor(savedAuthor); searchServiceAdapter.indexAuthor(savedAuthor);
return savedAuthor; return savedAuthor;
@@ -150,7 +150,7 @@ public class AuthorService {
updateAuthorFields(existingAuthor, authorUpdates); updateAuthorFields(existingAuthor, authorUpdates);
Author savedAuthor = authorRepository.save(existingAuthor); Author savedAuthor = authorRepository.save(existingAuthor);
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(savedAuthor); searchServiceAdapter.updateAuthor(savedAuthor);
return savedAuthor; return savedAuthor;
@@ -166,7 +166,7 @@ public class AuthorService {
authorRepository.delete(author); authorRepository.delete(author);
// Remove from OpenSearch // Remove from Solr
searchServiceAdapter.deleteAuthor(id); searchServiceAdapter.deleteAuthor(id);
} }
@@ -175,7 +175,7 @@ public class AuthorService {
author.addUrl(url); author.addUrl(url);
Author savedAuthor = authorRepository.save(author); Author savedAuthor = authorRepository.save(author);
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(savedAuthor); searchServiceAdapter.updateAuthor(savedAuthor);
return savedAuthor; return savedAuthor;
@@ -186,7 +186,7 @@ public class AuthorService {
author.removeUrl(url); author.removeUrl(url);
Author savedAuthor = authorRepository.save(author); Author savedAuthor = authorRepository.save(author);
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(savedAuthor); searchServiceAdapter.updateAuthor(savedAuthor);
return savedAuthor; return savedAuthor;
@@ -221,7 +221,7 @@ public class AuthorService {
logger.debug("Saved author rating: {} for author: {}", logger.debug("Saved author rating: {} for author: {}",
refreshedAuthor.getAuthorRating(), refreshedAuthor.getName()); refreshedAuthor.getAuthorRating(), refreshedAuthor.getName());
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(refreshedAuthor); searchServiceAdapter.updateAuthor(refreshedAuthor);
return refreshedAuthor; return refreshedAuthor;
@@ -265,7 +265,7 @@ public class AuthorService {
author.setAvatarImagePath(avatarPath); author.setAvatarImagePath(avatarPath);
Author savedAuthor = authorRepository.save(author); Author savedAuthor = authorRepository.save(author);
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(savedAuthor); searchServiceAdapter.updateAuthor(savedAuthor);
return savedAuthor; return savedAuthor;
@@ -276,7 +276,7 @@ public class AuthorService {
author.setAvatarImagePath(null); author.setAvatarImagePath(null);
Author savedAuthor = authorRepository.save(author); Author savedAuthor = authorRepository.save(author);
// Update in OpenSearch // Update in Solr
searchServiceAdapter.updateAuthor(savedAuthor); searchServiceAdapter.updateAuthor(savedAuthor);
return savedAuthor; return savedAuthor;

4
backend/src/main/java/com/storycove/service/CollectionService.java
View File

@@ -55,8 +55,8 @@ public class CollectionService {
*/ */
public SearchResultDto<Collection> searchCollections(String query, List<String> tags, boolean includeArchived, int page, int limit) { public SearchResultDto<Collection> searchCollections(String query, List<String> tags, boolean includeArchived, int page, int limit) {
// Collections are currently handled at database level, not indexed in search engine // Collections are currently handled at database level, not indexed in search engine
// Return empty result for now as collections search is not implemented in OpenSearch // Return empty result for now as collections search is not implemented in Solr
logger.warn("Collections search not yet implemented in OpenSearch, returning empty results"); logger.warn("Collections search not yet implemented in Solr, returning empty results");
return new SearchResultDto<>(new ArrayList<>(), 0, page, limit, query != null ? query : "", 0); return new SearchResultDto<>(new ArrayList<>(), 0, page, limit, query != null ? query : "", 0);
} }

22
backend/src/main/java/com/storycove/service/LibraryService.java
View File

@@ -115,7 +115,7 @@ public class LibraryService implements ApplicationContextAware {
/** /**
* Switch to library after authentication with forced reindexing * Switch to library after authentication with forced reindexing
* This ensures OpenSearch is always up-to-date after login * This ensures Solr is always up-to-date after login
*/ */
public synchronized void switchToLibraryAfterAuthentication(String libraryId) throws Exception { public synchronized void switchToLibraryAfterAuthentication(String libraryId) throws Exception {
logger.info("Switching to library after authentication: {} (forcing reindex)", libraryId); logger.info("Switching to library after authentication: {} (forcing reindex)", libraryId);
@@ -154,15 +154,15 @@ public class LibraryService implements ApplicationContextAware {
// Set new active library (datasource routing handled by SmartRoutingDataSource) // Set new active library (datasource routing handled by SmartRoutingDataSource)
currentLibraryId = libraryId; currentLibraryId = libraryId;
// OpenSearch indexes are global - no per-library initialization needed // Solr indexes are global - no per-library initialization needed
logger.debug("Library switched to OpenSearch mode for library: {}", libraryId); logger.debug("Library switched to Solr mode for library: {}", libraryId);
logger.info("Successfully switched to library: {}", library.getName()); logger.info("Successfully switched to library: {}", library.getName());
// Perform complete reindex AFTER library switch is fully complete // Perform complete reindex AFTER library switch is fully complete
// This ensures database routing is properly established // This ensures database routing is properly established
if (forceReindex || !libraryId.equals(previousLibraryId)) { if (forceReindex || !libraryId.equals(previousLibraryId)) {
logger.debug("Starting post-switch OpenSearch reindex for library: {}", libraryId); logger.debug("Starting post-switch Solr reindex for library: {}", libraryId);
// Run reindex asynchronously to avoid blocking authentication response // Run reindex asynchronously to avoid blocking authentication response
// and allow time for database routing to fully stabilize // and allow time for database routing to fully stabilize
@@ -171,7 +171,7 @@ public class LibraryService implements ApplicationContextAware {
try { try {
// Give routing time to stabilize // Give routing time to stabilize
Thread.sleep(500); Thread.sleep(500);
logger.debug("Starting async OpenSearch reindex for library: {}", finalLibraryId); logger.debug("Starting async Solr reindex for library: {}", finalLibraryId);
SearchServiceAdapter searchService = applicationContext.getBean(SearchServiceAdapter.class); SearchServiceAdapter searchService = applicationContext.getBean(SearchServiceAdapter.class);
// Get all stories and authors for reindexing // Get all stories and authors for reindexing
@@ -184,12 +184,12 @@ public class LibraryService implements ApplicationContextAware {
searchService.bulkIndexStories(allStories); searchService.bulkIndexStories(allStories);
searchService.bulkIndexAuthors(allAuthors); searchService.bulkIndexAuthors(allAuthors);
logger.info("Completed async OpenSearch reindexing for library: {} ({} stories, {} authors)", logger.info("Completed async Solr reindexing for library: {} ({} stories, {} authors)",
finalLibraryId, allStories.size(), allAuthors.size()); finalLibraryId, allStories.size(), allAuthors.size());
} catch (Exception e) { } catch (Exception e) {
logger.warn("Failed to async reindex OpenSearch for library {}: {}", finalLibraryId, e.getMessage()); logger.warn("Failed to async reindex Solr for library {}: {}", finalLibraryId, e.getMessage());
} }
}, "OpenSearchReindex-" + libraryId).start(); }, "SolrReindex-" + libraryId).start();
} }
} }
@@ -525,8 +525,8 @@ public class LibraryService implements ApplicationContextAware {
// 1. Create image directory structure // 1. Create image directory structure
initializeImageDirectories(library); initializeImageDirectories(library);
// 2. OpenSearch indexes are global and managed automatically // 2. Solr indexes are global and managed automatically
// No per-library initialization needed for OpenSearch // No per-library initialization needed for Solr
logger.debug("Successfully initialized resources for library: {}", library.getName()); logger.debug("Successfully initialized resources for library: {}", library.getName());
@@ -760,7 +760,7 @@ public class LibraryService implements ApplicationContextAware {
private void closeCurrentResources() { private void closeCurrentResources() {
// No need to close datasource - SmartRoutingDataSource handles this // No need to close datasource - SmartRoutingDataSource handles this
// OpenSearch service is managed by Spring - no explicit cleanup needed // Solr service is managed by Spring - no explicit cleanup needed
// Don't clear currentLibraryId here - only when explicitly switching // Don't clear currentLibraryId here - only when explicitly switching
} }

133
backend/src/main/java/com/storycove/service/OpenSearchHealthService.java
View File

@@ -1,133 +0,0 @@
package com.storycove.service;
import com.storycove.config.OpenSearchProperties;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch.cluster.HealthRequest;
import org.opensearch.client.opensearch.cluster.HealthResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.time.LocalDateTime;
import java.util.concurrent.atomic.AtomicReference;
@Service
@ConditionalOnProperty(name = "storycove.search.engine", havingValue = "opensearch")
public class OpenSearchHealthService implements HealthIndicator {
private static final Logger logger = LoggerFactory.getLogger(OpenSearchHealthService.class);
private final OpenSearchClient openSearchClient;
private final OpenSearchProperties properties;
private final AtomicReference<Health> lastKnownHealth = new AtomicReference<>(Health.unknown().build());
private LocalDateTime lastCheckTime = LocalDateTime.now();
@Autowired
public OpenSearchHealthService(OpenSearchClient openSearchClient, OpenSearchProperties properties) {
this.openSearchClient = openSearchClient;
this.properties = properties;
}
@Override
public Health health() {
return lastKnownHealth.get();
}
@Scheduled(fixedDelayString = "#{@openSearchProperties.health.checkInterval}")
public void performHealthCheck() {
try {
HealthResponse clusterHealth = openSearchClient.cluster().health(
HealthRequest.of(h -> h.timeout(t -> t.time("10s")))
);
Health.Builder healthBuilder = Health.up()
.withDetail("cluster_name", clusterHealth.clusterName())
.withDetail("status", clusterHealth.status().jsonValue())
.withDetail("number_of_nodes", clusterHealth.numberOfNodes())
.withDetail("number_of_data_nodes", clusterHealth.numberOfDataNodes())
.withDetail("active_primary_shards", clusterHealth.activePrimaryShards())
.withDetail("active_shards", clusterHealth.activeShards())
.withDetail("relocating_shards", clusterHealth.relocatingShards())
.withDetail("initializing_shards", clusterHealth.initializingShards())
.withDetail("unassigned_shards", clusterHealth.unassignedShards())
.withDetail("last_check", LocalDateTime.now());
// Check if cluster status is concerning
switch (clusterHealth.status()) {
case Red:
healthBuilder = Health.down()
.withDetail("reason", "Cluster status is RED - some primary shards are unassigned");
break;
case Yellow:
if (isProduction()) {
healthBuilder = Health.down()
.withDetail("reason", "Cluster status is YELLOW - some replica shards are unassigned (critical in production)");
} else {
// Yellow is acceptable in development (single node clusters)
healthBuilder.withDetail("warning", "Cluster status is YELLOW - acceptable for development");
}
break;
case Green:
// All good
break;
}
lastKnownHealth.set(healthBuilder.build());
lastCheckTime = LocalDateTime.now();
if (properties.getHealth().isEnableMetrics()) {
logMetrics(clusterHealth);
}
} catch (Exception e) {
logger.error("OpenSearch health check failed", e);
Health unhealthyStatus = Health.down()
.withDetail("error", e.getMessage())
.withDetail("last_successful_check", lastCheckTime)
.withDetail("current_time", LocalDateTime.now())
.build();
lastKnownHealth.set(unhealthyStatus);
}
}
private void logMetrics(HealthResponse clusterHealth) {
logger.info("OpenSearch Cluster Metrics - Status: {}, Nodes: {}, Active Shards: {}, Unassigned: {}",
clusterHealth.status().jsonValue(),
clusterHealth.numberOfNodes(),
clusterHealth.activeShards(),
clusterHealth.unassignedShards());
}
private boolean isProduction() {
return "production".equalsIgnoreCase(properties.getProfile());
}
/**
* Manual health check for immediate status
*/
public boolean isClusterHealthy() {
Health currentHealth = lastKnownHealth.get();
return currentHealth.getStatus() == org.springframework.boot.actuate.health.Status.UP;
}
/**
* Get detailed cluster information
*/
public String getClusterInfo() {
try {
var info = openSearchClient.info();
return String.format("OpenSearch %s (Cluster: %s, Lucene: %s)",
info.version().number(),
info.clusterName(),
info.version().luceneVersion());
} catch (Exception e) {
return "Unable to retrieve cluster information: " + e.getMessage();
}
}
}

1077
backend/src/main/java/com/storycove/service/OpenSearchService.java
View File

File diff suppressed because it is too large Load Diff

87
backend/src/main/java/com/storycove/service/SearchServiceAdapter.java
View File

@@ -16,7 +16,7 @@ import java.util.UUID;
/** /**
* Service adapter that provides a unified interface for search operations. * Service adapter that provides a unified interface for search operations.
* *
* This adapter directly delegates to OpenSearchService. * This adapter directly delegates to SolrService.
*/ */
@Service @Service
public class SearchServiceAdapter { public class SearchServiceAdapter {
@@ -24,7 +24,7 @@ public class SearchServiceAdapter {
private static final Logger logger = LoggerFactory.getLogger(SearchServiceAdapter.class); private static final Logger logger = LoggerFactory.getLogger(SearchServiceAdapter.class);
@Autowired @Autowired
private OpenSearchService openSearchService; private SolrService solrService;
// =============================== // ===============================
// SEARCH OPERATIONS // SEARCH OPERATIONS
@@ -46,11 +46,20 @@ public class SearchServiceAdapter {
String sourceDomain, String seriesFilter, String sourceDomain, String seriesFilter,
Integer minTagCount, Boolean popularOnly, Integer minTagCount, Boolean popularOnly,
Boolean hiddenGemsOnly) { Boolean hiddenGemsOnly) {
return openSearchService.searchStories(query, tags, author, series, minWordCount, maxWordCount, logger.info("SearchServiceAdapter: delegating search to SolrService");
minRating, isRead, isFavorite, sortBy, sortOrder, page, size, facetBy, try {
createdAfter, createdBefore, lastReadAfter, lastReadBefore, unratedOnly, readingStatus, SearchResultDto<StorySearchDto> result = solrService.searchStories(query, tags, author, series, minWordCount, maxWordCount,
hasReadingProgress, hasCoverImage, sourceDomain, seriesFilter, minTagCount, popularOnly, minRating, isRead, isFavorite, sortBy, sortOrder, page, size, facetBy,
hiddenGemsOnly); createdAfter, createdBefore, lastReadAfter, lastReadBefore, unratedOnly, readingStatus,
hasReadingProgress, hasCoverImage, sourceDomain, seriesFilter, minTagCount, popularOnly,
hiddenGemsOnly);
logger.info("SearchServiceAdapter: received result with {} stories and {} facets",
result.getResults().size(), result.getFacets().size());
return result;
} catch (Exception e) {
logger.error("SearchServiceAdapter: error during search", e);
throw e;
}
} }
/** /**
@@ -60,7 +69,7 @@ public class SearchServiceAdapter {
String series, Integer minWordCount, Integer maxWordCount, String series, Integer minWordCount, Integer maxWordCount,
Float minRating, Boolean isRead, Boolean isFavorite, Float minRating, Boolean isRead, Boolean isFavorite,
Long seed) { Long seed) {
return openSearchService.getRandomStories(count, tags, author, series, minWordCount, maxWordCount, return solrService.getRandomStories(count, tags, author, series, minWordCount, maxWordCount,
minRating, isRead, isFavorite, seed); minRating, isRead, isFavorite, seed);
} }
@@ -69,7 +78,7 @@ public class SearchServiceAdapter {
*/ */
public void recreateIndices() { public void recreateIndices() {
try { try {
openSearchService.recreateIndices(); solrService.recreateIndices();
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to recreate search indices", e); logger.error("Failed to recreate search indices", e);
throw new RuntimeException("Failed to recreate search indices", e); throw new RuntimeException("Failed to recreate search indices", e);
@@ -93,21 +102,21 @@ public class SearchServiceAdapter {
* Get random story ID with unified interface * Get random story ID with unified interface
*/ */
public String getRandomStoryId(Long seed) { public String getRandomStoryId(Long seed) {
return openSearchService.getRandomStoryId(seed); return solrService.getRandomStoryId(seed);
} }
/** /**
* Search authors with unified interface * Search authors with unified interface
*/ */
public List<AuthorSearchDto> searchAuthors(String query, int limit) { public List<AuthorSearchDto> searchAuthors(String query, int limit) {
return openSearchService.searchAuthors(query, limit); return solrService.searchAuthors(query, limit);
} }
/** /**
* Get tag suggestions with unified interface * Get tag suggestions with unified interface
*/ */
public List<String> getTagSuggestions(String query, int limit) { public List<String> getTagSuggestions(String query, int limit) {
return openSearchService.getTagSuggestions(query, limit); return solrService.getTagSuggestions(query, limit);
} }
// =============================== // ===============================
@@ -115,88 +124,88 @@ public class SearchServiceAdapter {
// =============================== // ===============================
/** /**
* Index a story in OpenSearch * Index a story in Solr
*/ */
public void indexStory(Story story) { public void indexStory(Story story) {
try { try {
openSearchService.indexStory(story); solrService.indexStory(story);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to index story {}", story.getId(), e); logger.error("Failed to index story {}", story.getId(), e);
} }
} }
/** /**
* Update a story in OpenSearch * Update a story in Solr
*/ */
public void updateStory(Story story) { public void updateStory(Story story) {
try { try {
openSearchService.updateStory(story); solrService.updateStory(story);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to update story {}", story.getId(), e); logger.error("Failed to update story {}", story.getId(), e);
} }
} }
/** /**
* Delete a story from OpenSearch * Delete a story from Solr
*/ */
public void deleteStory(UUID storyId) { public void deleteStory(UUID storyId) {
try { try {
openSearchService.deleteStory(storyId); solrService.deleteStory(storyId);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to delete story {}", storyId, e); logger.error("Failed to delete story {}", storyId, e);
} }
} }
/** /**
* Index an author in OpenSearch * Index an author in Solr
*/ */
public void indexAuthor(Author author) { public void indexAuthor(Author author) {
try { try {
openSearchService.indexAuthor(author); solrService.indexAuthor(author);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to index author {}", author.getId(), e); logger.error("Failed to index author {}", author.getId(), e);
} }
} }
/** /**
* Update an author in OpenSearch * Update an author in Solr
*/ */
public void updateAuthor(Author author) { public void updateAuthor(Author author) {
try { try {
openSearchService.updateAuthor(author); solrService.updateAuthor(author);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to update author {}", author.getId(), e); logger.error("Failed to update author {}", author.getId(), e);
} }
} }
/** /**
* Delete an author from OpenSearch * Delete an author from Solr
*/ */
public void deleteAuthor(UUID authorId) { public void deleteAuthor(UUID authorId) {
try { try {
openSearchService.deleteAuthor(authorId); solrService.deleteAuthor(authorId);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to delete author {}", authorId, e); logger.error("Failed to delete author {}", authorId, e);
} }
} }
/** /**
* Bulk index stories in OpenSearch * Bulk index stories in Solr
*/ */
public void bulkIndexStories(List<Story> stories) { public void bulkIndexStories(List<Story> stories) {
try { try {
openSearchService.bulkIndexStories(stories); solrService.bulkIndexStories(stories);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to bulk index {} stories", stories.size(), e); logger.error("Failed to bulk index {} stories", stories.size(), e);
} }
} }
/** /**
* Bulk index authors in OpenSearch * Bulk index authors in Solr
*/ */
public void bulkIndexAuthors(List<Author> authors) { public void bulkIndexAuthors(List<Author> authors) {
try { try {
openSearchService.bulkIndexAuthors(authors); solrService.bulkIndexAuthors(authors);
} catch (Exception e) { } catch (Exception e) {
logger.error("Failed to bulk index {} authors", authors.size(), e); logger.error("Failed to bulk index {} authors", authors.size(), e);
} }
@@ -210,14 +219,14 @@ public class SearchServiceAdapter {
* Check if search service is available and healthy * Check if search service is available and healthy
*/ */
public boolean isSearchServiceAvailable() { public boolean isSearchServiceAvailable() {
return openSearchService.testConnection(); return solrService.testConnection();
} }
/** /**
* Get current search engine name * Get current search engine name
*/ */
public String getCurrentSearchEngine() { public String getCurrentSearchEngine() {
return "opensearch"; return "solr";
} }
/** /**
@@ -228,10 +237,10 @@ public class SearchServiceAdapter {
} }
/** /**
* Check if we can switch to OpenSearch * Check if we can switch to Solr
*/ */
public boolean canSwitchToOpenSearch() { public boolean canSwitchToSolr() {
return true; // Already using OpenSearch return true; // Already using Solr
} }
/** /**
@@ -246,10 +255,10 @@ public class SearchServiceAdapter {
*/ */
public SearchStatus getSearchStatus() { public SearchStatus getSearchStatus() {
return new SearchStatus( return new SearchStatus(
"opensearch", "solr",
false, // no dual-write false, // no dual-write
false, // no typesense false, // no typesense
openSearchService.testConnection() solrService.testConnection()
); );
} }
@@ -260,19 +269,19 @@ public class SearchServiceAdapter {
private final String primaryEngine; private final String primaryEngine;
private final boolean dualWrite; private final boolean dualWrite;
private final boolean typesenseAvailable; private final boolean typesenseAvailable;
private final boolean openSearchAvailable; private final boolean solrAvailable;
public SearchStatus(String primaryEngine, boolean dualWrite, public SearchStatus(String primaryEngine, boolean dualWrite,
boolean typesenseAvailable, boolean openSearchAvailable) { boolean typesenseAvailable, boolean solrAvailable) {
this.primaryEngine = primaryEngine; this.primaryEngine = primaryEngine;
this.dualWrite = dualWrite; this.dualWrite = dualWrite;
this.typesenseAvailable = typesenseAvailable; this.typesenseAvailable = typesenseAvailable;
this.openSearchAvailable = openSearchAvailable; this.solrAvailable = solrAvailable;
} }
public String getPrimaryEngine() { return primaryEngine; } public String getPrimaryEngine() { return primaryEngine; }
public boolean isDualWrite() { return dualWrite; } public boolean isDualWrite() { return dualWrite; }
public boolean isTypesenseAvailable() { return typesenseAvailable; } public boolean isTypesenseAvailable() { return typesenseAvailable; }
public boolean isOpenSearchAvailable() { return openSearchAvailable; } public boolean isSolrAvailable() { return solrAvailable; }
} }
} }

931
backend/src/main/java/com/storycove/service/SolrService.java Normal file
View File

@@ -0,0 +1,931 @@
package com.storycove.service;
import com.storycove.config.SolrProperties;
import com.storycove.dto.AuthorSearchDto;
import com.storycove.dto.FacetCountDto;
import com.storycove.dto.SearchResultDto;
import com.storycove.dto.StorySearchDto;
import com.storycove.entity.Author;
import com.storycove.entity.Story;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.*;
import java.util.stream.Collectors;
@Service
@ConditionalOnProperty(
value = "storycove.search.engine",
havingValue = "solr",
matchIfMissing = false
)
public class SolrService {
private static final Logger logger = LoggerFactory.getLogger(SolrService.class);
@Autowired(required = false)
private SolrClient solrClient;
@Autowired
private SolrProperties properties;
@Autowired
@Lazy
private ReadingTimeService readingTimeService;
@PostConstruct
public void initializeCores() {
if (!isAvailable()) {
logger.debug("Solr client not available - skipping core initialization");
return;
}
try {
logger.debug("Testing Solr cores availability...");
testCoreAvailability(properties.getCores().getStories());
testCoreAvailability(properties.getCores().getAuthors());
logger.debug("Solr cores are available");
} catch (Exception e) {
logger.error("Failed to test Solr cores availability", e);
}
}
// ===============================
// CORE MANAGEMENT
// ===============================
private void testCoreAvailability(String coreName) throws IOException, SolrServerException {
SolrQuery query = new SolrQuery("*:*");
query.setRows(0);
QueryResponse response = solrClient.query(coreName, query);
logger.debug("Core {} is available - found {} documents", coreName, response.getResults().getNumFound());
}
// ===============================
// STORY INDEXING
// ===============================
public void indexStory(Story story) throws IOException {
if (!isAvailable()) {
logger.debug("Solr not available - skipping story indexing");
return;
}
try {
logger.debug("Indexing story: {} ({})", story.getTitle(), story.getId());
SolrInputDocument doc = createStoryDocument(story);
UpdateResponse response = solrClient.add(properties.getCores().getStories(), doc,
properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully indexed story: {}", story.getId());
} else {
logger.warn("Story indexing returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to index story: {}", story.getId(), e);
throw new IOException("Failed to index story", e);
}
}
public void updateStory(Story story) throws IOException {
// For Solr, update is the same as index (upsert behavior)
indexStory(story);
}
public void deleteStory(UUID storyId) throws IOException {
if (!isAvailable()) {
logger.debug("Solr not available - skipping story deletion");
return;
}
try {
logger.debug("Deleting story from index: {}", storyId);
UpdateResponse response = solrClient.deleteById(properties.getCores().getStories(),
storyId.toString(), properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully deleted story: {}", storyId);
} else {
logger.warn("Story deletion returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to delete story: {}", storyId, e);
throw new IOException("Failed to delete story", e);
}
}
// ===============================
// AUTHOR INDEXING
// ===============================
public void indexAuthor(Author author) throws IOException {
if (!isAvailable()) {
logger.debug("Solr not available - skipping author indexing");
return;
}
try {
logger.debug("Indexing author: {} ({})", author.getName(), author.getId());
SolrInputDocument doc = createAuthorDocument(author);
UpdateResponse response = solrClient.add(properties.getCores().getAuthors(), doc,
properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully indexed author: {}", author.getId());
} else {
logger.warn("Author indexing returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to index author: {}", author.getId(), e);
throw new IOException("Failed to index author", e);
}
}
public void updateAuthor(Author author) throws IOException {
// For Solr, update is the same as index (upsert behavior)
indexAuthor(author);
}
public void deleteAuthor(UUID authorId) throws IOException {
if (!isAvailable()) {
logger.debug("Solr not available - skipping author deletion");
return;
}
try {
logger.debug("Deleting author from index: {}", authorId);
UpdateResponse response = solrClient.deleteById(properties.getCores().getAuthors(),
authorId.toString(), properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully deleted author: {}", authorId);
} else {
logger.warn("Author deletion returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to delete author: {}", authorId, e);
throw new IOException("Failed to delete author", e);
}
}
// ===============================
// BULK OPERATIONS
// ===============================
public void bulkIndexStories(List<Story> stories) throws IOException {
if (!isAvailable() || stories.isEmpty()) {
logger.debug("Solr not available or empty stories list - skipping bulk indexing");
return;
}
try {
logger.debug("Bulk indexing {} stories", stories.size());
List<SolrInputDocument> docs = stories.stream()
.map(this::createStoryDocument)
.collect(Collectors.toList());
UpdateResponse response = solrClient.add(properties.getCores().getStories(), docs,
properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully bulk indexed {} stories", stories.size());
} else {
logger.warn("Bulk story indexing returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to bulk index stories", e);
throw new IOException("Failed to bulk index stories", e);
}
}
public void bulkIndexAuthors(List<Author> authors) throws IOException {
if (!isAvailable() || authors.isEmpty()) {
logger.debug("Solr not available or empty authors list - skipping bulk indexing");
return;
}
try {
logger.debug("Bulk indexing {} authors", authors.size());
List<SolrInputDocument> docs = authors.stream()
.map(this::createAuthorDocument)
.collect(Collectors.toList());
UpdateResponse response = solrClient.add(properties.getCores().getAuthors(), docs,
properties.getCommit().getCommitWithin());
if (response.getStatus() == 0) {
logger.debug("Successfully bulk indexed {} authors", authors.size());
} else {
logger.warn("Bulk author indexing returned non-zero status: {}", response.getStatus());
}
} catch (SolrServerException e) {
logger.error("Failed to bulk index authors", e);
throw new IOException("Failed to bulk index authors", e);
}
}
// ===============================
// DOCUMENT CREATION
// ===============================
private SolrInputDocument createStoryDocument(Story story) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", story.getId().toString());
doc.addField("title", story.getTitle());
doc.addField("description", story.getDescription());
doc.addField("sourceUrl", story.getSourceUrl());
doc.addField("coverPath", story.getCoverPath());
doc.addField("wordCount", story.getWordCount());
doc.addField("rating", story.getRating());
doc.addField("volume", story.getVolume());
doc.addField("isRead", story.getIsRead());
doc.addField("readingPosition", story.getReadingPosition());
if (story.getLastReadAt() != null) {
doc.addField("lastReadAt", formatDateTime(story.getLastReadAt()));
}
if (story.getAuthor() != null) {
doc.addField("authorId", story.getAuthor().getId().toString());
doc.addField("authorName", story.getAuthor().getName());
}
if (story.getSeries() != null) {
doc.addField("seriesId", story.getSeries().getId().toString());
doc.addField("seriesName", story.getSeries().getName());
}
if (story.getTags() != null && !story.getTags().isEmpty()) {
List<String> tagNames = story.getTags().stream()
.map(tag -> tag.getName())
.collect(Collectors.toList());
doc.addField("tagNames", tagNames);
}
doc.addField("createdAt", formatDateTime(story.getCreatedAt()));
doc.addField("updatedAt", formatDateTime(story.getUpdatedAt()));
doc.addField("dateAdded", formatDateTime(story.getCreatedAt()));
return doc;
}
private SolrInputDocument createAuthorDocument(Author author) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", author.getId().toString());
doc.addField("name", author.getName());
doc.addField("notes", author.getNotes());
doc.addField("authorRating", author.getAuthorRating());
doc.addField("avatarImagePath", author.getAvatarImagePath());
if (author.getUrls() != null && !author.getUrls().isEmpty()) {
doc.addField("urls", author.getUrls());
}
// Calculate derived fields
if (author.getStories() != null) {
doc.addField("storyCount", author.getStories().size());
OptionalDouble avgRating = author.getStories().stream()
.filter(story -> story.getRating() != null && story.getRating() > 0)
.mapToInt(Story::getRating)
.average();
if (avgRating.isPresent()) {
doc.addField("averageStoryRating", avgRating.getAsDouble());
}
}
doc.addField("createdAt", formatDateTime(author.getCreatedAt()));
doc.addField("updatedAt", formatDateTime(author.getUpdatedAt()));
return doc;
}
private String formatDateTime(LocalDateTime dateTime) {
if (dateTime == null) return null;
return dateTime.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME) + "Z";
}
// ===============================
// UTILITY METHODS
// ===============================
public boolean isAvailable() {
return solrClient != null;
}
public boolean testConnection() {
if (!isAvailable()) {
return false;
}
try {
// Test connection by pinging a core
testCoreAvailability(properties.getCores().getStories());
return true;
} catch (Exception e) {
logger.debug("Solr connection test failed", e);
return false;
}
}
// ===============================
// SEARCH OPERATIONS
// ===============================
public SearchResultDto<StorySearchDto> searchStories(String query, List<String> tags, String author,
String series, Integer minWordCount, Integer maxWordCount,
Float minRating, Boolean isRead, Boolean isFavorite,
String sortBy, String sortOrder, int page, int size,
List<String> facetBy,
// Advanced filters
String createdAfter, String createdBefore,
String lastReadAfter, String lastReadBefore,
Boolean unratedOnly, String readingStatus,
Boolean hasReadingProgress, Boolean hasCoverImage,
String sourceDomain, String seriesFilter,
Integer minTagCount, Boolean popularOnly,
Boolean hiddenGemsOnly) {
if (!isAvailable()) {
logger.debug("Solr not available - returning empty search results");
return new SearchResultDto<StorySearchDto>(Collections.emptyList(), 0, 0, Collections.emptyList());
}
try {
SolrQuery solrQuery = new SolrQuery();
// Set query
if (query == null || query.trim().isEmpty()) {
solrQuery.setQuery("*:*");
} else {
// Use dismax query parser for better relevance
solrQuery.setQuery(query);
solrQuery.set("defType", "edismax");
solrQuery.set("qf", "title^3.0 description^2.0 authorName^2.0 seriesName^1.5 tagNames^1.0");
solrQuery.set("mm", "2<-1 5<-2 6<90%"); // Minimum should match
}
// Apply filters
applySearchFilters(solrQuery, tags, author, series, minWordCount, maxWordCount,
minRating, isRead, isFavorite, createdAfter, createdBefore, lastReadAfter,
lastReadBefore, unratedOnly, readingStatus, hasReadingProgress, hasCoverImage,
sourceDomain, seriesFilter, minTagCount, popularOnly, hiddenGemsOnly);
// Pagination
solrQuery.setStart(page * size);
solrQuery.setRows(size);
// Sorting
if (sortBy != null && !sortBy.isEmpty()) {
SolrQuery.ORDER order = "desc".equalsIgnoreCase(sortOrder) ?
SolrQuery.ORDER.desc : SolrQuery.ORDER.asc;
solrQuery.setSort(sortBy, order);
} else {
// Default relevance sorting
solrQuery.setSort("score", SolrQuery.ORDER.desc);
}
// Enable highlighting
if (properties.getQuery().isHighlight()) {
solrQuery.setHighlight(true);
solrQuery.addHighlightField("title");
solrQuery.addHighlightField("description");
solrQuery.setHighlightSimplePre("<em>");
solrQuery.setHighlightSimplePost("</em>");
solrQuery.setHighlightFragsize(150);
}
// Enable faceting
if (properties.getQuery().isFacets()) {
solrQuery.setFacet(true);
// Use optimized facet fields for better performance
solrQuery.addFacetField("authorName_facet");
solrQuery.addFacetField("tagNames_facet");
solrQuery.addFacetField("seriesName_facet");
solrQuery.addFacetField("rating");
solrQuery.addFacetField("isRead");
solrQuery.setFacetMinCount(1);
solrQuery.setFacetSort("count");
solrQuery.setFacetLimit(100); // Limit facet results for performance
}
// Debug: Log the query being sent to Solr
logger.info("SolrService: Executing Solr query: {}", solrQuery);
QueryResponse response = solrClient.query(properties.getCores().getStories(), solrQuery);
logger.info("SolrService: Query executed successfully, found {} results", response.getResults().getNumFound());
return buildStorySearchResult(response);
} catch (Exception e) {
logger.error("Story search failed for query: {}", query, e);
return new SearchResultDto<StorySearchDto>(Collections.emptyList(), 0, 0, Collections.emptyList());
}
}
public List<AuthorSearchDto> searchAuthors(String query, int limit) {
if (!isAvailable()) {
logger.debug("Solr not available - returning empty author search results");
return Collections.emptyList();
}
try {
SolrQuery solrQuery = new SolrQuery();
// Set query
if (query == null || query.trim().isEmpty()) {
solrQuery.setQuery("*:*");
} else {
solrQuery.setQuery(query);
solrQuery.set("defType", "edismax");
solrQuery.set("qf", "name^3.0 notes^1.0 urls^0.5");
}
solrQuery.setRows(limit);
solrQuery.setSort("storyCount", SolrQuery.ORDER.desc);
QueryResponse response = solrClient.query(properties.getCores().getAuthors(), solrQuery);
return buildAuthorSearchResults(response);
} catch (Exception e) {
logger.error("Author search failed for query: {}", query, e);
return Collections.emptyList();
}
}
public List<String> getTagSuggestions(String query, int limit) {
if (!isAvailable()) {
return Collections.emptyList();
}
try {
SolrQuery solrQuery = new SolrQuery();
solrQuery.setQuery("tagNames:*" + query + "*");
solrQuery.setRows(0);
solrQuery.setFacet(true);
solrQuery.addFacetField("tagNames_facet");
solrQuery.setFacetMinCount(1);
solrQuery.setFacetLimit(limit);
QueryResponse response = solrClient.query(properties.getCores().getStories(), solrQuery);
return response.getFacetField("tagNames_facet").getValues().stream()
.map(facet -> facet.getName())
.filter(name -> name.toLowerCase().contains(query.toLowerCase()))
.collect(Collectors.toList());
} catch (Exception e) {
logger.error("Tag suggestions failed for query: {}", query, e);
return Collections.emptyList();
}
}
public String getRandomStoryId(Long seed) {
if (!isAvailable()) {
return null;
}
try {
SolrQuery solrQuery = new SolrQuery("*:*");
solrQuery.setRows(1);
if (seed != null) {
solrQuery.setSort("random_" + seed, SolrQuery.ORDER.asc);
} else {
solrQuery.setSort("random_" + System.currentTimeMillis(), SolrQuery.ORDER.asc);
}
QueryResponse response = solrClient.query(properties.getCores().getStories(), solrQuery);
if (response.getResults().size() > 0) {
return (String) response.getResults().get(0).getFieldValue("id");
}
} catch (Exception e) {
logger.error("Random story ID retrieval failed", e);
}
return null;
}
// ===============================
// RESULT BUILDING
// ===============================
private SearchResultDto<StorySearchDto> buildStorySearchResult(QueryResponse response) {
SolrDocumentList results = response.getResults();
List<StorySearchDto> stories = new ArrayList<>();
for (SolrDocument doc : results) {
StorySearchDto story = convertToStorySearchDto(doc);
// Add highlights
if (response.getHighlighting() != null) {
String id = (String) doc.getFieldValue("id");
Map<String, List<String>> highlights = response.getHighlighting().get(id);
if (highlights != null) {
List<String> allHighlights = highlights.values().stream()
.flatMap(List::stream)
.collect(Collectors.toList());
story.setHighlights(allHighlights);
}
}
stories.add(story);
}
// Build facets organized by field name
Map<String, List<FacetCountDto>> facetsMap = new HashMap<>();
if (response.getFacetFields() != null) {
response.getFacetFields().forEach(facetField -> {
String fieldName = facetField.getName();
List<FacetCountDto> fieldFacets = new ArrayList<>();
facetField.getValues().forEach(count -> {
fieldFacets.add(new FacetCountDto(count.getName(), (int) count.getCount()));
});
facetsMap.put(fieldName, fieldFacets);
});
}
SearchResultDto<StorySearchDto> result = new SearchResultDto<StorySearchDto>(stories, (int) results.getNumFound(),
0, stories.size(), "", 0, facetsMap);
return result;
}
private List<AuthorSearchDto> buildAuthorSearchResults(QueryResponse response) {
return response.getResults().stream()
.map(this::convertToAuthorSearchDto)
.collect(Collectors.toList());
}
private StorySearchDto convertToStorySearchDto(SolrDocument doc) {
StorySearchDto story = new StorySearchDto();
story.setId(UUID.fromString((String) doc.getFieldValue("id")));
story.setTitle((String) doc.getFieldValue("title"));
story.setDescription((String) doc.getFieldValue("description"));
story.setSourceUrl((String) doc.getFieldValue("sourceUrl"));
story.setCoverPath((String) doc.getFieldValue("coverPath"));
story.setWordCount((Integer) doc.getFieldValue("wordCount"));
story.setRating((Integer) doc.getFieldValue("rating"));
story.setVolume((Integer) doc.getFieldValue("volume"));
story.setIsRead((Boolean) doc.getFieldValue("isRead"));
story.setReadingPosition((Integer) doc.getFieldValue("readingPosition"));
// Handle dates
story.setLastReadAt(parseDateTimeFromSolr(doc.getFieldValue("lastReadAt")));
story.setCreatedAt(parseDateTimeFromSolr(doc.getFieldValue("createdAt")));
story.setUpdatedAt(parseDateTimeFromSolr(doc.getFieldValue("updatedAt")));
story.setDateAdded(parseDateTimeFromSolr(doc.getFieldValue("dateAdded")));
// Handle author
String authorIdStr = (String) doc.getFieldValue("authorId");
if (authorIdStr != null) {
story.setAuthorId(UUID.fromString(authorIdStr));
}
story.setAuthorName((String) doc.getFieldValue("authorName"));
// Handle series
String seriesIdStr = (String) doc.getFieldValue("seriesId");
if (seriesIdStr != null) {
story.setSeriesId(UUID.fromString(seriesIdStr));
}
story.setSeriesName((String) doc.getFieldValue("seriesName"));
// Handle tags
Collection<Object> tagValues = doc.getFieldValues("tagNames");
if (tagValues != null) {
List<String> tagNames = tagValues.stream()
.map(Object::toString)
.collect(Collectors.toList());
story.setTagNames(tagNames);
}
return story;
}
private AuthorSearchDto convertToAuthorSearchDto(SolrDocument doc) {
AuthorSearchDto author = new AuthorSearchDto();
author.setId(UUID.fromString((String) doc.getFieldValue("id")));
author.setName((String) doc.getFieldValue("name"));
author.setNotes((String) doc.getFieldValue("notes"));
author.setAuthorRating((Integer) doc.getFieldValue("authorRating"));
author.setAvatarImagePath((String) doc.getFieldValue("avatarImagePath"));
author.setStoryCount((Integer) doc.getFieldValue("storyCount"));
Double avgRating = (Double) doc.getFieldValue("averageStoryRating");
if (avgRating != null) {
author.setAverageStoryRating(avgRating);
}
// Handle URLs
Collection<Object> urlValues = doc.getFieldValues("urls");
if (urlValues != null) {
List<String> urls = urlValues.stream()
.map(Object::toString)
.collect(Collectors.toList());
author.setUrls(urls);
}
// Handle dates
author.setCreatedAt(parseDateTimeFromSolr(doc.getFieldValue("createdAt")));
author.setUpdatedAt(parseDateTimeFromSolr(doc.getFieldValue("updatedAt")));
return author;
}
private LocalDateTime parseDateTime(String dateStr) {
if (dateStr == null || dateStr.isEmpty()) {
return null;
}
try {
// Remove 'Z' suffix if present and parse
String cleanDate = dateStr.endsWith("Z") ? dateStr.substring(0, dateStr.length() - 1) : dateStr;
return LocalDateTime.parse(cleanDate, DateTimeFormatter.ISO_LOCAL_DATE_TIME);
} catch (Exception e) {
logger.warn("Failed to parse date: {}", dateStr, e);
return null;
}
}
private LocalDateTime parseDateTimeFromSolr(Object dateValue) {
if (dateValue == null) {
return null;
}
if (dateValue instanceof Date) {
// Convert java.util.Date to LocalDateTime
return ((Date) dateValue).toInstant()
.atZone(java.time.ZoneId.systemDefault())
.toLocalDateTime();
} else if (dateValue instanceof String) {
return parseDateTime((String) dateValue);
} else {
logger.warn("Unexpected date type: {}", dateValue.getClass());
return null;
}
}
public List<StorySearchDto> getRandomStories(int count, List<String> tags, String author,
String series, Integer minWordCount, Integer maxWordCount,
Float minRating, Boolean isRead, Boolean isFavorite,
Long seed) {
if (!isAvailable()) {
return Collections.emptyList();
}
try {
SolrQuery solrQuery = new SolrQuery("*:*");
// Apply filters
applySearchFilters(solrQuery, tags, author, series, minWordCount, maxWordCount,
minRating, isRead, isFavorite, null, null, null, null, null, null,
null, null, null, null, null, null, null);
solrQuery.setRows(count);
// Use random sorting
if (seed != null) {
solrQuery.setSort("random_" + seed, SolrQuery.ORDER.asc);
} else {
solrQuery.setSort("random_" + System.currentTimeMillis(), SolrQuery.ORDER.asc);
}
QueryResponse response = solrClient.query(properties.getCores().getStories(), solrQuery);
return response.getResults().stream()
.map(this::convertToStorySearchDto)
.collect(Collectors.toList());
} catch (Exception e) {
logger.error("Random stories retrieval failed", e);
return Collections.emptyList();
}
}
// ===============================
// FILTER APPLICATION
// ===============================
private void applySearchFilters(SolrQuery solrQuery, List<String> tags, String author,
String series, Integer minWordCount, Integer maxWordCount,
Float minRating, Boolean isRead, Boolean isFavorite,
String createdAfter, String createdBefore,
String lastReadAfter, String lastReadBefore,
Boolean unratedOnly, String readingStatus,
Boolean hasReadingProgress, Boolean hasCoverImage,
String sourceDomain, String seriesFilter,
Integer minTagCount, Boolean popularOnly,
Boolean hiddenGemsOnly) {
List<String> filters = new ArrayList<>();
// Tag filters - use facet field for exact matching
if (tags != null && !tags.isEmpty()) {
String tagFilter = tags.stream()
.map(tag -> "tagNames_facet:\"" + escapeQueryChars(tag) + "\"")
.collect(Collectors.joining(" AND "));
filters.add("(" + tagFilter + ")");
}
// Author filter - use facet field for exact matching
if (author != null && !author.trim().isEmpty()) {
filters.add("authorName_facet:\"" + escapeQueryChars(author.trim()) + "\"");
}
// Series filter - use facet field for exact matching
if (series != null && !series.trim().isEmpty()) {
filters.add("seriesName_facet:\"" + escapeQueryChars(series.trim()) + "\"");
}
// Word count filters
if (minWordCount != null && maxWordCount != null) {
filters.add("wordCount:[" + minWordCount + " TO " + maxWordCount + "]");
} else if (minWordCount != null) {
filters.add("wordCount:[" + minWordCount + " TO *]");
} else if (maxWordCount != null) {
filters.add("wordCount:[* TO " + maxWordCount + "]");
}
// Rating filter
if (minRating != null) {
filters.add("rating:[" + minRating.intValue() + " TO *]");
}
// Read status filter
if (isRead != null) {
filters.add("isRead:" + isRead);
}
// Date filters - convert to ISO format for Solr
if (createdAfter != null) {
filters.add("createdAt:[" + formatDateForSolr(createdAfter) + " TO *]");
}
if (createdBefore != null) {
filters.add("createdAt:[* TO " + formatDateForSolr(createdBefore) + "]");
}
// Last read date filters
if (lastReadAfter != null) {
filters.add("lastReadAt:[" + formatDateForSolr(lastReadAfter) + " TO *]");
}
if (lastReadBefore != null) {
filters.add("lastReadAt:[* TO " + formatDateForSolr(lastReadBefore) + "]");
}
// Unrated filter
if (unratedOnly != null && unratedOnly) {
filters.add("(-rating:[1 TO *] OR rating:0)");
}
// Cover image filter
if (hasCoverImage != null) {
if (hasCoverImage) {
filters.add("coverPath:[\"\" TO *]"); // Has non-empty value
} else {
filters.add("-coverPath:[\"\" TO *]"); // No value or empty
}
}
// Reading status filter
if (readingStatus != null && !readingStatus.equals("all")) {
switch (readingStatus) {
case "unread":
// Unread: isRead is false AND readingPosition is 0 (or null)
filters.add("isRead:false AND (readingPosition:0 OR (*:* -readingPosition:[1 TO *]))");
break;
case "started":
// Started: has reading progress but not finished
filters.add("readingPosition:[1 TO *] AND isRead:false");
break;
case "completed":
// Completed: isRead is true
filters.add("isRead:true");
break;
}
}
// Reading progress filter
if (hasReadingProgress != null) {
if (hasReadingProgress) {
filters.add("readingPosition:[1 TO *]"); // Has reading progress
} else {
filters.add("(readingPosition:0 OR (*:* -readingPosition:[1 TO *]))"); // No reading progress
}
}
// Series filter
if (seriesFilter != null && !seriesFilter.equals("all")) {
switch (seriesFilter) {
case "standalone":
// Standalone: no series (seriesId is null or empty)
filters.add("(*:* -seriesId:[\"\" TO *])");
break;
case "series":
// Part of series: has seriesId
filters.add("seriesId:[\"\" TO *]");
break;
case "firstInSeries":
// First in series: has seriesId and volume is 1
filters.add("seriesId:[\"\" TO *] AND volume:1");
break;
case "lastInSeries":
// This would require complex logic to determine last volume per series
// For now, just filter by having a series (can be enhanced later)
filters.add("seriesId:[\"\" TO *]");
break;
}
}
// Source domain filter
if (sourceDomain != null && !sourceDomain.trim().isEmpty()) {
// Extract domain from sourceUrl field
filters.add("sourceUrl:*" + escapeQueryChars(sourceDomain.trim()) + "*");
}
// Apply all filters
for (String filter : filters) {
solrQuery.addFilterQuery(filter);
}
}
public void recreateCores() throws IOException {
logger.warn("Solr core recreation not supported through API - cores must be managed via Solr admin");
// Note: Core recreation in Solr requires admin API calls that are typically done
// through the Solr admin interface or by restarting with fresh cores
}
public void recreateIndices() throws IOException {
recreateCores();
}
/**
* Escape special characters in Solr query strings
*/
private String escapeQueryChars(String s) {
if (s == null) return null;
StringBuilder sb = new StringBuilder();
for (int i = 0; i < s.length(); i++) {
char c = s.charAt(i);
// These characters are part of the query syntax and must be escaped
if (c == '\\' || c == '+' || c == '-' || c == '!' || c == '(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '\"' || c == '{' || c == '}' || c == '~'
|| c == '*' || c == '?' || c == '|' || c == '&' || c == ';' || c == '/'
|| Character.isWhitespace(c)) {
sb.append('\\');
}
sb.append(c);
}
return sb.toString();
}
/**
* Format date string for Solr queries
*/
private String formatDateForSolr(String dateStr) {
if (dateStr == null || dateStr.isEmpty()) {
return dateStr;
}
try {
// If it's already in ISO format, return as-is
if (dateStr.contains("T") || dateStr.endsWith("Z")) {
return dateStr;
}
// Convert date string like "2025-08-23" to ISO format "2025-08-23T00:00:00Z"
if (dateStr.matches("\\d{4}-\\d{2}-\\d{2}")) {
return dateStr + "T00:00:00Z";
}
return dateStr;
} catch (Exception e) {
logger.warn("Failed to format date for Solr: {}", dateStr, e);
return dateStr;
}
}
}

74
backend/src/main/resources/application.yml
View File

@@ -39,54 +39,46 @@ storycove:
auth: auth:
password: ${APP_PASSWORD} # REQUIRED: No default password for security password: ${APP_PASSWORD} # REQUIRED: No default password for security
search: search:
engine: opensearch # OpenSearch is the only search engine engine: solr # Apache Solr search engine
opensearch: solr:
# Connection settings # Connection settings
host: ${OPENSEARCH_HOST:localhost} url: ${SOLR_URL:http://solr:8983/solr}
port: ${OPENSEARCH_PORT:9200} username: ${SOLR_USERNAME:}
scheme: ${OPENSEARCH_SCHEME:http} password: ${SOLR_PASSWORD:}
username: ${OPENSEARCH_USERNAME:}
password: ${OPENSEARCH_PASSWORD:} # Empty when security is disabled
# Environment-specific configuration # Core configuration
profile: ${SPRING_PROFILES_ACTIVE:development} # development, staging, production cores:
stories: ${SOLR_STORIES_CORE:storycove_stories}
authors: ${SOLR_AUTHORS_CORE:storycove_authors}
# Security settings # Connection settings
security:
ssl-verification: ${OPENSEARCH_SSL_VERIFICATION:false}
trust-all-certificates: ${OPENSEARCH_TRUST_ALL_CERTS:true}
keystore-path: ${OPENSEARCH_KEYSTORE_PATH:}
keystore-password: ${OPENSEARCH_KEYSTORE_PASSWORD:}
truststore-path: ${OPENSEARCH_TRUSTSTORE_PATH:}
truststore-password: ${OPENSEARCH_TRUSTSTORE_PASSWORD:}
# Connection pool settings
connection: connection:
timeout: ${OPENSEARCH_CONNECTION_TIMEOUT:30000} # 30 seconds timeout: ${SOLR_CONNECTION_TIMEOUT:30000} # 30 seconds
socket-timeout: ${OPENSEARCH_SOCKET_TIMEOUT:60000} # 60 seconds socket-timeout: ${SOLR_SOCKET_TIMEOUT:60000} # 60 seconds
max-connections-per-route: ${OPENSEARCH_MAX_CONN_PER_ROUTE:10} max-connections-per-route: ${SOLR_MAX_CONN_PER_ROUTE:10}
max-connections-total: ${OPENSEARCH_MAX_CONN_TOTAL:30} max-connections-total: ${SOLR_MAX_CONN_TOTAL:30}
retry-on-failure: ${OPENSEARCH_RETRY_ON_FAILURE:true} retry-on-failure: ${SOLR_RETRY_ON_FAILURE:true}
max-retries: ${OPENSEARCH_MAX_RETRIES:3} max-retries: ${SOLR_MAX_RETRIES:3}
# Index settings # Query settings
indices: query:
default-shards: ${OPENSEARCH_DEFAULT_SHARDS:1} default-rows: ${SOLR_DEFAULT_ROWS:10}
default-replicas: ${OPENSEARCH_DEFAULT_REPLICAS:0} max-rows: ${SOLR_MAX_ROWS:1000}
refresh-interval: ${OPENSEARCH_REFRESH_INTERVAL:1s} default-operator: ${SOLR_DEFAULT_OPERATOR:AND}
highlight: ${SOLR_ENABLE_HIGHLIGHT:true}
facets: ${SOLR_ENABLE_FACETS:true}
# Bulk operations # Commit settings
bulk: commit:
actions: ${OPENSEARCH_BULK_ACTIONS:1000} soft-commit: ${SOLR_SOFT_COMMIT:true}
size: ${OPENSEARCH_BULK_SIZE:5242880} # 5MB commit-within: ${SOLR_COMMIT_WITHIN:1000} # 1 second
timeout: ${OPENSEARCH_BULK_TIMEOUT:10000} # 10 seconds wait-searcher: ${SOLR_WAIT_SEARCHER:false}
concurrent-requests: ${OPENSEARCH_BULK_CONCURRENT:1}
# Health and monitoring # Health and monitoring
health: health:
check-interval: ${OPENSEARCH_HEALTH_CHECK_INTERVAL:30000} # 30 seconds check-interval: ${SOLR_HEALTH_CHECK_INTERVAL:30000} # 30 seconds
slow-query-threshold: ${OPENSEARCH_SLOW_QUERY_THRESHOLD:5000} # 5 seconds slow-query-threshold: ${SOLR_SLOW_QUERY_THRESHOLD:5000} # 5 seconds
enable-metrics: ${OPENSEARCH_ENABLE_METRICS:true} enable-metrics: ${SOLR_ENABLE_METRICS:true}
images: images:
storage-path: ${IMAGE_STORAGE_PATH:/app/images} storage-path: ${IMAGE_STORAGE_PATH:/app/images}
@@ -100,8 +92,8 @@ management:
show-details: when-authorized show-details: when-authorized
show-components: always show-components: always
health: health:
opensearch: solr:
enabled: ${OPENSEARCH_HEALTH_ENABLED:true} enabled: ${SOLR_HEALTH_ENABLED:true}
logging: logging:
level: level:

178
backend/src/main/resources/opensearch/README.md
View File

@@ -1,178 +0,0 @@
# OpenSearch Configuration - Best Practices Implementation
## Overview
This directory contains a production-ready OpenSearch configuration following industry best practices for security, scalability, and maintainability.
## Architecture
### 📁 Directory Structure
```
opensearch/
├── config/
│ ├── opensearch-development.yml # Development-specific settings
│ └── opensearch-production.yml # Production-specific settings
├── mappings/
│ ├── stories-mapping.json # Story index mapping
│ ├── authors-mapping.json # Author index mapping
│ └── collections-mapping.json # Collection index mapping
├── templates/
│ ├── stories-template.json # Index template for stories_*
│ └── index-lifecycle-policy.json # ILM policy for index management
└── README.md # This file
```
## ✅ Best Practices Implemented
### 🔒 **Security**
- **Environment-Aware SSL Configuration**
- Production: Full certificate validation with custom truststore support
- Development: Optional certificate validation for local development
- **Proper Authentication**: Basic auth with secure credential management
- **Connection Security**: TLS 1.3 support with hostname verification
### 🏗️ **Configuration Management**
- **Externalized Configuration**: JSON/YAML files instead of hardcoded values
- **Environment-Specific Settings**: Different configs for dev/staging/prod
- **Type-Safe Properties**: Strongly-typed configuration classes
- **Validation**: Configuration validation at startup
### 📈 **Scalability & Performance**
- **Connection Pooling**: Configurable connection pool with timeout management
- **Environment-Aware Sharding**:
- Development: 1 shard, 0 replicas (single node)
- Production: 3 shards, 1 replica (high availability)
- **Bulk Operations**: Optimized bulk indexing with configurable batch sizes
- **Index Templates**: Automatic application of settings to new indexes
### 🔄 **Index Lifecycle Management**
- **Automated Index Rollover**: Based on size, document count, and age
- **Hot-Warm-Cold Architecture**: Optimized storage costs
- **Retention Policies**: Automatic cleanup of old data
- **Force Merge**: Optimization in warm phase
### 📊 **Monitoring & Observability**
- **Health Checks**: Automatic cluster health monitoring
- **Spring Boot Actuator**: Health endpoints for monitoring systems
- **Metrics Collection**: Configurable performance metrics
- **Slow Query Detection**: Configurable thresholds for query performance
### 🛡️ **Error Handling & Resilience**
- **Connection Retry Logic**: Automatic retry with backoff
- **Circuit Breaker Pattern**: Fail-fast for unhealthy clusters
- **Graceful Degradation**: Graceful handling when OpenSearch unavailable
- **Detailed Error Logging**: Comprehensive error tracking
## 🚀 Usage
### Development Environment
```yaml
# application-development.yml
storycove:
opensearch:
profile: development
security:
ssl-verification: false
trust-all-certificates: true
indices:
default-shards: 1
default-replicas: 0
```
### Production Environment
```yaml
# application-production.yml
storycove:
opensearch:
profile: production
security:
ssl-verification: true
trust-all-certificates: false
truststore-path: /etc/ssl/opensearch-truststore.jks
indices:
default-shards: 3
default-replicas: 1
```
## 📋 Environment Variables
### Required
- `OPENSEARCH_PASSWORD`: Admin password for OpenSearch cluster
### Optional (with sensible defaults)
- `OPENSEARCH_HOST`: Cluster hostname (default: localhost)
- `OPENSEARCH_PORT`: Cluster port (default: 9200)
- `OPENSEARCH_USERNAME`: Admin username (default: admin)
- `OPENSEARCH_SSL_VERIFICATION`: Enable SSL verification (default: false for dev)
- `OPENSEARCH_MAX_CONN_TOTAL`: Max connections (default: 30 for dev, 200 for prod)
## 🎯 Index Templates
Index templates automatically apply configuration to new indexes:
```json
{
"index_patterns": ["stories_*"],
"template": {
"settings": {
"number_of_shards": "#{ENV_SPECIFIC}",
"analysis": {
"analyzer": {
"story_analyzer": {
"type": "standard",
"stopwords": "_english_"
}
}
}
}
}
}
```
## 🔍 Health Monitoring
Access health information:
- **Application Health**: `/actuator/health`
- **OpenSearch Specific**: `/actuator/health/opensearch`
- **Detailed Metrics**: Available when `enable-metrics: true`
## 🔄 Deployment Strategy
Recommended deployment approach:
1. **Development**: Test OpenSearch configuration locally
2. **Staging**: Validate performance and accuracy in staging environment
3. **Production**: Deploy with proper monitoring and backup procedures
## 🛠️ Troubleshooting
### Common Issues
1. **SSL Certificate Errors**
- Development: Set `trust-all-certificates: true`
- Production: Provide valid truststore path
2. **Connection Timeouts**
- Increase `connection.timeout` values
- Check network connectivity and firewall rules
3. **Index Creation Failures**
- Verify cluster health with `/actuator/health/opensearch`
- Check OpenSearch logs for detailed error messages
4. **Performance Issues**
- Monitor slow queries with configurable thresholds
- Adjust bulk operation settings
- Review shard allocation and replica settings
## 🔮 Future Enhancements
- **Multi-Cluster Support**: Connect to multiple OpenSearch clusters
- **Advanced Security**: Integration with OpenSearch Security plugin
- **Custom Analyzers**: Domain-specific text analysis
- **Index Aliases**: Zero-downtime index updates
- **Machine Learning**: Integration with OpenSearch ML features
---
This configuration provides a solid foundation that scales from development to enterprise production environments while maintaining security, performance, and operational excellence.

32
backend/src/main/resources/opensearch/config/opensearch-development.yml
View File

@@ -1,32 +0,0 @@
# OpenSearch Development Configuration
opensearch:
cluster:
name: "storycove-dev"
initial_master_nodes: ["opensearch-node"]
# Development settings - single node, minimal resources
indices:
default_settings:
number_of_shards: 1
number_of_replicas: 0
refresh_interval: "1s"
# Security settings for development
security:
ssl_verification: false
trust_all_certificates: true
# Connection settings
connection:
timeout: "30s"
socket_timeout: "60s"
max_connections_per_route: 10
max_connections_total: 30
# Index management
index_management:
auto_create_templates: true
template_patterns:
stories: "stories_*"
authors: "authors_*"
collections: "collections_*"

60
backend/src/main/resources/opensearch/config/opensearch-production.yml
View File

@@ -1,60 +0,0 @@
# OpenSearch Production Configuration
opensearch:
cluster:
name: "storycove-prod"
# Production settings - multi-shard, with replicas
indices:
default_settings:
number_of_shards: 3
number_of_replicas: 1
refresh_interval: "30s"
max_result_window: 50000
# Index lifecycle policies
lifecycle:
hot_phase_duration: "7d"
warm_phase_duration: "30d"
cold_phase_duration: "90d"
delete_after: "1y"
# Security settings for production
security:
ssl_verification: true
trust_all_certificates: false
certificate_verification: true
tls_version: "TLSv1.3"
# Connection settings
connection:
timeout: "10s"
socket_timeout: "30s"
max_connections_per_route: 50
max_connections_total: 200
retry_on_failure: true
max_retries: 3
retry_delay: "1s"
# Performance tuning
performance:
bulk_actions: 1000
bulk_size: "5MB"
bulk_timeout: "10s"
concurrent_requests: 4
# Monitoring and observability
monitoring:
health_check_interval: "30s"
slow_query_threshold: "5s"
enable_metrics: true
# Index management
index_management:
auto_create_templates: true
template_patterns:
stories: "stories_*"
authors: "authors_*"
collections: "collections_*"
retention_policy:
enabled: true
default_retention: "1y"

79
backend/src/main/resources/opensearch/mappings/authors-mapping.json
View File

@@ -1,79 +0,0 @@
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"name_analyzer": {
"type": "standard",
"stopwords": "_english_"
},
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "edge_ngram"]
}
},
"filter": {
"edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "text",
"analyzer": "name_analyzer",
"fields": {
"autocomplete": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"keyword": {
"type": "keyword"
}
}
},
"bio": {
"type": "text",
"analyzer": "name_analyzer"
},
"urls": {
"type": "keyword"
},
"imageUrl": {
"type": "keyword"
},
"storyCount": {
"type": "integer"
},
"averageRating": {
"type": "float"
},
"totalWordCount": {
"type": "long"
},
"totalReadingTime": {
"type": "integer"
},
"createdAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"updatedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"libraryId": {
"type": "keyword"
}
}
}
}

73
backend/src/main/resources/opensearch/mappings/collections-mapping.json
View File

@@ -1,73 +0,0 @@
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"collection_analyzer": {
"type": "standard",
"stopwords": "_english_"
},
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "edge_ngram"]
}
},
"filter": {
"edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "text",
"analyzer": "collection_analyzer",
"fields": {
"autocomplete": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"keyword": {
"type": "keyword"
}
}
},
"description": {
"type": "text",
"analyzer": "collection_analyzer"
},
"storyCount": {
"type": "integer"
},
"totalWordCount": {
"type": "long"
},
"averageRating": {
"type": "float"
},
"isPublic": {
"type": "boolean"
},
"createdAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"updatedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"libraryId": {
"type": "keyword"
}
}
}
}

120
backend/src/main/resources/opensearch/mappings/stories-mapping.json
View File

@@ -1,120 +0,0 @@
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"story_analyzer": {
"type": "standard",
"stopwords": "_english_"
},
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "edge_ngram"]
}
},
"filter": {
"edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"title": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"autocomplete": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"keyword": {
"type": "keyword"
}
}
},
"content": {
"type": "text",
"analyzer": "story_analyzer"
},
"summary": {
"type": "text",
"analyzer": "story_analyzer"
},
"authorNames": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"authorIds": {
"type": "keyword"
},
"tagNames": {
"type": "keyword"
},
"seriesTitle": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"seriesId": {
"type": "keyword"
},
"wordCount": {
"type": "integer"
},
"rating": {
"type": "float"
},
"readingTime": {
"type": "integer"
},
"language": {
"type": "keyword"
},
"status": {
"type": "keyword"
},
"createdAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"updatedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"publishedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"isRead": {
"type": "boolean"
},
"isFavorite": {
"type": "boolean"
},
"readingProgress": {
"type": "float"
},
"libraryId": {
"type": "keyword"
}
}
}
}

77
backend/src/main/resources/opensearch/templates/index-lifecycle-policy.json
View File

@@ -1,77 +0,0 @@
{
"policy": {
"description": "StoryCove index lifecycle policy",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_size": "50gb",
"min_doc_count": 1000000,
"min_age": "7d"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_age": "7d"
}
}
]
},
{
"name": "warm",
"actions": [
{
"replica_count": {
"number_of_replicas": 0
}
},
{
"force_merge": {
"max_num_segments": 1
}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_age": "30d"
}
}
]
},
{
"name": "cold",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_age": "365d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
]
}
],
"ism_template": [
{
"index_patterns": ["stories_*", "authors_*", "collections_*"],
"priority": 100
}
]
}
}

124
backend/src/main/resources/opensearch/templates/stories-template.json
View File

@@ -1,124 +0,0 @@
{
"index_patterns": ["stories_*"],
"priority": 1,
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"analyzer": {
"story_analyzer": {
"type": "standard",
"stopwords": "_english_"
},
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "edge_ngram"]
}
},
"filter": {
"edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 20
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"title": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"autocomplete": {
"type": "text",
"analyzer": "autocomplete_analyzer"
},
"keyword": {
"type": "keyword"
}
}
},
"content": {
"type": "text",
"analyzer": "story_analyzer"
},
"summary": {
"type": "text",
"analyzer": "story_analyzer"
},
"authorNames": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"authorIds": {
"type": "keyword"
},
"tagNames": {
"type": "keyword"
},
"seriesTitle": {
"type": "text",
"analyzer": "story_analyzer",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"seriesId": {
"type": "keyword"
},
"wordCount": {
"type": "integer"
},
"rating": {
"type": "float"
},
"readingTime": {
"type": "integer"
},
"language": {
"type": "keyword"
},
"status": {
"type": "keyword"
},
"createdAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"updatedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"publishedAt": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"isRead": {
"type": "boolean"
},
"isFavorite": {
"type": "boolean"
},
"readingProgress": {
"type": "float"
},
"libraryId": {
"type": "keyword"
}
}
}
}
}

9
backend/src/test/resources/application-test.yml
View File

@@ -19,11 +19,14 @@ storycove:
auth: auth:
password: test-password password: test-password
search: search:
engine: opensearch engine: solr
opensearch: solr:
host: localhost host: localhost
port: 9200 port: 8983
scheme: http scheme: http
cores:
stories: storycove_stories
authors: storycove_authors
images: images:
storage-path: /tmp/test-images storage-path: /tmp/test-images

84
docker-compose.yml
View File

@@ -34,18 +34,10 @@ services:
- SPRING_DATASOURCE_USERNAME=storycove - SPRING_DATASOURCE_USERNAME=storycove
- SPRING_DATASOURCE_PASSWORD=${DB_PASSWORD} - SPRING_DATASOURCE_PASSWORD=${DB_PASSWORD}
- JWT_SECRET=${JWT_SECRET} - JWT_SECRET=${JWT_SECRET}
- OPENSEARCH_HOST=opensearch - SOLR_HOST=solr
- OPENSEARCH_PORT=9200 - SOLR_PORT=8983
- OPENSEARCH_SCHEME=http - SOLR_SCHEME=http
- OPENSEARCH_USERNAME= - SEARCH_ENGINE=${SEARCH_ENGINE:-solr}
- OPENSEARCH_PASSWORD=
- OPENSEARCH_SSL_VERIFICATION=false
- OPENSEARCH_TRUST_ALL_CERTS=true
- OPENSEARCH_CONNECTION_TIMEOUT=60000
- OPENSEARCH_SOCKET_TIMEOUT=120000
- OPENSEARCH_MAX_RETRIES=5
- OPENSEARCH_RETRY_ON_FAILURE=true
- SEARCH_ENGINE=${SEARCH_ENGINE:-opensearch}
- IMAGE_STORAGE_PATH=/app/images - IMAGE_STORAGE_PATH=/app/images
- APP_PASSWORD=${APP_PASSWORD} - APP_PASSWORD=${APP_PASSWORD}
- STORYCOVE_CORS_ALLOWED_ORIGINS=${STORYCOVE_CORS_ALLOWED_ORIGINS:-http://localhost:3000,http://localhost:6925} - STORYCOVE_CORS_ALLOWED_ORIGINS=${STORYCOVE_CORS_ALLOWED_ORIGINS:-http://localhost:3000,http://localhost:6925}
@@ -55,7 +47,7 @@ services:
depends_on: depends_on:
postgres: postgres:
condition: service_started condition: service_started
opensearch: solr:
condition: service_started condition: service_started
networks: networks:
- storycove-network - storycove-network
@@ -75,22 +67,32 @@ services:
- storycove-network - storycove-network
opensearch: solr:
build: image: solr:9.9.0
context: . ports:
dockerfile: opensearch.Dockerfile - "8983:8983" # Expose Solr Admin UI for development
# No port mapping - only accessible within the Docker network
environment: environment:
- cluster.name=storycove-opensearch - SOLR_HEAP=512m
- node.name=opensearch-node - SOLR_JAVA_MEM=-Xms256m -Xmx512m
- discovery.type=single-node volumes:
- bootstrap.memory_lock=false - solr_data:/var/solr
- "OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx512m --add-opens=java.base/java.lang=ALL-UNNAMED -Dlucene.useVectorAPI=false -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false" - ./solr/stories:/opt/solr-9.9.0/server/solr/configsets/storycove_stories
- "DISABLE_INSTALL_DEMO_CONFIG=true" - ./solr/authors:/opt/solr-9.9.0/server/solr/configsets/storycove_authors
- "DISABLE_SECURITY_PLUGIN=true" command: >
- "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true" sh -c "
- "ES_TMPDIR=/tmp" echo 'Starting Solr...' &&
- "_JAVA_OPTIONS=-Djdk.net.useExclusiveBind=false" solr start -f &
SOLR_PID=$$! &&
echo 'Waiting for Solr to be ready...' &&
sleep 15 &&
echo 'Creating cores...' &&
(solr create_core -c storycove_stories -d storycove_stories || echo 'Stories core already exists') &&
echo 'Stories core ready' &&
(solr create_core -c storycove_authors -d storycove_authors || echo 'Authors core already exists') &&
echo 'Authors core ready' &&
echo 'Both cores are available' &&
wait $$SOLR_PID
"
deploy: deploy:
resources: resources:
limits: limits:
@@ -99,39 +101,19 @@ services:
memory: 512M memory: 512M
stop_grace_period: 30s stop_grace_period: 30s
healthcheck: healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"] test: ["CMD-SHELL", "curl -f http://localhost:8983/solr/admin/ping || exit 1"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 5 retries: 5
start_period: 120s start_period: 60s
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch_data:/usr/share/opensearch/data
networks: networks:
- storycove-network - storycove-network
restart: unless-stopped restart: unless-stopped
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.2.0
ports:
- "5601:5601" # Expose OpenSearch Dashboard
environment:
- OPENSEARCH_HOSTS=http://opensearch:9200
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"
depends_on:
- opensearch
networks:
- storycove-network
volumes: volumes:
postgres_data: postgres_data:
opensearch_data: solr_data:
images_data: images_data:
library_config: library_config:

4
frontend/src/app/library/LibraryContent.tsx
View File

@@ -69,11 +69,11 @@ export default function LibraryContent() {
}, []); }, []);
const convertFacetsToTags = (facets?: Record<string, FacetCount[]>): Tag[] => { const convertFacetsToTags = (facets?: Record<string, FacetCount[]>): Tag[] => {
if (!facets || !facets.tagNames) { if (!facets || !facets.tagNames_facet) {
return []; return [];
} }
return facets.tagNames.map(facet => { return facets.tagNames_facet.map(facet => {
// Find the full tag data by name // Find the full tag data by name
const fullTag = fullTags.find(tag => tag.name.toLowerCase() === facet.value.toLowerCase()); const fullTag = fullTags.find(tag => tag.name.toLowerCase() === facet.value.toLowerCase());

12
frontend/src/app/scrape/bulk/route.ts
View File

@@ -493,11 +493,11 @@ async function processIndividualMode(
console.log(`Bulk import completed: ${importedCount} imported, ${skippedCount} skipped, ${errorCount} errors`); console.log(`Bulk import completed: ${importedCount} imported, ${skippedCount} skipped, ${errorCount} errors`);
// Trigger OpenSearch reindex if any stories were imported // Trigger Solr reindex if any stories were imported
if (importedCount > 0) { if (importedCount > 0) {
try { try {
console.log('Triggering OpenSearch reindex after bulk import...'); console.log('Triggering Solr reindex after bulk import...');
const reindexUrl = `http://backend:8080/api/admin/search/opensearch/reindex`; const reindexUrl = `http://backend:8080/api/admin/search/solr/reindex`;
const reindexResponse = await fetch(reindexUrl, { const reindexResponse = await fetch(reindexUrl, {
method: 'POST', method: 'POST',
headers: { headers: {
@@ -508,12 +508,12 @@ async function processIndividualMode(
if (reindexResponse.ok) { if (reindexResponse.ok) {
const reindexResult = await reindexResponse.json(); const reindexResult = await reindexResponse.json();
console.log('OpenSearch reindex completed:', reindexResult); console.log('Solr reindex completed:', reindexResult);
} else { } else {
console.warn('OpenSearch reindex failed:', reindexResponse.status); console.warn('Solr reindex failed:', reindexResponse.status);
} }
} catch (error) { } catch (error) {
console.warn('Failed to trigger OpenSearch reindex:', error); console.warn('Failed to trigger Solr reindex:', error);
// Don't fail the whole request if reindex fails // Don't fail the whole request if reindex fails
} }
} }

78
frontend/src/components/settings/SystemSettings.tsx
View File

@@ -11,18 +11,18 @@ interface SystemSettingsProps {
export default function SystemSettings({}: SystemSettingsProps) { export default function SystemSettings({}: SystemSettingsProps) {
const [searchEngineStatus, setSearchEngineStatus] = useState<{ const [searchEngineStatus, setSearchEngineStatus] = useState<{
currentEngine: string; currentEngine: string;
openSearchAvailable: boolean; solrAvailable: boolean;
loading: boolean; loading: boolean;
message: string; message: string;
success?: boolean; success?: boolean;
}>({ }>({
currentEngine: 'opensearch', currentEngine: 'solr',
openSearchAvailable: false, solrAvailable: false,
loading: false, loading: false,
message: '' message: ''
}); });
const [openSearchStatus, setOpenSearchStatus] = useState<{ const [solrStatus, setSolrStatus] = useState<{
reindex: { loading: boolean; message: string; success?: boolean }; reindex: { loading: boolean; message: string; success?: boolean };
recreate: { loading: boolean; message: string; success?: boolean }; recreate: { loading: boolean; message: string; success?: boolean };
}>({ }>({
@@ -312,7 +312,7 @@ export default function SystemSettings({}: SystemSettingsProps) {
setSearchEngineStatus(prev => ({ setSearchEngineStatus(prev => ({
...prev, ...prev,
currentEngine: status.primaryEngine, currentEngine: status.primaryEngine,
openSearchAvailable: status.openSearchAvailable, solrAvailable: status.solrAvailable,
})); }));
} catch (error: any) { } catch (error: any) {
console.error('Failed to load search engine status:', error); console.error('Failed to load search engine status:', error);
@@ -321,16 +321,16 @@ export default function SystemSettings({}: SystemSettingsProps) {
const handleOpenSearchReindex = async () => { const handleSolrReindex = async () => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
reindex: { loading: true, message: 'Reindexing OpenSearch...', success: undefined } reindex: { loading: true, message: 'Reindexing Solr...', success: undefined }
})); }));
try { try {
const result = await searchAdminApi.reindexOpenSearch(); const result = await searchAdminApi.reindexSolr();
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
reindex: { reindex: {
loading: false, loading: false,
@@ -340,13 +340,13 @@ export default function SystemSettings({}: SystemSettingsProps) {
})); }));
setTimeout(() => { setTimeout(() => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
reindex: { loading: false, message: '', success: undefined } reindex: { loading: false, message: '', success: undefined }
})); }));
}, 8000); }, 8000);
} catch (error: any) { } catch (error: any) {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
reindex: { reindex: {
loading: false, loading: false,
@@ -356,7 +356,7 @@ export default function SystemSettings({}: SystemSettingsProps) {
})); }));
setTimeout(() => { setTimeout(() => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
reindex: { loading: false, message: '', success: undefined } reindex: { loading: false, message: '', success: undefined }
})); }));
@@ -364,16 +364,16 @@ export default function SystemSettings({}: SystemSettingsProps) {
} }
}; };
const handleOpenSearchRecreate = async () => { const handleSolrRecreate = async () => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
recreate: { loading: true, message: 'Recreating OpenSearch indices...', success: undefined } recreate: { loading: true, message: 'Recreating Solr indices...', success: undefined }
})); }));
try { try {
const result = await searchAdminApi.recreateOpenSearchIndices(); const result = await searchAdminApi.recreateSolrIndices();
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
recreate: { recreate: {
loading: false, loading: false,
@@ -383,13 +383,13 @@ export default function SystemSettings({}: SystemSettingsProps) {
})); }));
setTimeout(() => { setTimeout(() => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
recreate: { loading: false, message: '', success: undefined } recreate: { loading: false, message: '', success: undefined }
})); }));
}, 8000); }, 8000);
} catch (error: any) { } catch (error: any) {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
recreate: { recreate: {
loading: false, loading: false,
@@ -399,7 +399,7 @@ export default function SystemSettings({}: SystemSettingsProps) {
})); }));
setTimeout(() => { setTimeout(() => {
setOpenSearchStatus(prev => ({ setSolrStatus(prev => ({
...prev, ...prev,
recreate: { loading: false, message: '', success: undefined } recreate: { loading: false, message: '', success: undefined }
})); }));
@@ -418,7 +418,7 @@ export default function SystemSettings({}: SystemSettingsProps) {
<div className="theme-card theme-shadow rounded-lg p-6"> <div className="theme-card theme-shadow rounded-lg p-6">
<h2 className="text-xl font-semibold theme-header mb-4">Search Management</h2> <h2 className="text-xl font-semibold theme-header mb-4">Search Management</h2>
<p className="theme-text mb-6"> <p className="theme-text mb-6">
Manage OpenSearch indices for stories and authors. Use these tools if search isn't returning expected results. Manage Solr indices for stories and authors. Use these tools if search isn't returning expected results.
</p> </p>
<div className="space-y-6"> <div className="space-y-6">
@@ -427,9 +427,9 @@ export default function SystemSettings({}: SystemSettingsProps) {
<h3 className="text-lg font-semibold theme-header mb-3">Search Status</h3> <h3 className="text-lg font-semibold theme-header mb-3">Search Status</h3>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 text-sm"> <div className="grid grid-cols-1 sm:grid-cols-2 gap-3 text-sm">
<div className="flex justify-between"> <div className="flex justify-between">
<span>OpenSearch:</span> <span>Solr:</span>
<span className={`font-medium ${searchEngineStatus.openSearchAvailable ? 'text-green-600 dark:text-green-400' : 'text-red-600 dark:text-red-400'}`}> <span className={`font-medium ${searchEngineStatus.solrAvailable ? 'text-green-600 dark:text-green-400' : 'text-red-600 dark:text-red-400'}`}>
{searchEngineStatus.openSearchAvailable ? 'Available' : 'Unavailable'} {searchEngineStatus.solrAvailable ? 'Available' : 'Unavailable'}
</span> </span>
</div> </div>
</div> </div>
@@ -444,43 +444,43 @@ export default function SystemSettings({}: SystemSettingsProps) {
<div className="flex flex-col sm:flex-row gap-3 mb-4"> <div className="flex flex-col sm:flex-row gap-3 mb-4">
<Button <Button
onClick={handleOpenSearchReindex} onClick={handleSolrReindex}
disabled={openSearchStatus.reindex.loading || openSearchStatus.recreate.loading || !searchEngineStatus.openSearchAvailable} disabled={solrStatus.reindex.loading || solrStatus.recreate.loading || !searchEngineStatus.solrAvailable}
loading={openSearchStatus.reindex.loading} loading={solrStatus.reindex.loading}
variant="ghost" variant="ghost"
className="flex-1" className="flex-1"
> >
{openSearchStatus.reindex.loading ? 'Reindexing...' : '🔄 Reindex All'} {solrStatus.reindex.loading ? 'Reindexing...' : '🔄 Reindex All'}
</Button> </Button>
<Button <Button
onClick={handleOpenSearchRecreate} onClick={handleSolrRecreate}
disabled={openSearchStatus.reindex.loading || openSearchStatus.recreate.loading || !searchEngineStatus.openSearchAvailable} disabled={solrStatus.reindex.loading || solrStatus.recreate.loading || !searchEngineStatus.solrAvailable}
loading={openSearchStatus.recreate.loading} loading={solrStatus.recreate.loading}
variant="secondary" variant="secondary"
className="flex-1" className="flex-1"
> >
{openSearchStatus.recreate.loading ? 'Recreating...' : '🏗️ Recreate Indices'} {solrStatus.recreate.loading ? 'Recreating...' : '🏗️ Recreate Indices'}
</Button> </Button>
</div> </div>
{/* Status Messages */} {/* Status Messages */}
{openSearchStatus.reindex.message && ( {solrStatus.reindex.message && (
<div className={`text-sm p-3 rounded mb-3 ${ <div className={`text-sm p-3 rounded mb-3 ${
openSearchStatus.reindex.success solrStatus.reindex.success
? 'bg-green-50 dark:bg-green-900/20 text-green-800 dark:text-green-200' ? 'bg-green-50 dark:bg-green-900/20 text-green-800 dark:text-green-200'
: 'bg-red-50 dark:bg-red-900/20 text-red-800 dark:text-red-200' : 'bg-red-50 dark:bg-red-900/20 text-red-800 dark:text-red-200'
}`}> }`}>
{openSearchStatus.reindex.message} {solrStatus.reindex.message}
</div> </div>
)} )}
{openSearchStatus.recreate.message && ( {solrStatus.recreate.message && (
<div className={`text-sm p-3 rounded mb-3 ${ <div className={`text-sm p-3 rounded mb-3 ${
openSearchStatus.recreate.success solrStatus.recreate.success
? 'bg-green-50 dark:bg-green-900/20 text-green-800 dark:text-green-200' ? 'bg-green-50 dark:bg-green-900/20 text-green-800 dark:text-green-200'
: 'bg-red-50 dark:bg-red-900/20 text-red-800 dark:text-red-200' : 'bg-red-50 dark:bg-red-900/20 text-red-800 dark:text-red-200'
}`}> }`}>
{openSearchStatus.recreate.message} {solrStatus.recreate.message}
</div> </div>
)} )}
</div> </div>

16
frontend/src/lib/api.ts
View File

@@ -576,7 +576,7 @@ export const searchAdminApi = {
getStatus: async (): Promise<{ getStatus: async (): Promise<{
primaryEngine: string; primaryEngine: string;
dualWrite: boolean; dualWrite: boolean;
openSearchAvailable: boolean; solrAvailable: boolean;
}> => { }> => {
const response = await api.get('/admin/search/status'); const response = await api.get('/admin/search/status');
return response.data; return response.data;
@@ -600,8 +600,8 @@ export const searchAdminApi = {
}, },
// Switch engines // Switch engines
switchToOpenSearch: async (): Promise<{ message: string }> => { switchToSolr: async (): Promise<{ message: string }> => {
const response = await api.post('/admin/search/switch/opensearch'); const response = await api.post('/admin/search/switch/solr');
return response.data; return response.data;
}, },
@@ -612,8 +612,8 @@ export const searchAdminApi = {
return response.data; return response.data;
}, },
// OpenSearch operations // Solr operations
reindexOpenSearch: async (): Promise<{ reindexSolr: async (): Promise<{
success: boolean; success: boolean;
message: string; message: string;
storiesCount?: number; storiesCount?: number;
@@ -621,11 +621,11 @@ export const searchAdminApi = {
totalCount?: number; totalCount?: number;
error?: string; error?: string;
}> => { }> => {
const response = await api.post('/admin/search/opensearch/reindex'); const response = await api.post('/admin/search/solr/reindex');
return response.data; return response.data;
}, },
recreateOpenSearchIndices: async (): Promise<{ recreateSolrIndices: async (): Promise<{
success: boolean; success: boolean;
message: string; message: string;
storiesCount?: number; storiesCount?: number;
@@ -633,7 +633,7 @@ export const searchAdminApi = {
totalCount?: number; totalCount?: number;
error?: string; error?: string;
}> => { }> => {
const response = await api.post('/admin/search/opensearch/recreate'); const response = await api.post('/admin/search/solr/recreate');
return response.data; return response.data;
}, },
}; };

7
opensearch.Dockerfile
View File

@@ -51,12 +51,15 @@ RUN mkdir -p /usr/share/opensearch/data && \
echo "-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-Djava.awt.headless=true" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "-Djava.awt.headless=true" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-XX:+UseContainerSupport" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "-XX:+UseContainerSupport" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-Dorg.opensearch.bootstrap.start_timeout=60s" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "-Dorg.opensearch.bootstrap.start_timeout=300s" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-Dopensearch.logger.level=INFO" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "-Dopensearch.logger.level=INFO" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "--add-opens=jdk.unsupported/sun.misc=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "--add-opens=jdk.unsupported/sun.misc=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "--add-opens=java.base/java.util=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "--add-opens=java.base/java.util=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "--add-opens=java.base/java.lang=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "--add-opens=java.base/java.lang=ALL-UNNAMED" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "--add-modules=jdk.unsupported" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \ echo "--add-modules=jdk.unsupported" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-XX:+UnlockExperimentalVMOptions" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-XX:-UseVectorApi" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
echo "-Djdk.incubator.vector.VECTOR_ACCESS_OOB_CHECK=0" >> /usr/share/opensearch/config/jvm.options.d/synology.options && \
sed -i '/javaagent/d' /usr/share/opensearch/config/jvm.options && \ sed -i '/javaagent/d' /usr/share/opensearch/config/jvm.options && \
echo '#!/bin/bash' > /usr/share/opensearch/start-opensearch.sh && \ echo '#!/bin/bash' > /usr/share/opensearch/start-opensearch.sh && \
echo 'echo "Starting OpenSearch with Java 21..."' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'echo "Starting OpenSearch with Java 21..."' >> /usr/share/opensearch/start-opensearch.sh && \
@@ -73,7 +76,7 @@ RUN mkdir -p /usr/share/opensearch/data && \
echo 'cat /usr/share/opensearch/config/jvm.options.d/synology.options' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'cat /usr/share/opensearch/config/jvm.options.d/synology.options' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'echo "Environment OPENSEARCH_JAVA_OPTS: $OPENSEARCH_JAVA_OPTS"' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'echo "Environment OPENSEARCH_JAVA_OPTS: $OPENSEARCH_JAVA_OPTS"' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'echo "Attempting to force disable vector operations..."' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'echo "Attempting to force disable vector operations..."' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'export OPENSEARCH_JAVA_OPTS="$OPENSEARCH_JAVA_OPTS -Dlucene.useVectorAPI=false -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false"' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'export OPENSEARCH_JAVA_OPTS="$OPENSEARCH_JAVA_OPTS -Dlucene.useVectorAPI=false -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false --limit-modules=java.base,java.logging,java.xml,java.management,java.naming,java.desktop,java.security.jgss,jdk.unsupported"' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'echo "Final OPENSEARCH_JAVA_OPTS: $OPENSEARCH_JAVA_OPTS"' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'echo "Final OPENSEARCH_JAVA_OPTS: $OPENSEARCH_JAVA_OPTS"' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'echo "Starting OpenSearch binary..."' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'echo "Starting OpenSearch binary..."' >> /usr/share/opensearch/start-opensearch.sh && \
echo 'timeout 300s /usr/share/opensearch/bin/opensearch &' >> /usr/share/opensearch/start-opensearch.sh && \ echo 'timeout 300s /usr/share/opensearch/bin/opensearch &' >> /usr/share/opensearch/start-opensearch.sh && \

93
solr/authors/conf/managed-schema Executable file
View File

@@ -0,0 +1,93 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Solr Schema for StoryCove Authors Core
Based on AuthorSearchDto data model
-->
<schema name="storycove-authors" version="1.6">
<!-- Field Types -->
<!-- String field type for exact matching -->
<fieldType name="string" class="solr.StrField" sortMissingLast="true" />
<!-- Text field type for full-text search -->
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Enhanced text field for names -->
<fieldType name="text_enhanced" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="15"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Integer field type -->
<fieldType name="pint" class="solr.IntPointField" docValues="true"/>
<!-- Long field type -->
<fieldType name="plong" class="solr.LongPointField" docValues="true"/>
<!-- Double field type -->
<fieldType name="pdouble" class="solr.DoublePointField" docValues="true"/>
<!-- Date field type -->
<fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
<!-- Multi-valued string for URLs -->
<fieldType name="strings" class="solr.StrField" sortMissingLast="true" multiValued="true"/>
<!-- Fields -->
<!-- Required Fields -->
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="_version_" type="plong" indexed="false" stored="false"/>
<!-- Core Author Fields -->
<field name="name" type="text_enhanced" indexed="true" stored="true" required="true"/>
<field name="notes" type="text_general" indexed="true" stored="true"/>
<field name="authorRating" type="pint" indexed="true" stored="true"/>
<field name="averageStoryRating" type="pdouble" indexed="true" stored="true"/>
<field name="storyCount" type="pint" indexed="true" stored="true"/>
<field name="urls" type="strings" indexed="true" stored="true"/>
<field name="avatarImagePath" type="string" indexed="false" stored="true"/>
<!-- Timestamp Fields -->
<field name="createdAt" type="pdate" indexed="true" stored="true"/>
<field name="updatedAt" type="pdate" indexed="true" stored="true"/>
<!-- Search-specific Fields -->
<field name="searchScore" type="plong" indexed="false" stored="true"/>
<!-- Combined search field for general queries -->
<field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>
<!-- Copy Fields for comprehensive search -->
<copyField source="name" dest="text"/>
<copyField source="notes" dest="text"/>
<copyField source="urls" dest="text"/>
<!-- Default Search Field -->
<!-- UniqueKey -->
<uniqueKey>id</uniqueKey>
</schema>

140
solr/authors/conf/solrconfig.xml Executable file
View File

@@ -0,0 +1,140 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Solr Configuration for StoryCove Authors Core
Optimized for author search with highlighting and faceting
-->
<config>
<luceneMatchVersion>9.9.0</luceneMatchVersion>
<!-- DataDir configuration -->
<dataDir>${solr.data.dir:}</dataDir>
<!-- Directory Factory -->
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<!-- CodecFactory -->
<codecFactory class="solr.SchemaCodecFactory"/>
<!-- Index Configuration -->
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
<infoStream>true</infoStream>
</indexConfig>
<!-- JMX Configuration -->
<jmx />
<!-- Update Handler -->
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
<int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
</updateHandler>
<!-- Query Configuration -->
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
</query>
<!-- Request Dispatcher -->
<requestDispatcher handleSelect="false" >
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000"
formdataUploadLimitInKB="2048"
addHttpRequestToContext="false"/>
<httpCaching never304="true" />
</requestDispatcher>
<!-- Request Handlers -->
<!-- Standard Select Handler -->
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="hl">true</str>
<str name="hl.fl">name,notes</str>
<str name="hl.simple.pre">&lt;em&gt;</str>
<str name="hl.simple.post">&lt;/em&gt;</str>
<str name="hl.fragsize">150</str>
<str name="hl.maxAnalyzedChars">51200</str>
<str name="facet">true</str>
<str name="facet.field">authorRating</str>
<str name="facet.range">averageStoryRating</str>
<str name="facet.range">storyCount</str>
<str name="facet.mincount">1</str>
<str name="facet.sort">count</str>
</lst>
</requestHandler>
<!-- Update Handler -->
<requestHandler name="/update" class="solr.UpdateRequestHandler" />
<!-- Admin Handlers -->
<requestHandler name="/admin/ping" class="solr.PingRequestHandler">
<lst name="invariants">
<str name="q">*:*</str>
</lst>
<lst name="defaults">
<str name="echoParams">all</str>
</lst>
</requestHandler>
<!-- Suggester Handler -->
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
<!-- Search Components -->
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">authorSuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">name</str>
<str name="weightField">storyCount</str>
<str name="suggestAnalyzerFieldType">text_general</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<!-- Response Writers -->
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">application/json; charset=UTF-8</str>
</queryResponseWriter>
</config>

34
solr/authors/conf/stopwords.txt Executable file
View File

@@ -0,0 +1,34 @@
# English stopwords for author search
a
an
and
are
as
at
be
but
by
for
if
in
into
is
it
no
not
of
on
or
such
that
the
their
then
there
these
they
this
to
was
will
with

9
solr/authors/conf/synonyms.txt Executable file
View File

@@ -0,0 +1,9 @@
# Synonyms for author search
# Format: word1,word2,word3 => synonym1,synonym2
writer,author,novelist
pen name,pseudonym,alias
prolific,productive
acclaimed,famous,renowned
bestselling,popular
contemporary,modern
classic,traditional

129
solr/stories/conf/managed-schema Executable file
View File

@@ -0,0 +1,129 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Solr Schema for StoryCove Stories Core
Based on StorySearchDto data model
-->
<schema name="storycove-stories" version="1.6">
<!-- Field Types -->
<!-- String field type for exact matching -->
<fieldType name="string" class="solr.StrField" sortMissingLast="true" />
<!-- Text field type for full-text search -->
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Enhanced text field for titles and important content -->
<fieldType name="text_enhanced" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="15"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- Integer field type -->
<fieldType name="pint" class="solr.IntPointField" docValues="true"/>
<!-- Long field type -->
<fieldType name="plong" class="solr.LongPointField" docValues="true"/>
<!-- Double field type -->
<fieldType name="pdouble" class="solr.DoublePointField" docValues="true"/>
<!-- Boolean field type -->
<fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
<!-- Date field type -->
<fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
<!-- Multi-valued string for tags and faceting -->
<fieldType name="strings" class="solr.StrField" sortMissingLast="true" multiValued="true" docValues="true"/>
<!-- Single string for exact matching and faceting -->
<fieldType name="string_facet" class="solr.StrField" sortMissingLast="true" docValues="true"/>
<!-- Fields -->
<!-- Required Fields -->
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="_version_" type="plong" indexed="false" stored="false"/>
<!-- Core Story Fields -->
<field name="title" type="text_enhanced" indexed="true" stored="true" required="true"/>
<field name="description" type="text_general" indexed="true" stored="true"/>
<field name="sourceUrl" type="string" indexed="true" stored="true"/>
<field name="coverPath" type="string" indexed="false" stored="true"/>
<field name="wordCount" type="pint" indexed="true" stored="true"/>
<field name="rating" type="pint" indexed="true" stored="true"/>
<field name="averageStoryRating" type="pdouble" indexed="true" stored="true"/>
<field name="volume" type="pint" indexed="true" stored="true"/>
<!-- Reading Status Fields -->
<field name="isRead" type="boolean" indexed="true" stored="true"/>
<field name="readingPosition" type="pint" indexed="true" stored="true"/>
<field name="lastReadAt" type="pdate" indexed="true" stored="true"/>
<field name="lastRead" type="pdate" indexed="true" stored="true"/>
<!-- Author Fields -->
<field name="authorId" type="string" indexed="true" stored="true"/>
<field name="authorName" type="text_enhanced" indexed="true" stored="true"/>
<field name="authorName_facet" type="string_facet" indexed="true" stored="false"/>
<!-- Series Fields -->
<field name="seriesId" type="string" indexed="true" stored="true"/>
<field name="seriesName" type="text_enhanced" indexed="true" stored="true"/>
<field name="seriesName_facet" type="string_facet" indexed="true" stored="false"/>
<!-- Tag Fields -->
<field name="tagNames" type="strings" indexed="true" stored="true"/>
<field name="tagNames_facet" type="strings" indexed="true" stored="false"/>
<!-- Timestamp Fields -->
<field name="createdAt" type="pdate" indexed="true" stored="true"/>
<field name="updatedAt" type="pdate" indexed="true" stored="true"/>
<field name="dateAdded" type="pdate" indexed="true" stored="true"/>
<!-- Search-specific Fields -->
<field name="searchScore" type="pdouble" indexed="false" stored="true"/>
<field name="highlights" type="strings" indexed="false" stored="true"/>
<!-- Combined search field for general queries -->
<field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>
<!-- Copy Fields for comprehensive search -->
<copyField source="title" dest="text"/>
<copyField source="description" dest="text"/>
<copyField source="authorName" dest="text"/>
<copyField source="seriesName" dest="text"/>
<copyField source="tagNames" dest="text"/>
<!-- Copy Fields for faceting -->
<copyField source="authorName" dest="authorName_facet"/>
<copyField source="seriesName" dest="seriesName_facet"/>
<copyField source="tagNames" dest="tagNames_facet"/>
<!-- Default Search Field -->
<!-- UniqueKey -->
<uniqueKey>id</uniqueKey>
</schema>

153
solr/stories/conf/solrconfig.xml Executable file
View File

@@ -0,0 +1,153 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Solr Configuration for StoryCove Stories Core
Optimized for story search with highlighting and faceting
-->
<config>
<luceneMatchVersion>9.9.0</luceneMatchVersion>
<!-- DataDir configuration -->
<dataDir>${solr.data.dir:}</dataDir>
<!-- Directory Factory -->
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<!-- CodecFactory -->
<codecFactory class="solr.SchemaCodecFactory"/>
<!-- Index Configuration -->
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
<infoStream>true</infoStream>
</indexConfig>
<!-- JMX Configuration -->
<jmx />
<!-- Update Handler -->
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
<int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
</updateHandler>
<!-- Query Configuration -->
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
</query>
<!-- Request Dispatcher -->
<requestDispatcher handleSelect="false" >
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000"
formdataUploadLimitInKB="2048"
addHttpRequestToContext="false"/>
<httpCaching never304="true" />
</requestDispatcher>
<!-- Request Handlers -->
<!-- Standard Select Handler -->
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="hl">true</str>
<str name="hl.fl">title,description</str>
<str name="hl.simple.pre">&lt;em&gt;</str>
<str name="hl.simple.post">&lt;/em&gt;</str>
<str name="hl.fragsize">150</str>
<str name="hl.maxAnalyzedChars">51200</str>
<str name="facet">true</str>
<str name="facet.field">authorName</str>
<str name="facet.field">tagNames</str>
<str name="facet.field">seriesName</str>
<str name="facet.field">rating</str>
<str name="facet.field">isRead</str>
<str name="facet.mincount">1</str>
<str name="facet.sort">count</str>
</lst>
</requestHandler>
<!-- Update Handler -->
<requestHandler name="/update" class="solr.UpdateRequestHandler" />
<!-- Admin Handlers -->
<requestHandler name="/admin/ping" class="solr.PingRequestHandler">
<lst name="invariants">
<str name="q">*:*</str>
</lst>
<lst name="defaults">
<str name="echoParams">all</str>
</lst>
</requestHandler>
<!-- More Like This Handler -->
<requestHandler name="/mlt" class="solr.MoreLikeThisHandler">
<lst name="defaults">
<str name="mlt.fl">title,description</str>
<int name="mlt.mindf">2</int>
<int name="mlt.mintf">2</int>
<str name="mlt.qf">title^2.0 description^1.0</str>
<int name="rows">5</int>
</lst>
</requestHandler>
<!-- Suggester Handler -->
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
<!-- Search Components -->
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">storySuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">title</str>
<str name="weightField">rating</str>
<str name="suggestAnalyzerFieldType">text_general</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<!-- Response Writers -->
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">application/json; charset=UTF-8</str>
</queryResponseWriter>
</config>

34
solr/stories/conf/stopwords.txt Executable file
View File

@@ -0,0 +1,34 @@
# English stopwords for story search
a
an
and
are
as
at
be
but
by
for
if
in
into
is
it
no
not
of
on
or
such
that
the
their
then
there
these
they
this
to
was
will
with

16
solr/stories/conf/synonyms.txt Executable file
View File

@@ -0,0 +1,16 @@
# Synonyms for story search
# Format: word1,word2,word3 => synonym1,synonym2
fantasy,magical,magic
sci-fi,science fiction,scifi
romance,romantic,love
mystery,detective,crime
adventure,action
horror,scary,frightening
drama,dramatic
comedy,funny,humor
thriller,suspense
historical,history
contemporary,modern
short,brief
novel,book
story,tale,narrative