- System Overview
- Architecture
- Quick Start Guide
- Core Components
- Templates
- Processing Pipeline
- Data Flow
- Block Types and Parsing
- Todo Management
- Activity System
- Mentions and Cross-References
- File Structure
- Configuration
- API Reference
- Testing
- Performance Optimization
- Security Considerations
- Troubleshooting
- Development Guidelines
- Examples and Use Cases
- Migration Guide
- FAQ
This is a sophisticated Personal Knowledge Management (PKM) system built on top of Obsidian, utilizing DataviewJS and CustomJS plugins to create an automated, intelligent note-taking and task management system.
- Automated Daily Notes: Self-updating daily notes with todo rollover
- Activity Management: Project-based task organization with automatic processing
- Smart Todo System: Context-aware todo rollover and activity association
- Cross-Reference System: Automatic mention processing and content aggregation
- Template-Based Architecture: Consistent note structure and processing
- Obsidian: Base note-taking application
- DataviewJS: Dynamic content generation and queries
- CustomJS: Shared JavaScript modules and utilities
- Moment.js: Date manipulation and formatting
- 🎉 Templater Plugin Eliminated: The system now uses pure DataviewJS for all template processing, eliminating the need for the Templater plugin dependency
- Smart File Movement: Non-date files are automatically moved to Activities folder with proper template replacement
- Cleaner Architecture: Simplified template structure using composer pattern for better maintainability
┌─────────────────────────────────────────────────────────────┐
│ OBSIDIAN VAULT │
├─────────────────────────────────────────────────────────────┤
│ Templates/ │ Scripts/ │ Journal/ │
│ ├─ DailyNote │ ├─ Composers │ └─ YYYY/ │
│ ├─ Activity │ ├─ Components │ └─ MM.Mon/ │
│ └─ People │ └─ Utilities │ └─ DD │
├─────────────────────────────────────────────────────────────┤
│ Activities/ │ People/ │ Spaces/ │
│ ├─ Active │ ├─ Contacts │ └─ Topics │
│ ├─ Archive/ │ └─ Meetings/ │ │
│ └─ Workflow/ │ │ │
└─────────────────────────────────────────────────────────────┘
Purpose: Orchestrates the complete daily note processing pipeline.
Process Flow:
- Load Dependencies: Imports all required modules
- Content Extraction: Separates frontmatter, DataviewJS, and content
- Block Parsing: Processes all journal pages for mentions and todos
- Todo Rollover: Rolls over incomplete todos from previous days (today only)
- Activity Integration: Adds in-progress activities (today only)
- Mention Processing: Aggregates cross-references from other notes
- Script Cleanup: Removes DataviewJS blocks (today only)
- Content Assembly: Combines all processed content and saves
Key Logic:
// Only process special features for today's note
const pageIsToday = fileIO.isDailyNote(currentPageFile.name);
if (pageIsToday) {
// Todo rollover, activities, script cleanup
}
// Always process mentions for all daily notesPurpose: Manages activity note processing and lifecycle.
Process Flow:
- Frontmatter Processing: Handles activity metadata (startDate, stage, responsible)
- Content Extraction: Separates structure from content
- Block Parsing: Processes journal pages for activity mentions
- Mention Aggregation: Collects references from daily notes
- Content Assembly: Combines processed content and saves
Purpose: Parses note content into structured blocks for processing.
Block Types:
- Header:
# ## ### #### #####- Multi-line blocks ending on next header,---, or 2+ empty lines - Callout:
> text- Single-line blocks - Code: ``` blocks - Multi-line code blocks
- Todo:
- [ ]- Individual todo items (created separately even within headers) - Done:
- [x]- Completed todo items - Mention:
[[Link]]- Single-line mention blocks
Critical Parsing Rules:
// Header blocks end when:
// 1. Next header of same or higher level
// 2. Horizontal separator (--- or ----)
// 3. Two consecutive empty lines
// Todos are ALWAYS created as separate blocks
// even when they appear within header blocksPurpose: Manages todo rollover from previous days to today.
Activity Association Logic:
- Direct Link Check: Todo contains
[[Activities/...]],[[MyObsidian]], or any[[link]] - Block Context Check: Todo appears within activity header block content
- Standalone Detection: Todo is not associated with any activity
Rollover Rules:
- Only runs for today's daily note (
pageIsToday = true) - Only processes todos from dates before today
- Excludes activity-related todos
- Removes duplicates already present in current note
Purpose: Aggregates content from notes that mention the current note.
Processing Logic:
- Mention Detection: Finds blocks containing current note's name
- Content Filtering: Excludes blocks from current note itself
- Directive Processing: Converts
{directive}to(directive from filename)for regular text - Code Block Protection: Preserves JavaScript syntax in DataviewJS blocks
- Deduplication: Avoids adding duplicate content
Directive System:
// Supported operations:
{attribute = value} // Set attribute
{attribute += value} // Add to attribute
{attribute -= value} // Subtract from attribute
{attribute : value} // Set string attribute
// Date arithmetic:
{startDate += 1d} // Add 1 day
{startDate += 2w} // Add 2 weeks
{startDate += 1m} // Add 1 monthPurpose: Adds activity sections to today's daily note.
Selection Criteria:
- Activities with
stage: active - Activities with
stage: planningand recent activity (mentions in last 7 days) - Excludes
MyObsidianactivity (handled separately)
Purpose: Centralized file operations and content generation.
Key Functions:
loadFile()/saveFile(): File I/O operationsgenerateDailyNoteHeader(): Creates daily note frontmattergenerateActivityHeader(): Creates activity frontmatterextractFrontmatterAndDataviewJs(): Content structure parsingisDailyNote(): Determines if current note is today's date
Purpose: Template for daily journal notes with date-based naming.
Structure:
---
---
### DD [[YYYY-MM|Month]] [[YYYY]]
#### Week: [[YYYY-WNN|NN]]
----
### Activities:
----
[DataviewJS block for processing]Processing: Uses dailyNoteComposer for complete pipeline processing.
Purpose: Template for project/activity management.
Structure:
---
startDate: YYYY-MM-DD
stage: active|planning|completed|archived
responsible: [Me]
---
[DataviewJS block for processing]Processing: Uses activityComposer for activity-specific processing.
1. Template Instantiation
↓
2. dailyNoteComposer.processDailyNote()
↓
3. Content Structure Extraction
├─ Frontmatter
├─ DataviewJS Block
└─ Page Content
↓
4. Block Parsing (noteBlocksParser)
├─ Parse all journal pages
├─ Extract todos, headers, mentions
└─ Create structured block array
↓
5. Todo Processing (if today)
├─ Filter todos from previous days
├─ Exclude activity-related todos
├─ Remove duplicates
└─ Add to page content
↓
6. Activity Integration (if today)
├─ Find active activities
├─ Generate activity sections
└─ Add to page content
↓
7. Mention Processing
├─ Find mentions of current note
├─ Process directives
├─ Aggregate content
└─ Add to page content
↓
8. Script Cleanup (if today)
├─ Remove DataviewJS blocks
└─ Clean up processing artifacts
↓
9. Content Assembly & Save
├─ Combine frontmatter + content
└─ Save to file
1. Template Instantiation
↓
2. activityComposer.processActivity()
↓
3. Frontmatter Processing
├─ Extract startDate, stage, responsible
├─ Apply defaults if missing
└─ Generate activity header
↓
4. Content Structure Extraction
├─ DataviewJS Block
└─ Page Content
↓
5. Block Parsing (noteBlocksParser)
├─ Parse all journal pages
├─ Filter out current activity
└─ Create structured block array
↓
6. Mention Processing
├─ Find mentions of current activity
├─ Process directives
├─ Aggregate content
└─ Add to page content
↓
7. Content Assembly & Save
├─ Combine frontmatter + dataviewjs + content
└─ Save to file
{
page: "Journal/2025/07.July/2025-07-05.md",
blockType: "todo|done|header|mention|callout|code",
data: "Block content as string",
mtime: 1625097600000,
headerLevel: 5 // Only for header blocks
}// Activity-related todos (NOT rolled over):
1. Todo contains [[Activities/...]] or [[MyObsidian]] or any [[link]]
2. Todo appears within activity header block content
// Standalone todos (WILL be rolled over):
1. Todo has no [[link]] references
2. Todo appears outside activity header blocks// Regular text: {directive} → (directive from filename)
// Code blocks: {directive} → preserved as-is
// Prevents JavaScript syntax corruption- Pattern:
^#+\s+.*$ - Levels: 1-6 (
#to######) - Termination: Next header (same/higher level),
---separator, or 2+ empty lines - Content: Multi-line, includes everything until termination
- Todo Pattern:
^- \[ \]or^> - \[ \] - Done Pattern:
^- \[x\]or^> - \[x\] - Behavior: Always created as separate blocks, even within headers
- Callout Support: Handles
>prefixed todos in callouts
- Pattern:
^#{5}\s+\[\[Activities/.*\]\] - Purpose: Level 5 headers linking to activity notes
- Block Boundary: Defines activity context for todo association
- Source: Incomplete todos from previous daily notes
- Target: Today's daily note only
- Filtering: Excludes activity-related todos
- Deduplication: Avoids adding existing todos
- Placement: After last
---separator or at end
// Activity todos (stay in place):
- [ ] Buy something [[Activities/Shopping]] // Has link
- [ ] Task under activity header // In activity block
// Standalone todos (roll over):
- [ ] Personal task // No links, outside blocks
- [ ] Call dentist // No activity context[[Review/Daily]]: Daily recurrence[[Review/Weekly]]: Weekly recurrence[[Review/Monthly]]: Monthly recurrence
- Planning:
stage: planning- Ideas and preparation - Active:
stage: active- Currently working on - Completed:
stage: completed- Finished successfully - Archived:
stage: archived- Inactive/cancelled
- Active Activities: Always appear in today's daily note
- Planning Activities: Appear if mentioned in last 7 days
- Special Case:
MyObsidianactivity handled separately
---
startDate: 2025-07-06 # Project start date
stage: active # Current lifecycle stage
responsible: [Me, Partner] # Who's responsible
---- Pattern:
[[NoteName]]or[[NoteName|Alias]] - Scope: Searches all journal pages
- Exclusion: Ignores mentions from current note itself
- Collection: Gathers blocks mentioning current note
- Processing: Applies directive transformations
- Organization: Groups by source file
- Insertion: Adds after last
---separator
Enables dynamic frontmatter updates from mentions:
// In daily note mentioning an activity:
{startDate += 1d} // Extends activity start date
{stage = completed} // Changes activity stage
{responsible += Partner} // Adds to responsible listVault/
├── Engine/
│ ├── Scripts/
│ │ ├── dailyNoteComposer.js # Daily note orchestrator
│ │ ├── activityComposer.js # Activity orchestrator
│ │ ├── components/
│ │ │ ├── noteBlocksParser.js # Content parser
│ │ │ ├── todoRollover.js # Todo management
│ │ │ ├── mentionsProcessor.js # Cross-reference handler
│ │ │ └── activitiesInProgress.js # Activity integration
│ │ └── utilities/
│ │ └── fileIO.js # File operations
│ └── Templates/
│ ├── DailyNote-template.md # Daily note template
│ └── Activity-template.md # Activity template
├── Journal/
│ └── YYYY/
│ └── MM.Month/
│ └── YYYY-MM-DD.md # Daily notes
├── Activities/
│ ├── ProjectName.md # Active projects
│ ├── Archive/ # Completed activities
│ └── Workflow/ # Process activities
└── People/
└── PersonName.md # Contact notes
Ensure all scripts are registered in CustomJS settings:
// Required modules:
- dailyNoteComposer
- activityComposer
- noteBlocksParser
- todoRollover
- mentionsProcessor
- activitiesInProgress
- fileIOTemplates should be placed in Engine/Templates/ and configured in Templater plugin settings.
Cause: MentionsProcessor converting {directive} to (directive from file) in JavaScript code blocks.
Solution: Code block protection is implemented - ensure isCodeBlock parameter is properly passed.
Cause: Todos incorrectly classified as activity-related.
Solution: Check todo content for [[links]] and verify activity header block boundaries.
Cause: Mention deduplication not working properly. Solution: Verify normalized line comparison logic in mentionsProcessor.
Cause: Activity stage not set to active or no recent mentions for planning stage.
Solution: Update activity frontmatter or add mentions in recent daily notes.
All components include console.log statements for debugging:
console.log("TodoRollover: Starting rollover process");
console.log("MentionsProcessor: Processing mentions for", tagId);- Use ES6+ features (async/await, arrow functions, destructuring)
- Implement comprehensive error handling
- Add descriptive console logging for debugging
- Follow consistent naming conventions
- Test with various note structures
- Verify edge cases (empty notes, malformed content)
- Test date boundary conditions
- Validate cross-reference integrity
- Minimize file I/O operations
- Cache parsed content when possible
- Use efficient filtering and searching
- Avoid unnecessary re-processing
- New Block Types: Extend noteBlocksParser with additional patterns
- Custom Directives: Add new directive operations in mentionsProcessor
- Activity Stages: Extend lifecycle stages in activityComposer
- Template Types: Create new templates with corresponding composers
try {
// Processing logic
} catch (error) {
console.error("ComponentName error:", error);
return {
success: false,
error: error.message
};
}class ComponentName {
// Private helper methods
// Public processing methods
// Main entry point
async run(app, ...params) {
return await this.processMethod(app, ...params);
}
}- Obsidian with the following plugins installed:
- DataviewJS
- CustomJS
Templater(No longer required! 🎉)
- Node.js knowledge for JavaScript development
- Basic understanding of Obsidian's file structure and linking system
- Clone/Copy Scripts: Place all scripts in
Engine/Scripts/directory - Configure CustomJS: Register all modules in CustomJS plugin settings
- Setup Templates: Place templates in
Engine/Templates/and configure Templater - Create Folder Structure: Ensure
Journal/,Activities/,People/folders exist - Test Basic Functionality: Create a daily note and verify processing works
1. Use DailyNote-template.md to create: Journal/2025/07.July/2025-07-06.md
2. The DataviewJS block will automatically process the note
3. Check console for any errors
4. Verify frontmatter and content are generated correctly1. Use Activity-template.md to create: Activities/TestProject.md
2. Set frontmatter: startDate, stage: active, responsible: [Me]
3. Add some content and todos
4. Create a daily note that mentions this activity
5. Verify cross-references work correctlyclass dailyNoteComposer {
async processDailyNote(app, dv, currentPageFile, title)
// Returns: { success: boolean, frontmatter: string, content: string, isToday: boolean }
}class activityComposer {
async processActivity(app, dv, currentPageFile)
// Returns: { success: boolean, frontmatter: string, content: string }
}class noteBlocksParser {
parse(page, content)
// Returns: Array<Block>
async run(app, pages, namePattern = "")
// Returns: Array<Block>
}
// Block Structure:
interface Block {
page: string;
blockType: 'todo'|'done'|'header'|'mention'|'callout'|'code';
data: string;
mtime: number;
headerLevel?: number;
}class todoRollover {
async rolloverTodos(app, blocks, todayDate, currentPageContent, removeOriginals)
// Returns: string (updated content)
isActivityRelatedTodo(todoBlock, allBlocks)
// Returns: boolean
async run(app, collectedBlocks, dailyNoteDate, currentPageContent, RemoveOriginals = false)
// Returns: string (updated content)
}class mentionsProcessor {
async processMentions(currentPageContent, blocks, tagId, frontmatterObj)
// Returns: string (updated content with mentions)
async run(currentPageContent, collectedBlocks, mentionStr, frontmatterObj)
// Returns: string (updated content)
}class fileIO {
static async loadFile(app, filename)
// Returns: string (file content)
static async saveFile(app, filename, content)
// Returns: void
static generateDailyNoteHeader(date)
// Returns: string (frontmatter)
static generateActivityHeader(startDate, stage, responsible)
// Returns: string (frontmatter)
static extractFrontmatterAndDataviewJs(content)
// Returns: { frontmatter: string, dataviewJsBlock: string, pageContent: string }
static isDailyNote(filename)
// Returns: boolean
static todayDate()
// Returns: string (YYYY-MM-DD format)
}// Supported directive patterns in mentions:
{attribute = value} // Set attribute to value
{attribute += value} // Add value to attribute
{attribute -= value} // Subtract value from attribute
{attribute : value} // Set string attribute
// Date arithmetic examples:
{startDate += 1d} // Add 1 day
{startDate += 2w} // Add 2 weeks
{startDate += 1m} // Add 1 month
{startDate += 1y} // Add 1 year
{startDate -= 3d} // Subtract 3 daysThe system includes a complete test suite located in Engine/TestSuite/ with comprehensive testing for all components.
📋 Test Categories:
- Core Component Tests: noteBlocksParser, fileIO, mentionsProcessor, attributesProcessor
- Feature Tests: todoRollover, activitiesInProgress, activityComposer, dailyNoteComposer
- Integration Tests: Full workflow testing, cross-component interactions
- Sample Data: Ready-to-use test files and scenarios
🚀 Quick Testing:
- Open any test file in
Engine/TestSuite/ - Execute the DataviewJS block to run tests
- Check console output (F12 → Console) for results
- Look for ✅ success or ❌ error indicators
📊 Test Runner:
- Use
Engine/TestSuite/TestRunner.mdto run all tests at once - Comprehensive test results with performance metrics
- Automated regression testing capabilities
📚 Full Documentation: See Engine/TestSuite/README.md for complete testing guide.
// Example test structure for todoRollover
async function testTodoRollover() {
console.log("=== TODO ROLLOVER TEST START ===");
const {todoRollover} = await cJS();
const testTodo = {
data: '- [ ] Call dentist',
page: 'Journal/2025/07.July/2025-07-05.md'
};
const isActivityRelated = todoRollover.isActivityRelatedTodo(testTodo, []);
console.log("✅ Standalone todo detection:", !isActivityRelated);
console.log("=== TODO ROLLOVER TEST END ===");
}// Test complete daily note processing
async function testDailyNoteProcessing() {
console.log("=== DAILY NOTE PROCESSING TEST START ===");
const {dailyNoteComposer} = await cJS();
const currentFile = dv.current().file;
try {
await dailyNoteComposer.processDailyNote(app, dv, currentFile, currentFile.name);
console.log("✅ Daily note processing completed successfully");
} catch (error) {
console.log("❌ Daily note processing failed:", error);
}
console.log("=== DAILY NOTE PROCESSING TEST END ===");
}Engine/TestSuite/
├── Core/ # Core component tests
├── Features/ # Feature-specific tests
├── Integration/ # End-to-end workflow tests
├── Samples/ # Sample data for testing
└── Utils/ # Testing utilities
// Cache parsed blocks to avoid re-parsing
class noteBlocksParser {
constructor() {
this.blockCache = new Map();
}
async run(app, pages, namePattern = "") {
const cacheKey = pages.map(p => p.file.path + p.file.stat.mtime).join('|');
if (this.blockCache.has(cacheKey)) {
return this.blockCache.get(cacheKey);
}
const blocks = await this.parsePages(app, pages, namePattern);
this.blockCache.set(cacheKey, blocks);
return blocks;
}
}// Process multiple files in batches to avoid memory issues
async function processBatch(files, batchSize = 10) {
for (let i = 0; i < files.length; i += batchSize) {
const batch = files.slice(i, i + batchSize);
await Promise.all(batch.map(file => processFile(file)));
// Allow other operations to run
await new Promise(resolve => setTimeout(resolve, 0));
}
}// Clear caches periodically
setInterval(() => {
if (blockCache.size > 1000) {
blockCache.clear();
console.log('Cleared block cache to prevent memory issues');
}
}, 300000); // Every 5 minutes// Validate file paths to prevent directory traversal
function validateFilePath(path) {
if (path.includes('..') || path.startsWith('/')) {
throw new Error('Invalid file path: ' + path);
}
return path;
}
// Sanitize directive values
function sanitizeDirectiveValue(value) {
// Remove potentially dangerous characters
return value.replace(/[<>\"'&]/g, '');
}// Prevent script injection in processed content
function sanitizeContent(content) {
// Remove script tags and dangerous HTML
return content
.replace(/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, '')
.replace(/javascript:/gi, '')
.replace(/on\w+\s*=/gi, '');
}// Restrict file operations to vault directory
function isValidVaultPath(app, path) {
const vaultPath = app.vault.adapter.basePath;
const fullPath = require('path').resolve(vaultPath, path);
return fullPath.startsWith(vaultPath);
}// Extend activityComposer to support custom stages
class extendedActivityComposer extends activityComposer {
getValidStages() {
return ['planning', 'active', 'blocked', 'review', 'completed', 'archived'];
}
shouldIncludeInDaily(activity, mentions) {
const stage = activity.stage;
if (stage === 'blocked') {
// Include blocked activities if mentioned in last 3 days
return mentions.some(m => moment().diff(moment(m.date), 'days') <= 3);
}
return super.shouldIncludeInDaily(activity, mentions);
}
}// Add support for habit tracking blocks
class extendedNoteBlocksParser extends noteBlocksParser {
isHabitLine(line) {
return line.match(/^- \[habit\]/);
}
parse(page, content) {
const blocks = super.parse(page, content);
// Add habit blocks
const lines = content.split('\n');
lines.forEach((line, index) => {
if (this.isHabitLine(line)) {
blocks.push({
page: page,
blockType: 'habit',
data: line,
mtime: Date.now(),
lineNumber: index
});
}
});
return blocks;
}
}// Add support for priority directives
function processCustomDirective(directive, frontmatterObj) {
if (directive.includes('priority')) {
const match = directive.match(/priority\s*([=:+\-])\s*(\w+)/);
if (match) {
const operation = match[1];
const value = match[2];
switch (operation) {
case '=':
case ':':
frontmatterObj.priority = value;
break;
case '+':
frontmatterObj.priority = increasePriority(frontmatterObj.priority || 'low');
break;
case '-':
frontmatterObj.priority = decreasePriority(frontmatterObj.priority || 'high');
break;
}
}
}
}
function increasePriority(current) {
const levels = ['low', 'medium', 'high', 'urgent'];
const index = levels.indexOf(current);
return levels[Math.min(index + 1, levels.length - 1)];
}// Old way (deprecated):
const {todoRollover} = await cJS();
await todoRollover.run(app, blocks, date, content, true);
// New way:
const {todoRollover} = await cJS();
const result = await todoRollover.run(app, blocks, date, content, true);
if (result.success) {
console.log('Rollover completed successfully');
}- Return Values: All processors now return structured objects instead of strings
- Error Handling: Errors are now returned in result objects instead of thrown
- Async Operations: All file operations are now properly async
// Run this script to migrate old notes to new format
async function migrateNotes(app) {
const files = app.vault.getMarkdownFiles();
for (const file of files) {
const content = await app.vault.read(file);
// Update old directive format
const updatedContent = content
.replace(/\{(\w+)\s*=\s*([^}]+)\}/g, '{$1 = $2}')
.replace(/\{(\w+)\s*\+=\s*([^}]+)\}/g, '{$1 += $2}');
if (content !== updatedContent) {
await app.vault.modify(file, updatedContent);
console.log('Migrated:', file.path);
}
}
}A: Check if the todos contain [[links]] or are under activity headers. Only standalone todos without links roll over.
A: Extend the processDirective function in mentionsProcessor.js to handle your new operation pattern.
A: Yes, modify DailyNote-template.md and update generateDailyNoteHeader() in fileIO.js accordingly.
A: Enable browser console and look for component-specific log messages. Each component logs its processing steps.
A: Yes, but you'll need to update the path patterns in noteBlocksParser.js and file structure assumptions in composers.
A: Use the provided journal_backup.py script or create manual backups of your vault before testing changes.
A: Each component has error handling that logs errors and returns failure status without corrupting existing content.
A: Not recommended. The system assumes single-user operation and may have conflicts with concurrent processing.
A: Consider adding user identification to frontmatter and filtering content by user in the processing pipeline.
A: Processing time scales with the number of journal pages. Consider implementing caching and batch processing for vaults with 1000+ notes.
This comprehensive manual provides developers with all the information needed to understand, maintain, extend, and troubleshoot the Personal Knowledge Management System effectively.