Searching entries performance in Mori

Tags ( )

Hi,
To create a plugin that has Notationnal Velocity features, I'd like to reproduce the search behavior that is provided in Mori via the toolbar with the following code:


//on recupere le content index
NSDocumentController *documentController = [NSDocumentController sharedDocumentController];
id document = [documentController currentDocument];

if (!document && [[documentController documents] count] > 0) {
document = [[NSApp orderedDocuments] objectAtIndex:0];
if (!document) {
//*error = BKLocalizedString(@"Couldn't find open Mori document to insert selection into.", @"");
return;
}
}

id index = [document contentIndex];

//we grab some needle in a NSTextView
NSString *noteString = [[[NSString alloc] initWithFormat:@"*%@*",[noteTextView string]] autorelease];

NSPredicate *predicate = [NSPredicate predicateWithFormat:@"note.noteString like[cd] %@ OR title like[cd] %@", noteString,noteString];
NSManagedObjectContext *moc = [document managedObjectContext];
id mod = [document managedObjectModel];
NSLog(@"Query is %@",predicate);

NSFetchRequest *fetchRequest = [[[NSFetchRequest alloc] init] autorelease];
NSError *fetchError = nil;
NSArray *fetchResults = nil;

[fetchRequest setEntity:[NSEntityDescription entityForName:@"EntryData" inManagedObjectContext:moc]];
[fetchRequest setPredicate:predicate];

//This is very very slow compared to the search performed directly in Mori
fetchResults = [moc executeFetchRequest:fetchRequest error:&fetchError];

NSLog(@"Got %d results",[fetchResults count]);

NSLog(@"%@",fetchResults);

who is incredibly slow compared to the search made via the builtin search command.

Any ideas on how I may get somehow near the speed of the builtin search ? Maybe a builtin blocks method that I've missed ?

Thanks !

I'm glad that you are

I'm glad that you are digging into the code. Also it would help me if you could describe the particular Notational Velocity features that you are trying to add.

Mori uses two kinds of built in search. It uses coredata for smart folder searches, and it uses a separate SearchKit text index for text searches from the toolbar. Depending on the feature that you are trying to add you might just be able to use Mori's SearchKit index for searching instead of using CoreData. To do that use MIEntryLiveFetchRequestProtocol that's defined in MIDocumentProtocols in the MIDocument plugin. The basics of what you'll need to do to use that are:

1. liveFetch = [doc createLiveFetchRequestOfType:MISearchKitFetchRequest];
2. [liveFetch setQueryString:@"some text in SearchKit query syntax"];

And then from MIUserInterfaceProtocols.h

3. [windowController setLiveFetchRequest:liveFetch];

Those protocols aren't well documented, but if you have the Mori source code you can find a few more detailed examples of how they should be used. If you don't yet have access to the source code let me know and I'll give it to you if you are a registered user.

So that's how Mori does it's text searches. But I think with some fiddling you should be able to make coredata search a lot faster also. Unfortunatly I'm not quite sure what fiddling will be required. One thing to try is build a smart folder in Mori that does the same search and see what the performance of that is. (smart folders use NSPredicates and core data so I would expect the results to be similar). If they are not then you can either look at how Mori optmizes these searches (see MILiveCoredataFetchRequest) or you can just reuse Mori's core data search machenery by specifying MICoreDataFetchRequest when creating the live search request instead of MISearchKitFetchRequest.

...

And if Mori's smart folders perform this type of query to slowly also then you'll likely need to do more fiddling. I think that "contains" searches are supposed to be faster then "like" searches so you could try that. The problem with them is that I've had trouble querying accross relationships when using contains searches, that's why I don't use them. But if you queried directly on the Note entity then I think it would work... but then you would need a separate search to query on the title entity.

...well that's a lot to try. let me know what you find out, and come back with questions.

Great !Got it working

Great !

Got it working nicely this morning with a live request (code posted underneath for reference...), thanks for the suggestion.
Now queries terminates under 1s for my 200Mo notebook (without attachements).


/* declarations
id <MIEntryLiveFetchRequestProtocol> req;

*/
id <MIDocumentProtocol> document = [self getCurrentDocument];

//on recupere le contenu de la note chan window
NSString *noteString = [[[NSString alloc] initWithFormat:@"%@",[noteTextView string]] autorelease];

if(req!= nil){
[req stopQuery];
[req release];
req=nil;
}

//This method should be implemented by <MIContentIndexProtocol> contentIndex but is not, still, <MIDocumentProtocol>document implements it
req = [[document createLiveFetchRequestOfType:MISearchKitFetchRequest]retain];

//This provides content as well !
[req setSearchKitIndex:MISearchKitIndexTitle];

//But this will only search in titles
// [req setSearchKitIndex:MISearchKitIndexAll];

[req setQueryString:noteString];
[req startQuery];

int waiting=0; //needed to upper bound the search, as some queries will run forever with 0 results
while([req isGathering] && ![req isStopped] && waiting < 20){
[NSThread sleepUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.05]];
NSLog(@"Got %d results",[req resultCount]);
waiting++;
}
NSLog(@"Got %d results",[req resultCount]);

if(req!= nil){
[req stopQuery];
[req release];
req=nil;
}

It took me more time than needed, as

  • The document method "(id <MIContentIndexProtocol>)contentIndex" returns an object that does not conforms to the advertised protocol, in fact only the mori document seems to have the MIContentIndexProtocol methods implemented
  • When setting the search type to MISearchKitIndexAll, only the titles are searched and conversely, when setting the search type to MISearchKitIndexTitle, titles and content are searched
  • The first query never returns anything, do you build the index lazily ?
  • Some searches seems to run in a endless loop, always returning 0 items. Have you run into this issue ? Does it have anything to do with the index being rebuild at the same time ?

Anyway, thanks for you post !

I'm currently reworking the notechan / mini input window from BMEguy, adding the following features:

  • System wide hot key to bring the window to front
  • When activated in an external app, the plugin saves an alias (or an url) to the currently active document and inserts it in the saved note
  • Display of notes similar to the one being typed

and I was having a lot of trouble to get the 3rd to work properly.

As you've pointed, the problem with the previous coreData search was that I performed a search on the title and the content, and the join between EntryData and Note was eating everything.

Two last questions:

  • How can I set the kSKSearchOptionFindSimilar to perform a similarity search ?
  • How can I restrict the search to a particular folder ?

My Mac is still being

My Mac is still being repaired so I don't have easy access to Mori's code right now to get examples from. But these two:


# How can I set the kSKSearchOptionFindSimilar to perform a similarity search ?
# How can I restrict the search to a particular folder ?

Options should be possible and you can find examples in Mori's sources. Mori already has a "Find Similar" commanded that was added recently. And you can restrict the search to particular folders using the set excluded folders and set included folders methods on a live fetch request object.

I'm going to send you an email with info on getting access to the source code.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.