Add ability to change lunr tokenizer #28

Activity

1. cigolpl commented on Sep 6, 2018

   

   cigolpl

   on Sep 6, 2018

   Last edited by cigolpl

   Member

   More actions

   @saminzadeh great idea! Unfortunately ItemsJS is using Lunr v1.0.0 and now Lunr is v2.3.3. Hope also Lunr is exposing public function or constructor for changing lunr.tokenizer.seperator. Ideally if simple change with v1.0.0 is possible otherwise ItemsJS should be upgraded to latest Lunr sooner or later. I could hopefully look into it when I find more free time
2. saminzadeh commented on Sep 7, 2018

   

   saminzadeh

   on Sep 7, 2018

   ContributorAuthor

   More actions

   Sounds good, might take a stab when I get a chance as well.

   For now, I did this and it seems to work since it changes the lunr global instance.

   import lunr from 'lunr';
   import itemsjs from 'itemsjs';
   
   lunr.tokenizer.separator = /[\s\-[\]:]+/g;
   
   // itemsjs init here
   const index = itemsjs(data, configuration);
3. cigolpl commented on Sep 7, 2018

   

   cigolpl

   on Sep 7, 2018

   Last edited by cigolpl

   Member

   More actions

   Nice hack! Integration should be easier than I thought

   I am wondering how to test easily separators:
   I've test it out with:

   var paragraph = 'The quick brown fox jumped over the lazy dog. It barked.';
   var regex = /[\s\-[\]:]+/g;
   var found = paragraph.split(regex);
   
   console.log(found);
   // ["The", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog.", "It", "barked."]

   Testing with Lunr function lunr.tokenizer seems to be more complicated for testing results

   I am wondering because if we implement this feature for devs then it could be as separator / regex option or as one of list of predefined analyzers (i.e. like in https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analyzer.html). Second seems nice but first (your suggestion) the simplest to start with and very flexible
4. saminzadeh commented on Sep 7, 2018

   

   saminzadeh

   on Sep 7, 2018

   ContributorAuthor

   More actions

   I am wondering because if we implement this feature for devs then it could be as separator / regex option or as one of list of predefined analyzers (i.e. like in https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analyzer.html).

   Hmm, yes that could be nice. But I agree, would probably want to iron that out a bit more before it is introduced into the API interface.

   Second seems nice but first (your suggestion) the simplest to start with and very flexible

   Yes I think this could be the best starting point. Just having access to the lunr object via itemsjs would be the most flexible for advanced users and will ensure the correct dependency

   Something like this:

   import lunr from 'itemjs/lunr';
   
   lunr.tokenizer.separator = /[\s\-[\]:]+/g;

   or

   import ItemsJS from 'itemjs';
   const index = ItemsJS(data, config);
   index.lunr.tokenizer.separator = /[\s\-[\]:]+/g;
   
   index.search({...})

   👍React with 👍1Reacted by Mateusz
5. cigolpl commented on Sep 7, 2018

   

   cigolpl

   on Sep 7, 2018

   Member

   More actions

   Makes sense!
6. cigolpl commented on Apr 9, 2021

   

   cigolpl

   on Apr 9, 2021

   Member

   More actions

   @saminzadeh I've introduced simple full text integration with all external search engines in the latest version. You can see here -> https://github.com/itemsapi/itemsjs/blob/master/docs/lunr2-integration.md or in Readme
7. cigolpl

   closed this as completedon Apr 9, 2021