Friday, August 5, 2011

Creating refinement query parameter for FS4SP by code

[Update: 2011-12-09]
Someone from somewhere took the time to use Reflector and pull out the appropriate code from the FS4SP dlls. Download the code below.

When using the Search Center with FAST Search for SharePoint you will see the query parameters being modified when you interact with the page.

The k= parameter holds your search query. This is useful if you create a link somewhere on your site which redirects you to the search page with a pre-set query. An example is creating a link which displays all the documents from a particular employee containing the word contract.

This can be accomplished by the query:


contract author:”Mikael Svenson”

which in the k= parameter looks like:


k=contract%20author%3A%22Mikael%20Svenson%22

And this works just fine. What doesn’t work is if you have specified a synonym for the word contract. When tacking on the author parameter to the query, it will no longer match in the synonym list, as the synonyms match the whole query term, not just a single word.

Another approach would be to add the author filter as a refiner instead in the r= query parameter, leaving only the word in the k= parameter which will match the synonym definition.

However, if you look at the r= parameter for a query refining on me as an author it looks like this:


r=author%3D%22AQ5NaWthZWwgU3ZlbnNvbgZhdXRob3IBAl4iAiIk%22

The quick ones out there spot this as being a base64 encoded string. If you decode the string and with some investigation you can find out how it’s being built up.

The structure looks like this:
(char)1
(char)length of property value
property value
(char)length of property name
property name
(char)1
(char)2 //length of the next two bytes
^”
(char)2 //length of the next two bytes
“$


For your convenience I have created a small class which does the heavy lifting for you.

The usage is like this:

BuildRefinerParameter builder = new BuildRefinerParameter();
builder.AddRefiner("format", "Adobe PDF");
builder.AddRefiner("companies", "Microsoft");
string url = builder.GetRefinerQueryParameter();

And the class itself:

class BuildRefinerParameter
{
    Dictionary<string,string> _refiners = new Dictionary<string, string>();
    public void AddRefiner(string key, string value)
    {
        _refiners[key] = value;
    }
    public string GetRefinerQueryParameter()
    {
        StringBuilder sb = new StringBuilder();
        foreach (var refiner in _refiners)
        {
            sb.Append(refiner.Key);
            sb.Append('=');
            sb.Append('"');
            string base64encodedRefiner = CodeRefinerParameter(refiner);
            sb.Append(Convert.ToBase64String(Encoding.UTF8.GetBytes(base64encodedRefiner)));
            sb.Append('"');
            sb.Append(' ');
        }
        string url = HttpUtility.UrlEncode(sb.ToString().Trim());
        return url;
    }
    private static string CodeRefinerParameter(KeyValuePair<string, string> refiner)
    {
        var refkey = refiner.Key;
        var refval = refiner.Value;
        StringBuilder coded = new StringBuilder();
        coded.Append((char) 1);
        coded.Append((char) refval.Length);
        coded.Append(refval);
        coded.Append((char) refkey.Length);
        coded.Append(refkey);
        coded.Append((char) 1);
        coded.Append((char) 2);
        coded.Append('^');
        coded.Append('"');
        coded.Append((char) 2);
        coded.Append('"');
        coded.Append('$');
        return coded.ToString();
    }
}
}

26 comments:

  1. While the r parameter isn't tamper proof, it's definitively not tamper-friendly. Good write-up!

    ReplyDelete
  2. great post Mikael. it saved me a lot of time.
    quick question, the paramaters generated by your code is slightly different then the fast search creates itself when we click on the refinement item.

    looks like what I generate is the same as the original value which I converted from base64 value. do you know what is causing this? thanks

    ReplyDelete
  3. I have taken a shortcut to include all refiners as phrases with "" around them. One word refiner values are not enclosed with "" in the refiner webpart. Could this be the difference you are seeing?

    ReplyDelete
  4. I had to modify the code as following to bring up correct results for parameters with one keyword as:
    builder.AddRefiner("suite", "Mobile");

    private static string CodeRefinerParameter(KeyValuePair refiner)
    {
    var refkey = refiner.Key;
    var refval = refiner.Value;
    StringBuilder coded = new StringBuilder();
    coded.Append((char)1);
    coded.Append((char)refval.Length);
    coded.Append(refval);
    coded.Append((char)refkey.Length);
    coded.Append(refkey);
    coded.Append((char)1);
    coded.Append((char)1);
    coded.Append('^');
    coded.Append((char)1);
    coded.Append('$');
    return coded.ToString();
    }


    when we have multiple refiner value as follows I need to put 3 spaces at the beginning. (char)1 is not helping. do you know how which character adds 3 characters?

    builder.AddRefiner("suite", "Enterprise Planning");

    appreciated your help

    ReplyDelete
  5. sema,
    Which AddRefiner lines are not working for you? And I'll test and improve my code. It's all about reverse engineering :)

    ReplyDelete
  6. ok this is how I do it.

    I first run the fast search and get the unique id for my value which is "ARQBTW9iaWxlIEZpZWxkIFNlcnZpY2UGbW9kdWxlAQJeIgIiJA"

    module%3D%22ARQBTW9iaWxlIEZpZWxkIFNlcnZpY2UGbW9kdWxlAQJeIgIiJA%3D%3D%22

    if you encode this, you get:

    " Mobile Field Service module ^" "$"

    than I run your code and get different results

    builder.AddRefiner("module", "Mobile Field Service");

    your code returns:

    module%3d%22AQEUTW9iaWxlIEZpZWxkIFNlcnZpY2UGbW9kdWxlAQJeIgIiJA%3d%3d%22

    when I decode the unique key
    AQEUTW9iaWxlIEZpZWxkIFNlcnZpY2UGbW9kdWxlAQJeIgIiJA

    I get:

    " Mobile Field Service module ^\" \"$"

    so I get different results for different refiners. thanks Mikael

    ReplyDelete
  7. Hello Mikael, I got the same problem as SEMA. Your code is working for few refiners like
    builer.AddRefiner("companies", "Microsoft")
    but it is not giving the same url as FAST search for many refiners such as
    builder.AddRefiner("companies","Sun Microsystems"..

    I am currently working on a Project which requires the url..So need quick help from you on this issue

    ReplyDelete
  8. Sanidip,
    ok, so the issue seems to be with the quotes around multiple word values. When using my code, are you getting fewer or more results compared to the ones from the UI?

    ReplyDelete
  9. Thanks Michael for the quick reply.
    If the FAST search URL and the URL generated by your code are same , I get exactly same number of results but when they are different I get "The search request was unable to execute on FAST Search Server."

    I think I was unable to explain my problem in my last comment. So this time I am explaining it elaborately. I have listed following examples(both FAST URL(URL generated by FAST search engine) and CODE URL(URL generated by your code)) in the NOT WORKING Category and WORKING category..


    NOT WORKING : builder.AddRefiner("companies", "Sun Microsystems");
    -----------------------------------------------------------------
    FAST URL: companies%3D"ARABU3VuIE1pY3Jvc3lzdGVtcwljb21wYW5pZXMBAl4iAiIk"

    CODE URL: companies%3d%22ARBTdW4gTWljcm9zeXN0ZW1zCWNvbXBhbmllcwEBXgEk%22

    NOT WORKING :builder.AddRefiner("companies", "Johnson & Johnson");
    -----------------------------------------------------------------
    FAST URL: companies%3D"AREBSm9obnNvbiAmIEpvaG5zb24JY29tcGFuaWVzAQJeIgIiJA%3D%3D"

    CODE URL: companies%3d%22ARFKb2huc29uICYgSm9obnNvbgljb21wYW5pZXMBAV4BJA%3d%3d%22

    WORKING : builder.AddRefiner("format", "Adobe PDF");
    -------------
    FAST URL: format%3D"AQlBZG9iZSBQREYGZm9ybWF0AQJeIgIiJA%3D%3D"

    CODE URL: format%3d%22AQlBZG9iZSBQREYGZm9ybWF0AQFeASQ%3d%22

    WORKING : builder.AddRefiner("companies", "Microsoft");
    ---------
    FAST URL: companies%3D"AQlNaWNyb3NvZnQJY29tcGFuaWVzAQFeASQ%3D"

    CODE URL: companies%3d%22AQlNaWNyb3NvZnQJY29tcGFuaWVzAQFeASQ%3d%22


    There is one more problem.When I search on the two refiners HAVING SAME MAPPED PROPERTY(in the follwing example , "companies"), I get only URL for the first refiner. I think this is caused because the dictionary can have unique key values ..So this thing also needs to be taken care of in the code..


    NOT WORKING: builder.AddRefiner("companies", "Microsoft");
    builder.AddRefiner("companies", "Google");
    ---------------------------------------------------------
    FAST URL: companies%3D"AQZHb29nbGUJY29tcGFuaWVzAQFeASQ%3D" AND companies%3D"AQlNaWNyb3NvZnQJY29tcGFuaWVzAQFeASQ%3D"

    CODE URL: companies%3d%22AQZHb29nbGUJY29tcGFuaWVzAQFeASQ%3d%22


    Anticipating your quick reply..

    ReplyDelete
    Replies
    1. I am also getting the same Error....

      Any sugesstion/helps are most welcome in order to resolve the issue......

      With Regards,
      Bunty

      Delete
  10. I use this code:
    class FS4SPUtil
    {
    public static string CreateRefinerToken(string name, string value)
    {
    var refkey = name;
    var refval = value;
    StringBuilder coded = new StringBuilder();

    if (refval.Contains(" "))
    {
    var refval_len = Encoding.UTF8.GetBytes(refval).Length;
    coded.Append((char)1);
    coded.Append((char)refval.Length);
    if (refval_len != refval.Length)
    coded.Append((char)2);
    else
    coded.Append((char)1);
    coded.Append(refval);
    coded.Append((char)refkey.Length);
    coded.Append(refkey);
    coded.Append((char)1);
    coded.Append((char)2);
    coded.Append('^');
    coded.Append('"');
    coded.Append((char)2);
    coded.Append('"');
    coded.Append('$');
    }
    else
    {
    var refval_len = Encoding.UTF8.GetBytes(refval).Length;
    coded.Append((char)1);
    coded.Append((char)Encoding.UTF8.GetBytes(refval).Length);
    if (refval_len != refval.Length)
    coded.Append((char)1);
    coded.Append(refval);
    coded.Append((char)refkey.Length);
    coded.Append(refkey);
    coded.Append((char)1);
    coded.Append((char)1);
    coded.Append('^');
    coded.Append((char)1);
    coded.Append('$');
    }

    return Convert.ToBase64String(Encoding.UTF8.GetBytes(coded.ToString()));
    }
    }

    Please email me at iamake AT gmail if it does not work.

    ReplyDelete
  11. iamake, thanks for posting your code :) Seems my shortcut of using quotes around single words fails, and you need to differentiate on that.

    Either way, glad to get you guys started on the correct path :)

    ReplyDelete
  12. Thanks for your help ..But the problem is that it is working for some refiners (which were not working in Mikael's code ) but not for many.

    for example
    It is not working for refiners for mapped property "format" .E.g.-("format","Adobe PDF") (It was working in Mikael's code)
    ("format", "Web Page");
    and mapped property "sitename" E.g -("sitename", "01hw56094:1111"); and many others

    It is working fine for the mapped property "companies" for e.g.- ("companies", "Sun Microsystems")

    I would be grateful to you if you rectify the code so that it gives the same encoded url as FAST search .

    ReplyDelete
  13. I unfortunately don't have time to look at this this week, but will do so when I have the time unless someone else reverse engineers this 100% first :)

    Shouldn't be to hard to get it right if spent an hour or two.

    ReplyDelete
  14. Thanks Mikael..Please make it 100% perfect next week as many requirements of my project will be solved once I get 100% perfect solution of the above problem.. Till then I will cover other requirements of my project..will remind you next week.

    ReplyDelete
  15. Please look into the above problem Mikael as soon as you get time..really waiting for the perfect solution from you...

    ReplyDelete
  16. Sandip,
    I'm all locked up with other things this week. And as I don't need the code myself it's not a priority. You should be able to fix it yourself pretty easy when you examine how it's being built up.

    Remember, I do this on my free time, unpaid.

    ReplyDelete
  17. I got an e-mail yesterday and have updated this post with a link to code at the top.

    ReplyDelete
  18. Thanks a lot. ! This is great.

    ReplyDelete
  19. Hi,
    I have developed two small utilities which will help you decode and encode the refiner values. You can get it from here..
    http://kalashnikovtechnoblogs.blogspot.com/2012/03/encode-and-decode-refiner-value-from.html
    Thanks,
    Kalashnikov
    http://kalashnikovtechnoblogs.blogspot.com

    ReplyDelete
  20. attached cs file worked perfectly when try building token for managed metadata column type.

    ReplyDelete
  21. Hi Mikael,

    I tried to write customclass for multivalue filtergenetator but when I use Type="MultiValueFilterGenerator1, ..." for one section let say COLOR(manageproperty) in the refinment panel then section itself will get disappear apart from that section other section will be there and working fine.
    Below is my XML part for manageproperty COLOR



    Can you Please Help me out in the Same.

    Thanks & Regards,
    Bunty.

    ReplyDelete
    Replies
    1. Hi,
      Your best option for multi-value is to split the data during indexing as multi-value. Then you don't have to deal with refinement modules in the front end. Currently doing this at a project where we are replacing the code from http://www.chaholl.com/archive/2011/03/29/building-a-custom-refinement-filter-generator-for-sharepoint-2010.aspx with a FAST proper multi-value navigator instead using the pipeline extensibility.

      Delete
  22. Hi Mikael,

    I have a requirement where i need to create a custom webpart which builds its own query based on the keyword and refinement panel elements selected to display the Top rated Content. This web part will be on the same Search Results page.

    What i am doing is reading the query string to get keyword and refinement panel filters by using request.QueryString["r"]. I am not sure how to read those refinement filters from query string and pass it to my KeywordQuery class to get exactly the same results as Core Search Results webpart.

    Can you please help me with this. I know there is like Refiners and RefinementFilters propery on KeywordQuery Class which needs to be used but i m not sure how to supply these from query string.

    Any help is much appreciated.

    Thanks,
    Paddy

    ReplyDelete
    Replies
    1. Hi,
      I haven't had time to look into it fully but you can either inherit from the RefinementPanel webpart and access the set refiners from there. Also take a look at the RefinementManager. Or you can disassemble SP code and find the decoder for the r= parameter.. much the way used to create the encoder class in this post.

      Delete
  23. Thanks for your reply Mikael. But once we decode the r parameter. How can we make use of the refiners and refinementfilters property of KeywordQuery to get proper results. Do you have any sample code of passing refiners and filter values to KeywordQuery Class

    Thanks in advance

    Paddy

    ReplyDelete