ColdFusion Levenshtein Distance: String comparison and highlighting
This is a fun project I put out there a while back. I recently went through and optimized the performance a bit so I could officially blog it. It is an implementation of the Levenshtein Distance Algorithm in CFScript that I based off of a C# version written by Siderite Zackwehdex. Finding the "distance" between two strings is a means of comparing two strings to see how similar they both are. This can be done by finding the Longest Common String or LCS. It is as much a brain bender as it can be occasionally useful.
The basic gist of the concept is this: Iterate over two strings making a note of how many characters were inserted, deleted, or transposed from one string to the other. When a difference is found, "bookmark" where you are and start looking ahead in each string to see if the strings are going to start matching up again down the line. How far down the line you look is controlled by in an input called maxOffset. The LCS is the number of characters between the two strings which were identical. The "distance" of the two strings is simply the average string length minus the longest common string. The similarity of the two strings can be expressed as a percentage given 1 minus the strings' distance divided the length of the longest string. Enough theory-- let's look at an example:
I love to go ride my go-kart. (29 chars)
I love to ride my go-cart outside. (34 chars)
- There is a deletion of the word "go "
- There is a substitution of "c" for "k" in go-cart
- There is an addition of the word "outside"
Overall, the average length of the strings is 31.5 characters.
There are 25 characters in the two strings that are identical. This is our LCS.
String 1 is 4 characters different than string 2, and string 2 has 9 characters different than string 1.
So on average, we will say 6.5 characters would need to be changed to make the strings identical. This is our distance. (4 + 9 = 13 / 2 = 6.5)
Divide that distance by the length of our longest string and we can come to the conclusion that the strings are an 81% match. (1 - (6.5 / 34) = .8088 = 81%)
Here's another example:
|
The rain in Spain stays mainly on the plains The rain in Spain stays mainly on the plains The rain in Spain stays mainly on the plains Lorum Ipsum, yadda yadda. Lorum Ipsum, yadda yadda. La La La La Luke, I am your father. |
The rain in Madrid stays totally on the plains The rain in Spain stays mainly on the plains The rain in Barcelona stays entirely in the air Lorum Ipsum, Yabba dabba doo. Whatcha eatin? Nutin' Honey. Da Da Da Duke, I am your father. |
- Roughly 67 characters are different between the two strings.
- The Longest Common String (LCS) is 180.
- The strings are a 73% match.
In addition to porting the logic over to CFScript, there are a few things I added in to my function:
- If both strings are empty, I short circuit and return a 0 for distance and LCS. Similarity is 100%.
- If either string is empty, I short circuit and return the length of the non-empty string as the distance. The LCS and similarity is 0.
- To better detect differences at the start of the string I check the first three characters. That way a matching first letter wouldn't be confused in two strings like "top hat" and "the hat"
- When looking ahead in the strings trying to reconcile a difference I search for the distance of the maxOffset until I find THREE contiguous matching characters. This is to try and eliminate false positives.
- My function will highlight the differences between the strings by wrapping the deviations in the HTML tag of your choice. Default is <span style="background: yellow;"></span>
The function is fairly effective for finding basic insertions, deletions, and transpositions between to strings ranging form a few words, to a few paragraphs. Since the algorithm iterates through two strings without looking back, it WON'T find sentences that had their order rearranged. The maxOffset controls how hard the code tries to look and ahead and reconcile an insert or deletion. If you expect entire sentences to be inserted, your maxOffset needs to be at least as big as the largest insertion. Of course, the larger offset you allow for, the more performance will be impacted and the more likely you are to get a false positive when looking ahead.
Have fun with it. If you can think of a way to improve the code I would love to hear about it. I have included the code for the function, an example way to call it, and a zipped version of both below.
2
3 /*
4
5 StringSimilarity
6 Brad Wood
7 brad@bradwood.com
8 May 2007
9 Code adopted from Siderite Zackwehdex's Blog
10 http://siderite.blogspot.com/2007/04/super-fast-and-accurate-string-distance.html
11
12 Parameters:
13 s1: First string to be compared
14 s2: Second string to be compared
15 maxOffset: Average number of characters that s1 will deviate from s2 at any given point.
16 This is used to control how far ahead the function looks to try and find the
17 end of a peice of inserted text. Play with it to suit.
18
19 */
20
21 function stringSimilarity(s1,s2,maxOffset)
22 {
23 var c = 0;
24 var offset1 = 0;
25 var offset2 = 0;
26 var lcs = 0;
27 // These two strings will contain the "highlighted" version
28 var _s1 = createObject("java","java.lang.StringBuffer").init(javacast("int",len(s1)*3));
29 var _s2 = createObject("java","java.lang.StringBuffer").init(javacast("int",len(s2)*3));
30 // These chaactes will surround differences in the strings
31 // (Inserted into _s1 and _s2)
32 var h1 = "<span style=""background: yellow;"">";
33 var h2 = "</span>";
34 var return_struct = structNew();
35 // If both strings are empty
36 if (not len(trim(s1)) and not len(trim(s2)))
37 {
38 return_struct.lcs = 0;
39 return_struct.similarity = 1;
40 return_struct.distance = 0;
41 return_struct.s1 = "";
42 return_struct.s2 = "";
43 return return_struct;
44 }
45 // If s2 is empty, but s1 isn't
46 if (len(trim(s1)) and not len(trim(s2)))
47 {
48 return_struct.lcs = 0;
49 return_struct.similarity = 0;
50 return_struct.distance = len(s1);
51 return_struct.s1 = h1 & s1 & h2;
52 return_struct.s2 = "";
53 return return_struct;
54 }
55 // If s1 is empty, but s2 isn't
56 else if (len(trim(s2)) and not len(trim(s1)))
57 {
58 return_struct.lcs = 0;
59 return_struct.similarity = 0;
60 return_struct.distance = len(s2);
61 return_struct.s1 = "";
62 return_struct.s2 = h1 & s2 & h2;
63 return return_struct;
64 }
65
66 // Examine the strings, one character at a time, anding at the shortest string
67 // The offset adjusts for extra characters in either string.
68 while ((c + offset1 lt len(s1))
69 and (c + offset2 lt len(s2)))
70 {
71 // Pull the next charactes out of s1 anbd s2
72 next_s1 = mid(s1,c + offset1+1,iif(not c,3,1)); // First time through check the first three
73 next_s2 = mid(s2,c + offset2+1,iif(not c,3,1)); // First time through check the first three
74 // If they are equal
75 if (compare(next_s1,next_s2) eq 0)
76 {
77 // Our longeset Common String just got one bigger
78 lcs = lcs + 1;
79 // Append the characters onto the "highlighted" version
80 _s1.append(left(next_s1,1));
81 _s2.append(left(next_s2,1));
82 }
83 // The next two charactes did not match
84 // Now we will go into a sub-loop while we attempt to
85 // find our place again. We will only search as long as
86 // our maxOffset allows us to.
87 else
88 {
89 // Don't reset the offsets, just back them up so you
90 // have a point of reference
91 old_offset1 = offset1;
92 old_offset2 = offset2;
93 _s1_deviation = "";
94 _s2_deviation = "";
95 // Loop for as long as allowed by our offset
96 // to see if we can match up again
97 for (i = 0; i lt maxOffset; i=i+1)
98 {
99 next_s1 = mid(s1,c + offset1 + i+1,3); // Increments each time through.
100 len_next_s1 = len(next_s1);
101 bookmarked_s1 = mid(s1,c + offset1+1,3); // stays the same
102 next_s2 = mid(s2,c + offset2 + i+1,3); // Increments each time through.
103 len_next_s2 = len(next_s2);
104 bookmarked_s2 = mid(s2,c + offset2+1,3); // stays the same
105
106 // If we reached the end of both of the strings
107 if(not len_next_s1 and not len_next_s2)
108 {
109 // Quit
110 break;
111 }
112 // These variables keep track of how far we have deviated in the
113 // string while trying to find our match again.
114 _s1_deviation = _s1_deviation & left(next_s1,1);
115 _s2_deviation = _s2_deviation & left(next_s2,1);
116 // It looks like s1 has a match down the line which fits
117 // where we left off in s2
118 if (compare(next_s1,bookmarked_s2) eq 0)
119 {
120 // s1 is now offset THIS far from s2
121 offset1 = offset1+i;
122 // Our longeset Common String just got bigger
123 lcs = lcs + 1;
124 // Now that we match again, break to the main loop
125 break;
126 }
127
128 // It looks like s2 has a match down the line which fits
129 // where we left off in s1
130 if (compare(next_s2,bookmarked_s1) eq 0)
131 {
132 // s2 is now offset THIS far from s1
133 offset2 = offset2+i;
134 // Our longeset Common String just got bigger
135 lcs = lcs + 1;
136 // Now that we match again, break to the main loop
137 break;
138 }
139 }
140 //This is the number of inserted characters were found
141 added_offset1 = offset1 - old_offset1;
142 added_offset2 = offset2 - old_offset2;
143
144 // We reached our maxoffset and couldn't match up the strings
145 if(added_offset1 eq 0 and added_offset2 eq 0)
146 {
147 _s1.append(h1 & left(_s1_deviation,added_offset1+1) & h2);
148 _s2.append(h1 & left(_s2_deviation,added_offset2+1) & h2);
149 }
150 // s2 had extra characters
151 else if(added_offset1 eq 0 and added_offset2 gt 0)
152 {
153 _s1.append(left(_s1_deviation,1));
154 _s2.append(h1 & left(_s2_deviation,added_offset2) & h2 & right(_s2_deviation,1));
155 }
156 // s1 had extra characters
157 else if(added_offset1 gt 0 and added_offset2 eq 0)
158 {
159 _s1.append(h1 & left(_s1_deviation,added_offset1) & h2 & right(_s1_deviation,1));
160 _s2.append(left(_s2_deviation,1));
161 }
162 }
163 c=c+1;
164 }
165 // Anything left at the end of s1 is extra
166 if(c + offset1 lt len(s1))
167 {
168 _s1.append(h1 & right(s1,len(s1)-(c + offset1)) & h2);
169 }
170 // Anything left at the end of s2 is extra
171 if(c + offset2 lt len(s2))
172 {
173 _s2.append(h1 & right(s2,len(s2)-(c + offset2)) & h2);
174 }
175
176 // Distance is the average string length minus the longest common string
177 distance = (len(s1) + len(s2))/2 - lcs;
178 // Whcih string was longest?
179 maxLen = iif(len(s1) gt len(s2),de(len(s1)),de(len(s2)));
180 // Similarity is the distance divided by the max length
181 similarity = iif(maxLen eq 0,1,1-(distance/maxLen));
182 // Return what we found.
183 return_struct.lcs = lcs;
184 return_struct.similarity = similarity;
185 return_struct.distance = distance;
186 return_struct.s1 = _s1.toString(); // "highlighted" version
187 return_struct.s2 = _s2.toString(); // "highlighted" version
188 return return_struct;
189 }
190
191
192 </cfscript>
2 The rain in Spain stays mainly on the plains
3 The rain in Spain stays mainly on the plains
4 Lorum Ipsum, yadda yadda.
5 Lorum Ipsum, yadda yadda.
6 La La La La Luke, I am your father.">
7
8 <cfset string2 = "The rain in Madrid stays totally on the plains
9 The rain in Spain stays mainly on the plains
10 The rain in Barcelona stays entirely in the air
11 Lorum Ipsum, Yabba dabba doo.
12 Whatcha eatin? Nutin' Honey.
13 Da Da Da Duke, I am your father.">
14
15 <cfset comparison_result = stringSimilarity(string1,string2,10)>
16
17 <cfoutput>
18 Roughly #comparison_result.distance# characters are different between the two strings.<br>
19 The strings are a #numberformat(comparison_result.similarity*100)#% match.<br>
20 The Longest Common String is #comparison_result.lcs#.<br>
21 <br>
22 <table border="1" cellpadding="10" cellspacing="0">
23 <tr>
24 <td>
25 #replacenocase(comparison_result.s1,chr(10),"<br>","all")#
26 </td>
27 <td>
28 #replacenocase(comparison_result.s2,chr(10),"<br>","all")#
29 </td>
30 <tr>
31 </table>
32 </cfoutput>
I was about to add a feature to a project that needs to tell the user that two pieces of text aren't different enough (for SEO purposes) and this looks like it'll work a treat.
Many thanks.
I need to correct an error in some software that has led to a database being out of touch with another reference database - don't ask about the bad DB design.
If this works, I'll owe you a cupcake!
Seriously though, if you need to compare two databases, Redgate software makes some very kick butt tools for that. They will show you line by line comparisons of stored procs and stuff, but when it comes to the differences in data I think it just tells you they aren't the same.
Nice script! .. .but you need to var the return_stuct too.
/ Andy
www.cfwheels.org
I updated the post and the download.