Bsrgvty

I have a text file like this:

tom
and
jerry
went
to
america
and
england

I want to get the frequency of each word.

When I tried the following command

cat test.txt |sort|uniq -c

I got the following output

 1 america
 2 and
 1 england
 1 jerry
 1 to
 1 tom
 1 went

But I need partial matches too. ie, the word to present in the word tom. So my expected word count of to is 2. Is it possible using unix commands?

Here's one way, but it isn't very elegant:

$ sort -u file | while IFS= read -r word; do 
 printf '%st%sn' "$word" "$(grep -cFe "$word" file)"; 
 done
america 1
and 3
england 1
jerry 1
to 2
tom 1
went 1

An awk approach:

awk '
 !x c[$0]; next
 for (i in c) if (index($0, i)) c[i]++
 ENDfor (i in c) print c[i]"t"i' file x=1 file | sort -k1rn

Which on your input give

3 and
2 to
1 america
1 england
1 jerry
1 tom
1 went

We process the input in two passes. In the first pass, we record the list of distinct words as the keys of the c hash table.

In the second pass, for each line in the file, we loop over all the keys in c and increment the corresponding value if the key is found in the line.

The list of distinct words in the file ends up being stored in memory. If those are English words, it shouldn't be a problem as there are fewer and 200,000 distinct words in the English language.

This won't crash the system but it may take a long time to run, since it parses the input multiple times. Assuming the input file is called "in":

sort -u < in | while read w
do
 printf "%dt%sn" `grep -c "$w" in` "$w"
done

which on your input got me:

1 america
3 and
1 england
1 jerry
2 to
1 tom
1 went

It's not clear to me if the partial matches are to be anchored to the beginning of the line.
Assuming the answer is yes, what might speed things up here is the use of binary search via the venerable look command.
Of course look needs that its input file be sorted.
Therefore, first create a sorted version of the original file

 sort file > file.sorted

Then loop through the original file, looking up one word at a time against the sorted file.

while read -r word; do 
 printf "%s %dn" "$word" "$(look -b "$word" file.sorted | wc -l)"; 
done <file

Some systems don't need the -b flag to be passed to look to force a binary search.
Disk caching of the sorted file could help speed things up even further