An indexing engine generates a full text index of English and non-English files provided to the indexing engine. The indexing engine receives an input file for indexing, and normalizes the unique words contained in the input file. The normalizing includes stripping the words of any diacritical marks, taking into account different multilingual issues, case folding the words into lowercase, and the like. The normalized words are stored in a dictionary, and a word record is generated for each stored word. Each word record includes a flag that indicates whether one or more variations exist in the input file for the normalized word. One or more tables store information on the variations for the normalized words. When a query engine is invoked to search for an input query word, the variations are searched only if the user has set an option to consider such variations.