A little white lie on your CV or false information on a job application may be much easier to detect from now on with the help of new text mining software.
Text mining's no genius, but it could catch you out
Text miner, still in test stages, is being designed by US firm SAS Institute. It follows in the footsteps of SAS's successful 'data mining' software, used by large organisations such as credit card firms to spot irregular shopping patterns.
Text miner works on the same principle as its predecessor, by recording shifts in patterns, groups and styles, in this case of words rather than numbers, to determine whether someone is concealing the truth.
"It is crucial to understand that the program cannot understand the meaning of words per se," said Peter Dorrington, business solutions manager for the SAS Institute in the UK. "Instead it looks for relationships between words by comparing a whole selection of answers to the same question. Isolating a pattern between those answers and then highlighting the answers outside the pattern."
In simple terms the software will recognise a connection between green and cucumber but not between red and cucumber and will therefore label this as an irregularity, as well as a possible lie, to be investigated.
"The software cannot work without people. Its job is to raise the level of suspicion and encourage a [human] investigation," said Dorrington.
"Anything which alerts suspicion has to be beneficial, as long as the software is run alongside an investigation team, not instead of, and can highlight issues which could be followed up," said a spokesman at the Metropolitan Police Service.
But the problem is where the line between checking facts and snooping should be drawn.
"There are laws which prevent companies from [snooping on emails] and action would be taken if companies were acting against them," said the Met Police.
Text mining would, according to Dorrington, be useful for many purposes, aside from identifying lies.
"It would be perfect for customer services purposes. If a customer sent in an email on a particular subject, the software could forward that email on to the right person by recognising patterns of words in the sentence," said Dorrington.
The software is due for release towards the end of the year. As yet no beta copies have been distributed to UK law enforcement agencies or companies.