For an english speaker? I'd say that one of the Northeast Caucasian languages would be the most difficult.
They have up to 60 consonant sounds, up to 30 vowels, etc.
http://en.wikipedia.org/wiki/Northea...sian_languages
There is also Finnish and hungarian, which sound nothing like indo-european languages, and finnish has 16 noun cases if I remember correctly.