This depends. Musical people will most likely find the Hard mode easiest, since it's more rhythmic. However, when I watched less musical people play the Hard mode, they could not tell the two "main" drum sounds apart from all the other rhythmic stuff, some people just don't have that skill. Those players will find the Easy mode a lot easier, since the sounds you should listen to stands out a lot more in that version and can't be confused with other sounds.
In fact, I've seen a lot of people not being able to play back the "S" drum at all, in neither of the modes, despite that they watched me play the game before. They mysteriously could not make out that sound from the rest.