I have this conversation quite often.
Living in Melbourne and working with both the Japanese community and local practitioners, I meet a lot of bilingual Japanese people. No matter where we are, the conversation will naturally turn to linguistics (an obvious common ground). The Japanese person will argue that it must be more difficult for a native English speaker to learn Japanese; I will rebut that it is more difficult the other way around.
Of course, I don’t think there is an actual conclusive answer to this debate, and we are both biased because we’ve only ever experienced learning the other language.
Let me lay out my reasoning for this, though. It makes for an interesting discussion.
The Japanese person will without fail say that reading and writing Japanese is a huge obstacle for a native English speaker. The Japanese language is a nightmare of symbols to decipher, comprising of three different writing systems and consisting of thousands of symbols. This is generally their main argument.
On the surface, that may be so. But speaking from experience, learning to read/write Japanese is much like learning to read and write in English. All Japanese symbols have meanings, and specific parts of each character show up in similar words, making it easy to guess the meaning of a word even if you’ve never seen it before. For kanji, in particular, the more you learn, the easier it gets.
I find this similar to spelling in English – you learn regular patterns for prefixes and suffixes (-tion and -un/-in, et cetera), increased exposure allows you to guess the meaning, similar pronounciation is spelt in similar ways. It’s the same basic concept, but Japanese kanji follows a more logical line of reasoning than English words do.
For me, the deciding factor is grammar (English grammar is incredibly complex; Japanese grammar is quite simple), as well as phonetics. Japanese has a very limited collection of sounds, so when learning English, Japanese people have to learn how to form sounds that don’t even exist in their own language. Crazy, right?
Japanese, on the other hand, is very easy to pronounce for an English speaker (save the ‘r’ sound, every sound already exists in English).
I could write for hours about the relatively difficulty of each language, but personally English seems harder to me. I teach English to Japanese students in Japanese, and even knowing the grammatical explanations for articles, singular/plural, tense, etc. can only help you so far. There is a point when you just end up confusing yourself and wondering how people are able to communicate in English in the first place.