For me its Honorifics and Counters.
Honorifics. because if you want to politely say "Eat" and you say 먹으세요 you will be WRONG. And if you want to politely tell somebody to sleep and you say 자세요 you will be WRONG. And who knows how many honorific verbs are hiding in the shadows? Its enough to drive me insane. If it was just adding 시 to a verb I would be OK, but there are so many rules it makes me want to know only people my exact same age!
Why oh why did the Korean people a loooong time ago decided that different things needed different counters? Why couldn't they just say "one of this and 3 of that"? Why does a horse need a different counter than a spoon? I'll never be able to remember them all.
Those 2 just kill me the most. The honorifics one is the worst because people will take it personally if you screw up. If you use the wrong counter when counting a car nobody will hit you with their cane.
What frustrates you the most?