Of course I do think that if the idea is specific, you should always ask the author, otherwise it's just rude.

That's what I was thinking. I credited authors for specific ideas in the past, but in this case it was a sort of vague idea, but I didn't want it to seem like I ripped off their story, so I went ahead and asked and got permission.

It made me curious what everyone else thought about this and the replies I got so far are all really interesting.