Yep, it's all about the velocity.
Personally, I wouldn't carry either a .380 or 9mm as a daily carry weapon. .40 is the smallest caliber I'll trust my life to and what I carry most often. If I'm in a situation where I simply can't get away with concealing a .40, I'll carry my .380 as it's better than nothing, but I sure wouldn't plan on dropping someone on first shot. Aim well and squeeze often.