In my opinion? Neither. You start with a story.
People will only care about the setting and characters if there’s a story that forms the backbone for the creative work.
Take for example, the utterly amazing Gunnerkrigg Court. Sure, the mysteries of the Gunnerkrigg Court and its characters continue to compel but it all began because we’re following the adventures of a young fire-headed girl as she makes friends, experiences a coming-of-age and stands at the gap between Man and Nature.
Once you have a story that draws people in, you can start building the world and characters around it. Personally, you’ll be much more successful that way. Cheers. :)