Take a look at any really well-done fantasy map that is attached to a fictional universe and you'll start noticing patterns.
There are, to date, well over one hundred nations and states in the Thaumopolis world, the 'earth of my choosing' so to speak in regards to my main fantasy series (and the main theme/purpose of this website at large.) There are, however, thousands of those nations that span the width and breadth of the world's history. Which is a lot. Unnecessarily so, even, when you think about how much can possibly be written by one man through the course of his time on this planet. But it's there as a passion project attached to the world that I lovingly crafted as much as it is there to understand how certain aspects of my world function.
And that is, I think, where many seem to go wrong in fantasy and science fiction.
To be clear, one does not require a nuanced history of all the locations of your world. Especially if you are a one or two person team crafting the entirety of it all. However, I posit that it will never hurt to add more depth to the backstory of your creations. Not because the audience will see it, but because it will inform your understanding of how your world and your characters function. The audience will likely never even know that you did the work, but it will show in your story that you did it.
Here I will switch professions for a moment to give an example that I think translates extraordinarily well. In the profession of acting and performance, it is one of the foundational notions that you need to know the following about the character you will be playing: "Where they are coming from, where they are going to, and why are they doing this?" And the need to know the answer to all three is paramount in literally every single moment that your character is present. The more refined your answers, the more advanced the performance will be. Even if it has nothing to do with what's going on immediately in front of the audience, and even if the detail will never be directly mentioned. You know what your character was doing/why/how/etc because it will make it more realistic in your imagining and interpretation of the events in the performance.
Likewise this translates over to any story, not least of all crafting a fantasy world.
Begin asking yourself questions about how and why people are how they are in your setting, or in the setting of a piece you're just reading. While you might not register it in your day to day life, how you act and perceive your world is informed by your cultural background, which is in turn informed by hundreds of years of progress by those that came before you. Even if you rail against the ideas of the society you exist in, that rebellion is informed by the decisions of men and women who came well before you and your choice to adhere to or fight against those ideas that made your existence even possible. So too does this apply on the grander scale of world politics and nationality. The United States today faces many issues that it does almost exclusively because of events and actions taken by their forefathers.
As such, when crafting a fantasy or science fiction setting, the background is important for more than just letting your audience know when and where people are. It's useful to you so that you can use the information to craft a more believable performance on a macrocosmic scale. Your characters will have opinions that are shared or disputed from their wider community. That community will in turn have opinions largely 'agreed on' that conflict or agree with other groups around them. And all of these opinions will have been grown, over time, depending on their past.
Take, for example, The Lord of the Rings.
The Shire was established by a wandering group, The Hobbits, after many years of homelessness and hardship. Not only do their nomadic roots come into play with their wider mentality, but also the fact that they are fiercely protected from the wider world's evils by elite rangers that the Hobbits don't even really seem to know about. This history has set the Hobbits up to have a focus on creature comforts that the rest of the world does not have. As such, they are secluded, if friendly, people who focus more on a satisfyingly comfortable life rather than an adventurous one. Meanwhile this greatly informs characters like Bilbo, Frodo, and Sam, because they have made decisions to go against or to try and adhere to Hobbit lifestyle.
Bilbo is nervous to begin a journey. Culturally it's just not what his people are supposed to do. The Shire as a political entity (loosely defined here) wants its people to stay in its borders to grow fat and happy. In part because they remember, historically, what it was like before, and because they have no reason to venture outside the borders. But Bilbo decides to go against the grain of his society and acceptable behaviors because of curiosity and a sense of wonder. Meanwhile, Frodo wants to intentionally defy the social order because Bilbo had such a grand adventure that, to Frodo, seems ideal. Frodo is intentionally rebellious in his desire to adventure and ultimately pays for this with a trauma that can't fade until he sails west with the elves.
And then Sam, who makes his decisions out of necessity - because of his loyalty to his friend, Frodo. He does not want to go on an adventure, it is not appealing to him in the slightest. He is entirely a product of The Shire's wider cultural zeitgeist. However what makes him distinctly heroic and real to us as an audience... Is that despite not wanting it, he does see the quest to the end. Even though he does not want it, partially because his society has bred him to not want it.
History and nations inform everything in your story, even if it's not immediately apparent how. The history informs the culture, the culture informs the people and their opinions, those people and opinions inform the actions taken and how, which in turn makes the story.