[This answer is from a Young Earth Creationist perspective, taking the first chapters of Genesis fairly literally compared to some other perspectives. It's informed by what I've heard over the years about Cain and the early history of humanity. I've put some resources at the end about Cain from groups with similar perspectives to the one I present here.]
The short of it is that by the time this happened, there were definitely more than the four humans you mentioned around. (I mean, the verse you're asking about would itself be evidence for that, right?) Adam and Eve had other children than just Cain, Abel, and Seth (see Genesis 5:4), it's just that we don't know the names of most of their children and most of them don't feature in the story. The start of Genesis 4 jumps straight from the births of Cain and Abel to them keeping flocks and working the soil, but presumably they weren't farmers as soon as they were newborns. So in this time between them being born and when the story starting at Genesis 4:3 starts, there were other children who had been born (and perhaps some of them even had children of their own as we don't know exactly how much time passed before Abel was murdered). Maybe some of them may have already been living over in Nod or at least they had spread out a bit.
The common related question that often confuses people is where Cain's wife came from, but I think that's answered by this too: Cain was probably married (to another of Adam and Eve's children, yes) and brought his wife with him to Nod. (They just didn't have children, or at least didn't have his son Enoch, until after they moved to Nod.) Or maybe he wasn't married until after he moved, but regardless there were certainly other people around at the time of Cain's big move.
Seth was born after Abel died (Genesis 4:25), and Adam did see Seth as some sort of special replacement for Abel. But it doesn't look to me that Seth was the thirdborn child, because there clearly were other people around the earth at this time. Seth wasn't born until Adam was 130 years old (Genesis 5:3), and Adam and Eve were told to fill the earth with offspring (Genesis 1:28) and in that well over a hundred years of having children they had probably done a decent job of doing so. There could have been quite a lot of people around by this time.