starting to sound like Zeno here...

Anyway, well first, some scientists would disagree, some scientists have begun trying to prove that time is quantized, that it comes in tiny, tiny packets. Like matter, we used to think you could divide it infinitely tiny, but now we understand that there are atoms, and inside those there are certain fundamental particles, which cannot be split any further.
Of course, those are made of energy, and I'm not aware of any restriction on how small amounts of energy can get...
But no matter, some scientists have tried to show time is quantized, and also not infinite. A lot of theories think before the Big Bang there was not only no space, but no time either, it seems like time might have had a beginning, although, as the universe appears destined to expand forever, we're not sure if it has an end.
And then you get into the very nature of time, which, although it seems to us like we understand it, in daily life, I'm here now, in 12 hours I will instead be at school. But when you actually really start thinking about time, and indeed space (As of course, they are the same thing, 4 dimensional spacetime) we realize we don't understand them at all. But it's really interesting.

Going back to the beginning of your comment, I've thought a lot on how the universe is either infinite, or really, really, really, really big, vastly huge, and we're just one of probably about 1000 billion billion planets, and how time may or may not have had a beginning but it might not have an end, and so could be infinite, and in any case my lifespan is less than a blink of an eye in cosmological time (If the life of the Earth was reduced to a day, the entire modern human race would have been around for less than a second).

But I don't see why that should be depressing. I mean, sure, if I think about it on a cosmological level, I'm tiny, beyond tiny, so miniscule that even if I blew myself up with a nuclear bomb, to the universe it would not have the slightest difference one way or the other, it is completely neutral to whatever I do.
But we don't live on a cosmological level, we live on sometimes a planetery, sometimes national, sometimes statewide, sometimes just our high school level, and this is a place where even a fishmonger lighting himself on fire can have repercussions across the entire world. And so, for example, while the universe doesn't consider you significant in the slightest at all, you're significant to me, and I don't think we really need anything more than that. We humans are significant to each other, and that's how we live our lives, where one human can change the world. Who needs the rest of the universe?
And also, I think the universe being really, really big and time being really, really long is actually kinda a good thing. If the Earth was really only 6000 years old, and humans were the only thing that mattered, to me, the universe would be a fuckin boring place. It'd be you, your job and house, and then everything else outside, that's God, that's all the explanation we need.
But the universe is so big and has been around so long that we fallible humans, I think, will never discover all its secrets, not if we work at it for a thousand years, and so that means there'll always be new things to know, new things you can look at through the telescope, new weird things about the early or late universe you can postulate, and we're finding new things all the time, such as those neutrinos that appear to be moving faster than light, and I think it's wonderful the world isn't small and simple, it's big and complex and beautiful, and if nothing else, you can never get bored working at trying to unlock its secrets, even if you're doomed never to find them, the purpose for living is the searching.

http://www.youtube.com/watch?v=xt5ghXdq6Z0&safe_search=on