Optimization and extra features are things that can often delay software projects. Sure, it's nice to have performant code with good set of features from the start. But released software is almost always better than couple of missing features or room to speed things up.
It's the same things as with the innovation versus stability debate: where is the line between those two. Minimum viable products have been popular in modern development: just ship the barebones and keep improving afterwards. There is no need to waste all that time on features nobody will ever use. And you usually have some time for optimization before you are overwhelmed with the users.
But there are things that should not be neglected no matter how stripped down first release you are aiming to release. Security is the most important one obviously. That should be taken into account right from the beginning. Maintainability and extensibility are good things to keep in mind as well. Even though it might be better idea to rewrite the whole things at some point, in practice that hardly ever happens. So you'll be stuck with it anyway. Apart from those you should leave it there. Some features might feel like easy additions, but it's still better not to increase the complexity too soon.