I'm going to go ahead and disagree with Sumner and say that if it's an account that you care about (financial, primary personal e-mail, major social media like Twitter and Facebook) that you should absolutely enable SMS, if there's no other alternative to a second factor authentication. Google Authenticator is the best option for any service that offers it.
There are a few different ways that your SMS can be compromised. The most common is going to be by hacking or socially engineering your cell phone provider. (Make sure that you've got whatever security options available enabled by your carrier). Once they can make changes to your account whether via browser or by calling into a call center they can change the SIM or ESN associated with your account to send SMS to another device. There are some pieces of mobile malware (mainly on Android requiring the user to turn off protection around 3rd party or non signed applications) which will intercept SMS based on the sending number, forward it and delete it without a notification. I've got a copy of the iBanking android malware and the command & control server, and while it's fun to play with, it's a lot less common than previous method. There is a physical device that can intercept SMS messages, but it requires both expensive (and illegal) equipment plus proximity to the user. For the sake of this conversation, let's assume that you're not a nation state target or that if you are your more sensitive information is already protected by something more secure than SMS.
The problem with not using SMS when offered is that you're otherwise left to fall back on either security questions or e-mail as both your step up authentication or, more importantly, for account recovery. How many of your accounts are tied back to your e-mail? If your e-mail is compromised, an attacker has access to every one of those accounts as well if you don't have additional recovery options configured. If your Facebook or Twitter are compromised, then there's a very direct line of access to your personal contacts under your identity.
Websites are generally embarrassingly (and often negligently) slow to implement security controls, and it's likely going to be a significant period of time until every site is following a best practices. They are by no means required to follow NIST 800 standards either. The FFIEC guidance that dictates security to your financial accounts is just as designed around user experience more than it is perfect security. (Most of you would switch banks to one with less security, when given an option of the former) If you're going to spite your security by "keeping your phone number secure", then you're going to open yourself up to a lot more risk.
Oh and we should probably also note that your authentication into SoSH is completely insecure and if you're using a password on SoSH that you're using on any other website then you should go ahead and change all of those accounts immediately.