Should I concern about mobile browsers? Should I optimize my site for responsive browsing? Yes and no!
At first - yes, you should invest some time and money to build a mobile devices friendly site. If your site visitors uses Facebook and or twitter, and you promote your content in those sites - answer is - you definitely should have a responsive site:
Take a look of device use in 2012
This is a screen for some site I maintain, for 2012. As you can see - mobile devices makes really small amount of devices used to browse that site.
Here is the screen for 2013
As you can see use of mobile and tablet devices have increased some 5 times.
But still it doesn't make 50%, 40% or even 30% of site users. Well it doesn't mean we should forget about site users browsing our sites with mobile devices. If you haven't invested yet, you should ASAP invest in your sites responsive design.
Probably you have heard - there is rumors that mobile web is making already some 50% of web traffic. That's probably true. But what does it mean for you business? Basically mobile has gone large thanks to sites like Facebook and Twitter.
Take a look at this screen:
This is screen of the same website I talked at start - in this screen you can see Facebook ads for promoting this pages Facebook fan page. As you can see - mobile users have really high CTR - 3.232% and many page likes came from mobile users. So if I'm willing to share links on my Facebook fan page - I must give a responsive web for Facebook fans - because the number is pretty high, right?
In conclusion - If you are not using any Social sharing, relying just on old good Google - then you shouldn't worry. But if you are using social networks to promote your content - invest in responsive web.