Wednesday, December 9, 2009

Senseq's Web hosting 101

Over the year that we've been perfecting our site, we have always had a beta version up and running.
For a bootstrapped startup, its vitally important to save wherever possible, and we've tried lots of possibilities. Here are some that might help you decide what hosting is right for you (from cheap to expensive)

Note: I haven't written about cloud based hosting here, you'll find a good article on cloud based hosting at this link.

1. Free or shared hosting:

Free or shared hosting providers such as godaddy or bluehost (recommended) can provide you really cheap hosting (under $10) per month, your own dot com domain, and huge amounts of data transfer, storage and even shell access.

The downside? You won't be able to run cpu intensive tasks, and your data transfer rates may not be impressive.

Recommended For: When your site when its just taking shape: mostly static, and few users.
Also, your blog
Cost: $10 or less per month

2. Home based hosting:

It is possible to use a spare computer (or your desktop that's always on) as a webserver that can serve out your site.
It isn't too difficult, needs average technical skills, and might not cost you a penny!
You can do this without a static ip, by using a service like dyndns, and can get free domains (non-dot-com) or dot-com paid domains.

Make sure, before you go for this, that you won't run into data overages with your ISP (most ISPs in the USA provide unlimited bandwidth, so it should not be a problem).

Also, there can be potential slow-down of your home based network. You can read on a possible situation here, and how to fix it.

Recommended For: Users with website setup skills. Excellent low cost/high performance initial investment
Cost: $30 or less per year. Excluding internet charges.

3. VPS Hosting:

VPS (Virtual Private Server) hosting can be the next step from home based hosting or shared hosting - especially when you aren't satisfied with the performance of the system or the network.

VPS hosting gives you shell access, and you are guaranteed a minimum RAM and system performance.

This would be a stop-gap arrangement before going for a dedicated server or managed service.

For VPS/Dedicated hosting, I recommend softlayer.

Recommended For: When you need more performance from the system - CPU or bandwidth
Cost: $50 - $100 per month.

3. Dedicated and Managed Hosting:

Once your website is on its way to success, or you want a site that you have full control over, and is ready to scale - consider dedicated or managed hosting.

Managed hosting is one, where the company makes sure your server is running just fine, and you only need to keep your code in place. They worry about patches, alerts etc.
Dedicated hosting, is where you get a server with an operating system of your choice. You do everything else!

You can get any configuration you like, but the price keeps going up. I do recommend starting with a lower end server with more RAM, and then going up with configuration as you need.

Recommended For: More scalability and advanced technical skills
Cost: $100 - ?? per month.

4. Co-located Hosting:

For the technically inclined, co-located hosting is the premium hosting solution.
In co-location, you own your own server, and possibly firewalls, and install them in the co-location facility.

Apart from having guaranteed power and network bandwidth, you also get secure entry to your server, fire protection, video surveillance etc. As you can imagine, co-located hosting is more than just about getting a dedicated port for your server.

Once your site is up and running, and needs a stable facility for your computer - which you manage - co-location is the option for you.

Recommended For: Ownership, high security and performance, and advanced technical skills
Cost: $150 - ?? per month.

I hope you find this blog useful. Do leave me feedback and ask any questions.

Why an in-home server can slow down the network

Even with the best (in-home) technologies, the server can hog your home network, and bring everything down to a crawl. Case in point, my 5MBPS connection started to feel like dialup, as I browsed in slow motion.
How can this happen? And how do you prevent this?

High active ip connections.

Here is what can happen with a poorly configured server:

Let's say you setup a Linksys WRT-64G router which you use at home, with your new webserver on a lan port. Over time, your server will establish new connections with users (based on the apache settings). These connections, and the router timeout limits, can severely hamper the overall network performance.

The reason? Connections aren't killed after they are used. And the router runs out of memory trying to keep track of all the connections.

The really odd symptom, is that when you open a new browser window, it struggles to connect to a website. But your other windows are running better.

Here is a simple remedy to the problem:


1. Router setting changes:

First lets check if your router is complaining:

a. Check in your router status page for the number of Active ip connections. Typically this should not be more than 30 per computer using the router.

b. If the Active ip connections, are high (500+), then check if most of the active connections are from your server (in dd-wrt, you can do this by going to Status > Click on the number against 'Active IP Connections'.

c. If you find it is your server, you've hit jackpot. Time to set things straight:
Set the router to terminate unnecessary persistent connections.
You can do this by reducing the TCP and UDP timeouts (I recommend TCP timeout 600 seconds and UDP timeout 120 seconds).

This will prevent the router from keeping unnecessary connections on.

2. Server setting changes:

Apache likes to setup a lot of connections so you get a decent amount of multi-user performance from your website. If yours is a new website, likely it will take time to get bombarded by too many users.
Given that you want to share your home network with your server, these settings will help reduce the demand from the server:

a. Go to the apache config file (in ubuntu, it is: /etc/apache2/apache2.conf), and modify the mpm worker and preform modules as follows. (If you already have the same Server settings, then considering reducing them further)

<IfModule mpm_prefork_module>
    StartServers         5
    MinSpareServers      5
    MaxSpareServers      10
    MaxClients          100
    MaxRequestsPerChild  500
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>


b. Restart your apache server (in Ubuntu: sudo apache2ctl restart)

That's it! And your apache server will run much lighter now - especially with fewer extra servers lurking around.

3. Check if everything looks good:

You should now see a drop in number of active ip connections.
Also, soon your in-home network will get back to its peppy self.

I hope this helps you!

Saturday, December 5, 2009

The mysql slow query log

The first step to tuning your database, is to find out what's running slow.

I recommend doing tuning your queries before changing your MySql memory allocation, sessions etc.

Its always better to treat the problem ground up (by finding rogue queries) rather than put a quick fix and forget about it (e.g. allocating more memory or setting sessions). Over time, the quick fixes will lead you to buy more hardware and cost more.

1. Setup the MySQL Slow query log:

To setup the log, first uncomment the log_slow_queries and long_query_time parameters:

# Here you can see queries with especially long duration
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 5
#log-queries-not-using-indexes

This will set the slow query log to: /var/log/mysql/mysql-slow.log and will log any queries that take more than 5 seconds.

To start with, you should set the long_query_time to a higher value, so you don't get overwhelmed with too many slow queries. Later you can bring this down to 2 seconds or so.

And later you should also set the 'log-queries-not-using-indexes, to identify queries that will slow significantly as data load increases.

2. Restart mysql:

In ubuntu, simply run the command:

sudo /etc/init.d/mysql restart

This will restart the mysql server and start logging your slow queries.

3. Use your database, and keep checking for slow queries!

I hope this helps you in tuning your MySql database from the ground up.

LAMP: Why does a page take too long to open?

If you're wondering why some pages take ages to open on your LAMP site, consider checking all the queries that hit your mysql database when you open that page.
Sometimes it is not slow queries, but too many reasonably fast queries, that are slowing you down.

Note: I suggest you try using your slow query log first! Here is a post that will help you.

For checking the database for all queries, you should be the only user on the database.

1. To know *all* that's hitting your mysql database, simply uncomment this command in your my.cnf file (in Ubuntu, it is located at: /etc/mysql; you need to be root or run 'sudo /etc/mysql/my.cnf')

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.

log = /var/log/mysql/mysql.log

2. Once you've uncommented this line, restart mysql server:

In ubuntu: sudo /etc/init.d/mysql restart

3. Clear startup logs:

Most likely you don't want anything to do with the startup queries, so simply clear the log:

sudo echo " " > /var/log/mysql/mysql.log

4. Run your test: Go ahead and open your troublesome page in your browser.

5. As soon as the page is opened fully, copy out the log (here it copies to your home directory):

sudo cp /var/log/mysql/mysql.log ~/slowpage.log

If you have more troublesome pages, clear the log again and copy it to another location, for analysis.

6. Once you're done with getting your logs, comment the log setup in my.cnf once more, and restart mysql again (steps 1 and 2). This will keep your mysql running fast.

7. Now go ahead and analyze what queries are run when you go to a particular page.

All the best with your tuning efforts!

MySql Session

To find out what is going on in your MySql database, you can check the sessions that are presently running, with the command:

show processlist;

Thursday, November 5, 2009

Apache MySQL Performance dip

Separating the MySQL and Apache servers has slowed my site down, instead of speeding it up! Any suggestions?

I have hosted a LAMP website on a 2GHz AMD Opteron server running with 2GB of RAM.
As the data needs grew, the MySQL database seemed to need its own area, and could not serve the apache server fast enough.

So I split the MySQL database to run on its own 2GHz AMD Opteron/2GB RAM server - with a single-hop connection between the two of them at the data center.

But performance has dipped - any clicks take a much longer time to show a result from the database.
Any ideas how to get better performance from the separate Apache/MySQL server setup?

Thanks!

Saturday, October 31, 2009

Password Management Using htpasswd

In Linux, several components use non-system standard passwords.

Some of these are svn, and ftp (e.g. vsftpd).

Here are the basics to get you through setting and resetting these passwords:

For svn, refer this guide.

For vsftpd, refer this guide.

Basically, htpasswd can be used to create encrypted passwords for such modules. Here are some basics to htpasswd:

1. htpasswd uses a password encrypted file, using an SSL certificate.
You can create your own ssl certificate too. But for public/production environments, you'd want to get a certificate from a third party so users don't get a security alert.

2. The first time, use the -c flag to set a password, as follows. You'll need to do this as root (or prefix sudo in ubuntu)

htpasswd -c /etc/apache2/my_passwd.passwd username

3. Subsequently, use the -m flag to modify this file for adding/editing users:

htpasswd -m /etc/apache2/my_passwd.passwd username

Tuesday, October 6, 2009

LAMP Server Crash

If you did some optimizations by adding memory or RAM to processes, and the next morning, you find your server is no longer online: chances are it ran out of space and triggered the oom-killer.

oom-killer is a process that goes randomly killing processes to help the system survive.
After this occurs you should always restart as soon as possible.

Here is where you can find the logs in ubuntu, and how to identify if your system shutdown because of memory:

location of log: /var/log/messages (you will have to have super user permissions to view)
What you'll see: Just before the server shutdown or went erratic, this message:

apache2 invoked oom-killer

To get out of this situation, try to put more conservative memory settings on your applications, or add more RAM!

All the best!

Saturday, October 3, 2009

5 features Ubuntu needs to compete

Though I dearly love Ubuntu, and many of its features make windows seem obsolete...
But no matter how much I'd like to bid Windows adieu... there are reasons I need to still hang on to Windows - and no, its not just because of Word.

1. IE Compatibility
It's tiresome to go online and find a site (turbotax, netflix, web outlook...) which doesn't work well with firefox - especially linux based firefox. Sure there are workarounds and spoofs, but that's not a system working out of the box.

2. Multimedia experience
Ubuntu has some really neat multimedia features. Compiz Fusion adds mouthwatering features to your desktop - allowing you to rotate it like a cube for example.
But when I plug in an HDMI cable to connect a big screen tv, windows still supports the extension much better. My nVidia ubuntu driver asks for a restart, but doesn't do anything even after that.
DRM support is not present, which means the music industry leaves Linux users with no form of entertainment. No rhapsody to go.

3. Application Setup
It takes a whole lot of effort to get some apps to work. Skype, for example, needs a whole lot of tweaking and driver upgrades before it can work. For an app that needs speakers and a mic for basic functionality, setup should not be so difficult.
The soundcard drivers and programs, though feature rich, are not seamlessly integrated yet.

4. Much needed Applications
Some applications are altogether missing linux support. For example, no iTunes, no Microsoft Office are completely missing (sure the MS Office issue has been beaten to death - but really, at least give us an application that formats word documents accurately!) - and I'm not even talking hit games like GTA.
And yes, Wine can help get some things working. But any serious OS can't rely on workarounds.

5. Teething issues
Linux is lightweight, and efficient. Not!
That's what I always believed, until I wondered why my computer was so sluggish, when all I was doing was browsing the internet. Turns out, the Adobe Flash plugin on linux is a resource hog. Even if its just pandora playing, or Rhapsody online, the cpu usage peaks, and everything slows to a crawl.
Its a known issue. But is anyone fixing it?

Most of these are minor issues, and most of them have workarounds. But the question is, if Ubuntu linux is really going to compete mainstream, these need to be addressed. Linux has come a long way since its inception and text based output. But it still needs some polishing before it can kill the competition.

Saturday, August 22, 2009

Ubuntu 9.04 with Vista on Dell XPS m1330

Tired of Windows Vista slowness on my souped-up Dell XPS laptop, it was time I at least dual-booted into Linux. So I went ahead and downloaded a copy of Ubuntu to setup my computer.

To my delight, installing Ubuntu was a breeze! Though there was some back tracking I needed to do, this guide will help you get it setup in one shot. Once setup, everything on the laptop works - as if it is out of the box. This includes function keys, the mute/eject buttons and wireless.

You may face trouble with some applications though - be forewarned!

1. Make space for Ubuntu:

Shrink the volume size in Vista, so you can install Ubuntu

1b: Backup your data (Yes - ALWAYS play it safe :-) this will most likely go through just fine - but you want a backup for that hard disk crash you never planned for, anyway!

2. Write an Ubuntu Jaunty CD. You can try using Wubi - which is a windows Ubuntu installer. But I went the traditional way - this works just fine.

3. Boot into the CD (you might need to change boot sequence so your computer looks at the cdrom first)

4. Go through the partition steps and install Ubuntu into the free partition size you will see available.
Make sure you don't assign all of it to Ubuntu - otherwise you'll have trouble using Vista, or your install will fail.

5. Let ubuntu go ahead and install itself. I'd suggest giving it at least 20GB (I gave it over 100GB :-p

6. Now Ubuntu won't be able to connect to wireless just yet (I have the default Broadcom wireless card).
So connect to an LAN cable and go to System > Administration > Update System

7. Go ahead and install all the updates (there were about 140MB of updates when I ran it).

8. Restart your computer, and presto: you have ubuntu working!
You might need to take a moment to get the wireless setup - by entering your wireless settings.

Monday, July 6, 2009

Evernote - one app for all your notes?


A couple of weeks ago, I installed EverNote on my HTC Touch Diamond, and just couldn't get enough of it!
Evernote would take my handwritten or typed notes - sync them online - and make them available to me on my desktop too.
I could see that I'd not forget anything now! And best of all, their handwriting options - with colours and caligraphic pens had me spellbound.

Next thing - my HTC Touch Diamond screen cracks, and my friends convince me, its time for an iphone.

Well - the first app I installed was EverNote. But much to my disappointment, Evernote for the iphone does not allow handwriting or drawing. Although you can see your previously drawn/written notes.

Evernote guys - if you're listening, can you please give the same features on an iphone - which are present on windows mobile? You really do have a great app.
And while you're at it - please also include:
- auto scroll when you finish writing on a page
- please allow us to edit our notes on phones- really, you are cutting out a lot of functionality by forcing us to pay for this. I don't mind paying for more space - which I will eventually need; but forcing me to pay for a disabled feature is upsetting!

Friday, May 29, 2009

Twitter for Windows Mobile?


Want to use Twitter for Windows Mobile? PockeTwit wins hands down for a robust, full-featured Twitter application for Windows Mobile - with a great customizable interface, and gesture support.

Best of all - it's free!

Get it here

HTC Touch Diamond Sync Issue with Vista

When I upgraded my HTC Touch Diamond ROM (available here), I had trouble sync-ing with Windows Vista. I first managed to overcome it, but on reinstalling vista (for some other reasons), I faced the same issues again.

Here's a step-by-step guide to getting your HTC Touch Diamond to sync with Vista.

1. With your HTC Touch Diamond connected to your computer, go to Control Panel > Switch to Classic View > Device Manager



2. Right click PocketPC USB Sync and select 'Uninstall'

3. Once this is uninstalled, go to Control Panel > Programs and Features


4. You should see Windows Mobile Device Center. Uninstall it.



5. Now unplug you HTC Touch Diamond, and plug it back in. Everything should get reinstalled properly, and you should be good.
This is the screen you should see:



Go ahead and Set up your device.

Hope this helps!

Thursday, March 26, 2009

Mysql 5.1 downgrade to Mysql 5.0 on Ubuntu Intrepid Ibex

Hi couldn't understand why apt-get would always give the following error when trying to downgrade mysql on ubuntu:

Unpacking mysql-server-5.0 (from .../mysql-server-5.0_5.0.51a-3ubuntu5.4_amd64.deb) ...
Aborting downgrade from (at least) 5.1 to 5.0.
dpkg: error processing /var/cache/apt/archives/mysql-server-5.0_5.0.51a-3ubuntu5.4_amd64.deb (--unpack):
 subprocess pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/mysql-server-5.0_5.0.51a-3ubuntu5.4_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Finally figured it out. For downgrading, you must delete /var/lib/mysql directory.

After deleting that directory, I ran the following command, and had mysql setup and ready to go:

sudo apt-get -f install

Home Theater setup - HDMI

I have a Sony STR-K7100 HDMI Pass Through system. When I got a laptop with integrated HDMI, I was confident I could get a digitally tuned home theater setup. What I wanted to do, ofcourse, was play netflix movies on my computer, and route them to my projector through the Sony receiver.
This way, I will be able to get digital quality sound through my receiver, and digital quality picture on my projector!

Unfortunately, HDMI pass through means your receiver does NOT read the data passing through it, and only switches two HDMI input ports to one HDMI output port. Quite a waste I think.
So bottomline, I am using a 3.5mm jack to listen to music right now.

Monday, March 2, 2009

Blue Proton Micro Card Reader Review

I accidentally bought a micro MemoryStick card for my Sony DSC-H3. While there was a mini card adapter for the micro, which let it fit fine in the camera, but Laptop SD Card readers declined to accept the adapter.

Looking for the best way a laptop can read a micro card, I found the 'Blue Proton' Micro Card Reader at Amazon.

This thing costs less than $20 with a 4GB Micro memory card included. You can buy it individually for about $6 too.

Here's a closer look at the BlueProton.

Open the BlueProton, and you'll notice, it first looks like a regular USB Memory Stick, and if you pop open the hood, you'll notice it can take Micro SD, MMC Micro, and Micro Memory Stick (M2) cards.

Build Quality:

Unfortunately, build quality of this memory stick is sub-par. It is clunky, and looks cheap. But, once you get over that, you'll find it's able to do its job just fine. I do wish it wasn't so plasticy.


Setup:

Using the memory reader is a breeze. Pop it into the USB memory reader (I used XP and Vista 64 machines to test), and it immediately is recognized and shows up two drives.

Performance:

Given the USB 2.0 specification, it is not surprising that this device gives you great read/write speeds. Upload/download photos like it was an attached hard disk.

Final Take:

If you want a micro card reader, that's easy to take along, and has good performance, this is it. If you want something that looks and feels good too, then the blueproton is not up to the challenge.
Posted by Picasa

Saturday, February 28, 2009

HTC Touch Diamond Charging

I've often been asked whether the HTC Touch Diamond phone can be charged using mini-A-Type USB connectors (second from left below).

You see, the HTC Touch Diamond comes with a proprietary HTC ExtUSB Cord - which has a slightly different shape.


The answer is - yes - the HTC can use regular mini-A type plugs, and you don't need to spend a fortune buying their proprietary cables. The HTC is backward compatible, and the extra connectors that are in the ExtUSB, are for using audio/video - such as their headphones.

I have been charging my own HTC Touch Diamond using regular mini-A cables, without any problem. I would say, though, that battery performance is dismal in this phone, but it may be due to the fact that all iphone competitors give low battery life.

,

Thursday, February 26, 2009

ScribeFire for Blogging

I have been giving blogging a shot for sometime now, and since I have a few blogs I write to, I wondered what's a good way to get these blogs of mine out.

Turns out, there's a great firefox extension called  ScribeFire which can help you manage multiple blogs, and turn half of firefox into a full featured blogger.



This is the first blog I'm writing with ScribeFire, and it seems pretty cool.
I also managed to connect to a Google Apps based blog I run on my website for online shoppers, after doing some trial and error.

Looks interesting! I think Blogging is going to be even more pain-free now!

I wonder if these guys have a IE extension, or something for Windows Mobile - on which my phone runs.

Monday, February 16, 2009

Pausing and dialing on windows mobile

On a windows mobile phone - how do you dial a number with intermediate pauses? Simple - put in a comma (sorry, 'p' does NOT do the trick).
You can store your calling card numbers, bridge line numbers, doctors line etc -with the intermediate commas, and you'll be able to get through by just that contact.

For example, to dial 545-328-2571 option 1 and then pound sign, this is what you should dial on your phone: 545-328-2571,1,#
You can put in 2 commas (,,) in stead of 1, for good measure.

Tuesday, February 10, 2009

Hosted Blogger

I was keen on hosting a blog on a website I spend some time with - Likebucks - which is a site that hopes to be a gold standard for online shopping.

Well, I found Blogger easy to setup and host on the server. I created a blogger account with my website email id as the username - this keeps the blog seperate from regular blogs like this one, which I maintain.

Publishing to my host was easy once I gave the ftp information (I did need to make sure that I don't prefix the location with a '/').

The blog initially showed a header with 'Next Blog', the Blogger logo etc. This was an annoyance - I don't want anyone to know that my blog is hosted on blogger. I did find the setting for turning this off.

In blogger, go to templates, and under 'Edit HTML', set the 'Change the Blogger NavBar' to 'Off'. That's it!

Saturday, January 31, 2009

Activesync with Bluetooth

I've been trying to setup a bluetooth sync with my HTC Touch Diamond (Windows Mobile based) pda-phone, and my Windows Vista Home Premium (64-bit) OS.

Here's how I got it to work:

1. Go to settings > menu > bluetooth devices
2. Click 'Mode' and select 'Turn on Bluetooth' and 'Make this device visible to other devices'
3. Make sure bluetooth is turned on on your computer, and select 'Add new device' (I had to click on the bluetooth item in the system tray and then click 'Add' under 'Phones and Modems'
4. In the wizard that opens, select 'My device is set up and ready to be found' and click 'next'
5. Your computer will look for your pda/phone and should show it in the screen.
6. Select your phone, and click next. You will need to select a passkey - go ahead and set your own passkey
7. Your pda should now prompt you for the passkey - enter the same one.
8. Now your computer will show some installing happening, and finally should say 'your devices are ready to use'
9. On your pda, go to your bluetooth devices page, and check the services that are in place for the computer - make sure activesync service is selected
10. On your computer, under phones and modems, select your device and click 'properties'. In the services tab, activesync bluetooth service should be selected.
11. Now startup the 'sync center' on your computer.
12. from your pda, now open activesync, and select 'connect via bluetooth'
13. once the connection is established, select 'sync'.

That should do it!

Let me know how this turned out.