Implementing a Queue with PHP and MongoDB

By On · Add Comment

And finally, I’ve decided to release my code for a queue I implemented in PHP and MongoDB. For those who want to look at the code, here

So first, why use MongoDB as a queue?

  1. I already had a distributed cluster of mongodb nodes
  2. Single back end server app thingy to reduce complexity of app. Since I had already been using mongodb as my backend datastore, and mongo’s gridfs as my filesystem, it naturally makes sense for me to continue using mongodb for other parts of the system as well
  3. I already had knowledge of mongodb
  4. mongodb has a pretty neat feature that actually makes sense to run a queue on (more on that in a bit)

Components of the Queue

  1. A PHP daemon
    A daemon is basically a long running process that will do stuff, in this case, call on other process to do the job. In the daemon, one of the more important aspects of it would be memory management, to prevent it from spiraling out of control.
  2. PHP scripts to do your job
    The php daemon will essentially call on these scripts to do the job.
  3. A mongodb capped collection
    This is by far in my opinion one of the most amazing aspects of mongodb. They have a collection(table in rdbms) called capped collections that you can actually tail. Yea, the tail -f command in unix. This means that you can literally have a persistent connection with the database, and it’ll just give you the new documents that have been added to the collection. Note that capped collections have a cap on it’s size, hence it’s name, and you cannot delete the entries. It’ll overwrite the earliest entry in the collection when it is full.

How it works

Basically, there’s 2 loops to keep the queue process running, the outer loop to recover from db going down, and the inner loop to basically run the queue process. It uses the tail function on capped collections to check for new additions to the table, and will then process each new job that comes in by running the exec function, which basically forks off a new process. This was done to limit memory usage since it’s a long running php process, and tests done have put it’s memory consumption at 13MB RAM.

If you’re using Codeigniter and is considering having a queue that’s implemented in MongoDB, check it out here!

Thoughts on the Lenovo Twist

By On · Add Comment

So I finally got the Twist, after waiting for it for over a month.

Just a couple of thoughts on it first.

  1. Microsoft has really changed everything
    Really. Putting touch into the OS was just brilliant. It makes things easier to do, and generally improve on the happiness level of me using the OS.
  2. This is a damn good machine
    With 8GB RAM, and a i7 processor that goes up to 3GHz, it is pretty powerful. So much that I can run 2 Linux VM’s (1 CentOS and 1 Ubuntu[no choice, for a module assignment])with the host OS and switch between them without facing much lag.
  3. The keyboard is awesome :D
    This new layout by Lenovo really makes typing alot easier.

Users might face problems with the laptop though. For example, when you install firewall’s such as Comodo, windows 8 will have weird behaviour, not sure if it’s just on the twist. Charms will lag really badly, and can only be fixed by logging out and logging back in. This behaviour happens only after you restart the machine.

The other gripe I had with the laptop was the intel Power Saving Technology. When the brightness is dimmed to minimum levels, this ‘technology’ kicks in, and will adjust your brightness based on the colours of the page, which will at times render your machine unusable. After poking around for a while, finally narrowed it down to this ‘technology’, and not Window’s adaptive brightness.

All in all, I have to say, this is the best laptop I’ve owned so far, and I have to say, better than my previous macbook pro, especially in the OS’s memory handling. Microsoft has really made a breakthrough this time round.

Setting up OpenVPN on a CentOS server and Windows8

By On · Add Comment

So I’m bored of using SSH Tunneling. Plus the fact that when I open too many connections, it sometimes dies. So I decided today morning that I will set up a VPN service that I can use to connect securely to the net that is actually designed for this (plus the fact it made the forwarding global helped too).

So my server is technically a OpenVZ server hosted by Alienlayer, with 512MB RAM and 2 2Ghz cores over in Las Vegas. Expected installation to be a breeze, a couple of yum install commands and I’d be done. Turned out I was wrong.

The initial setup was easy, following the instructions over here for CentOS and here for Windows. The problem that stumped me was the part when I couldn’t update the iptables. Searched the net for hours to no end. Finally, with the help of Olipro on StackExchange, realised that the problem was that the MASQUERADE module didn’t exist, and as far as I know, not virtualised yet. So couldn’t use it.

For people who are intending to set up a VPN server in future, here’s a link that you might find useful.