And finally, I’ve decided to release my code for a queue I implemented in PHP and MongoDB. For those who want to look at the code, here
So first, why use MongoDB as a queue?
- I already had a distributed cluster of mongodb nodes
- Single back end server app thingy to reduce complexity of app. Since I had already been using mongodb as my backend datastore, and mongo’s gridfs as my filesystem, it naturally makes sense for me to continue using mongodb for other parts of the system as well
- I already had knowledge of mongodb
- mongodb has a pretty neat feature that actually makes sense to run a queue on (more on that in a bit)
Components of the Queue
- A PHP daemon
A daemon is basically a long running process that will do stuff, in this case, call on other process to do the job. In the daemon, one of the more important aspects of it would be memory management, to prevent it from spiraling out of control.
- PHP scripts to do your job
The php daemon will essentially call on these scripts to do the job.
- A mongodb capped collection
This is by far in my opinion one of the most amazing aspects of mongodb. They have a collection(table in rdbms) called capped collections that you can actually tail. Yea, the
tail -fcommand in unix. This means that you can literally have a persistent connection with the database, and it’ll just give you the new documents that have been added to the collection. Note that capped collections have a cap on it’s size, hence it’s name, and you cannot delete the entries. It’ll overwrite the earliest entry in the collection when it is full.
How it works
Basically, there’s 2 loops to keep the queue process running, the outer loop to recover from db going down, and the inner loop to basically run the queue process. It uses the tail function on capped collections to check for new additions to the table, and will then process each new job that comes in by running the
exec function, which basically forks off a new process. This was done to limit memory usage since it’s a long running php process, and tests done have put it’s memory consumption at 13MB RAM.
If you’re using Codeigniter and is considering having a queue that’s implemented in MongoDB, check it out here!