I have a couple of servers with different Linux distributions (Ubuntu/Fedora) on my hands, which normally work continuously well. However, from time to time they reboot as they are hosted in a shared farm of virtual servers. It was in the first of those reboots when we noticed that neither Apache, neither MySQL servers were automatically launched on start-up, which should be the default configuration in an environment like ours.
So, I’d like to add another new trick to my the knowledge pool, which I expect to be useful for others as well: how to configure scripts and services to run after the system boots?
I’ve already talked about how I solved the problem of managing huge amounts of data in my last post. Now, I’m going to explain how to implement one of the solutions found in order to comfortably face this continuously increasing avalanche of information.
Let’s imagine that, as I explained before, I have separated data input into tables coded by a station ID. All of them are stored in the same database for maintenance simplicity sake. So, we have the ‘database’ in which there are hundreds of tables called ‘Station_XYZ’. Every table has the same structure: to simplify, SensorID, Time (UNIX), Value. All right then, time to make cubes!
Continue reading Implementing a database cubes system on MySQL
I’m currently working on a database which stores, literally, a million of new data registers every day. They come from several hundred weather stations spread along the world, and need to be processed on real time. That’s a huge amount to be managed with a simple – but powerful – MySQL server, so I’ve had to design and implement a few tricks in order to succeed.
In this post I’d like to talk about the main three techniques we’ve used to engage our high performance database, and the reasons to choose these solutions.
First of all, MySQL innodb tables start degrading at some million registers (7-10M on my current server, but not tested further), so the first though I had was: we need to divide if we want to conquer. Then, after some thinking about, I decided to split the big table into small chunks, and use a table for each station. That made the whole problem more affordable, and tables smaller enough not to become scary: the new problem was how to store about 5 M registers per year per table in our database.
Of course, some logic in both the application controller and the data input server was needed to correctly place data registers. Additionally, tables get often fragmented so some cron jobs were needed to keep the database running smoothly. But there was still the problem of data growing. As long as registers were separated depending on the station, the scalability problem was solved: more tables can be added, even more servers with ranges of stations and a hash function to calculate what server is a given station hosted in. Once again were we talking about several million registers per year per table, and with a standart MySQL 5.1 server (the main problem with an start-up company tends to be funding, so we didn’t have anything better).
Continue reading Managing big data sets with MySQL
In this post I’m going to cover a particular solution to a given problem we’ve faced up in our last project. Let’s set the background: there is a complex web application under development on a shared virtual machine, installed on a subdomain of a public network that will change when going to production. We have also a commercial website hosted in a shared server under a domain that should not change, since it is our first step out of this virtual farm.
For this project, we require to integrate some cameras which send pictures via GPRS connections to the server, and we want to provide some user authentication to avoid having anybody posting whatever to this web service. So, we have developed a controller in Code Igniter which gets a camera ID / password from POST, and also the bytes for the image, and proceeds to do some filtering. For instance, camera ID and password should match with the values previously registered in the database, and the picture should be MIME image/jpeg, with a limit of 2MBs. Up to now, nothing special, tested and working with POSTS from the within the website.
The problem came when we found out that cameras only allow to send their information to a root domain, to a particular file, let’s say upload.php. So, we would only need to put this file in the root path and it’s done. Unfortunately, doing so in the application virtual server would result on flashing again all the cameras once we move to a new server and set up a final domain name. We might be able to install this receiver in the commercial website root path, but then we would require to send the message back to the application server in any way, so it is not the best solution we want to solve this issue.
‘Just add a domain that points to this subdomain and you are done’ would you say. Well, yes, that would have done the trick. Unfortunately, we cannot host this domain in the machine since we have no administrator rights on the servers farm. Nor can we just use this external domain because it produces an HTTP redirection which, to make things worse, is not supported by the camera. We’ve even tried with cloaking the url of the receiver with an external subdomain, with no success – since cloaking wraps the real content inside an iframe and it breaks the POST message.
What a headache! it seemed there was no way to implement the system with the full functionality we wanted…
Continue reading Tunneling a multipart POST message through PHP and CURL