Jorge Albaladejo

ActionScript 3 and cross-domain problem

Jorge Albaladejo — Mon, 06 Sep 2010 19:37:27 +0000

One of the most typical problems when coding AS3 to, let’s say, read data from an external XML source, is the cross-domain problem. According to security policies, not an url outside the domain where the AS application is being executed can be read, and different subdomains (such as www.yourdomain.com and yourdomain.com) are considered as different ones.

If the external resources you are trying to use come from your own application in another domain / server that you manage, then it can be fixed by adding a cross-domain policy file. This file must be defined on the content source server, and your AS application has to explicitly declare this server as allowed.

^?View Code ACTIONSCRIPT

Security.allowDomain("domainone.com");
Security.allowDomain("domaintwo.com");

But if this is not the case and the content source is on a different remote server like, for instance, when implementing an RSS news reader which grabs content from sources like online news sites, then this problem can be worked out with a PHP proxy file.

^?View Code PHP

$url = $_GET['url'];
readfile($url);

Since PHP is not affected by this policy, all the RSS feeds would be read through this small script, by calling it like ‘proxy.php?url=http://rss.thedomain.com’. Being this file exactly on the same domain than the AS application, the cross-domain problem is solved.

However, I have also found an odd behavior regarding this proxy PHP file: it works perfectly when opening the SWF compiled application directly on the browser (http://yourdomain.com/app.swf), but not when it is loaded as a part of the HTML code (embedded into an HTML page like http://yourdomain.com/app.html). The SWF application just doesn’t find the file (but it is there!), despite it reads without any problem a test text file on the same folder. So it seems as if the question mark ‘?’ changed the way the SWF application looks for files or executes url addresses. I don’t have a reason for this yet, but I guess that it is related to the way the Flash plugin manages the security, by using different sandboxes for direct and embedded executions of the application.

The solution for this second collateral problem is passing the domain name as a parameter, so that the full path to the proxy file can be constructed inside the AS application.

^?View Code HTML4STRICT

<html>
	<head>
		<style type="text/css">
			...
		</style>
		<script type="text/javascript">
			...
		</script>
	</head>
	<body>
		<div id="flash">
			= getBasePath() ?>
			<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/" width="960" height="610">
		    	<param name="allowScriptAccess" value="always" />
		    	<param name="movie" value="main.swf?basepath=" />
		    	<param name="quality" value="high" />
		    	<param name="bgcolor" value="#ffffff" />
				<param name="basepath" value="" />
				src="main.swf?basepath=" 
				    quality=high bgcolor=#FFFFFF width="960" height="610" name="main"
				    type="application/x-shockwave-flash" 
				    pluginspace="http://www.macromedia.com/go/getflashplayer">
				</embed>
			</object>
		</div>
	</body>
</html>

^?View Code PHP


function getBasePath()
{
	$path = 'http://'.$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'];
	$aPath = split('/',$path);
	if ( eregi ( '\.php', $aPath[sizeof($aPath)-1] ) )
	{
		$aPath[sizeof($aPath)-1] = '';
	}
	$path = join('/',$aPath);
	return $path;
}
?>

^?View Code ACTIONSCRIPT

var proxyPath:String = this.loaderInfo.parameters.basepath + 'proxy.php'; // proxy for cross-domain problems

In any case, these problems only arise in online mode, I have not had any issue while working from my development suite, so it is when moving to a production environment when the pain in the ass can show out. Finally, this is nothing new, but handy and interesting not to forget.

Basic web mapper

Jorge Albaladejo — Sat, 21 Aug 2010 08:00:15 +0000

Sometimes it is useful to have an automated tool to get the full web map of your site. Perhaps not your own web site, since you have already implemented some kind of automatic generation and notification to Google (have not yet?), but a client’s one.

There are a few tools to map an external web site, I tried some in my particular case. They were just adware, or demos, or they obscured the links in the final report… Yeah, of course, sometimes a $30 license is worth it, but you might not want to acquire a new piece of proprietary software every time you need a new feature, might you?

So I decided to write it myself in PHP, not for the money, but for the fun

The architecture

First of all, a bit of planning the application:

Web mapper work flow diagram

Web mapper work flow diagram (Visio).

Let’s take a look at this work flow diagram. First of all, the mapper reads the entry url address in search of links. The blue color represents the entry and exit points of the application.

Then it reads the HTML code, parses it in search for links, and if found, it runs a loop on each one of them. The green color means the actions taken within the loop. Basically, this involves saving the link into the database, and, if the address is not external (that means, it points to a page within the same domain name), then reading it and starting the process again for this new url.

This practice is commonly known as ‘recursion’, and is represented in orange. When a link within a page is followed, the application has just entered a new recursion level, then reads the HTML code and seeks for links, starting again a new loop. Think of it like a set of Russian dolls: every link of a page contains a page with more links, so there we start a new entire loop from an iteration of the previous one, and so on.

The recursion will end when a page has no links, or all its links are external. Then the application has found a leaf page, nothing to do deeper than that, and can return one level up. When all the leaf pages are read, this algorithm will return to the main entry level, when, after going through all links found, will finish.

This, depending on the web site structure, can take a few minutes to complete all the mapping, so that is better to use a fast and reliable programming language. In my case, I have used PHP for simplicity’s sake, since is the language I am using daily, but on a test server where I can increase execution time. For more intensive uses of this script, I would recommend to have it translated into another language, or to run it from console with the PHP interpreter, not from the web server itself.

In addition, this architecture is really basic, that means that a lot of things can be improved. For instance, a check has been added to avoid following links twice. This, however, is not so easy to implement, since inner links can follow different conventions (“www.domain.com”, “domain.com”, “sub.domain.com”, “/”, “./”). Thus, the code presented below should be used for academic and learning purposes, I can not guarantee that it is going to work perfectly at all.

The code

Let’s present the code now. It has been written for the Code Igniter framework, but can be easily ported to no matter what architecture or language. A few conventions:

It is based on the class ‘spider’, whose main method ‘crawl()’ launches the initial entry point, and the inner and protected method ‘_crawl()’ implements the recursion.
All inner and protected methods are preceded by an underscore (_).
A few configuration attributes allow a sandbox definition, limiting the recursion levels and the domains to be crawled.
As said before, it is based on Code Igniter, but in this case that is only to provide an easy interface with the database. You can just replace all instances of $this->ci->db with your own MySQL connection class.

Using the class is very simple:

^?View Code PHP

$spider = new spider();
 
$spider->crawl( array(  'entry'		=> 'http://jorgealbaladejo.com',
			'domains'	=> array(	'jorgealbaladejo.com' ,
							'www.jorgealbaladejo.com',
							'jorgealbaladejo.ch',
							'jorgealbaladejo.es'	) 
			) );
 
$spider->printResults();

And finally, the core class code. It is commented so it may be easy to understand, eventually. However, comments, improvements and doubts are welcome, so if you find I am doing something not clearly enough, or you would have just done it in another way, please share your comments!

^?View Code PHP

/**
 * Class spider
 * Creates a spider which crawls the internet
 * 
 */
class spider
{
	/**
	 * @var Code Igniter object for database access
	 * 
	 */
	var $ci;
 
	/**
	 * @var maximum recursivity depth
	 * 
	 */
	var $max_depth 	= 100;
 
	/**
	 * @var default domains to restrict crawling
	 * 
	 */
	var $domains 	= array('jorgealbaladejo.com','vwww.jorgealbaladejo.com');
 
	/**
	 * @var indentation level
	 * 
	 */
	var $indent 	= 20;
 
	/**
	 * @var internal links array
	 * 
	 */
	var $links 	= array();
 
	/**
	 * Constructor
	 * 
	 */
	public function spider()
	{
		$this->ci =& get_instance();	
 
		// load links in database to internal list
		$this->_loadLinks();	
	}
 
	/**
	 * Entry point for recursive function crawl
	 * 
	 * @param object 	$params
	 * @param array 	$params['domains']
	 * @param string	$params['entry']
	 * 
	 * @return boolean
	 *  
	 */
	public function crawl($params)
	{
		if (isset($params['domains']))
		{
			$this->domains = $params['domains'];	
		}
 
		if (isset($params['entry']))
		{
			$this->_crawl($params['entry']);	
 
			return true;
		}
 
		return false;		
	}
 
	/** 
	 * Recursive crawling function
	 * 
	 * @param string	$page
	 * @param int 		$parent [optional]
	 * @param int 		$depth of current page [optional]
	 * 
	 * @return void
	 * 
	 */
	protected function _crawl($page, $parent = 0, $depth = 0)
	{
		// vars
		$doc 	= '';
		$title 	= '';
		$desc  	= '';
		$out 	= NULL;
		$links 	= array();
		$link 	= '';
		//
 
		// correct url
		$page 	= $this->_buildUrl($page,$parent);
 
		// avoid reading the same url twice
		if (array_key_exists($page,$this->links))
		{
			return false;
		}
 
		// only read info for inner pages
		if (in_array($this->_getDomain($page),$this->domains))
		{			
			// read page content
			$doc 	= $this->_getHTTPRequest($page);
 
			// get page meta data
			list($title, $desc) = $this->_analyzePage($doc,$page);
		}
 
		// log into inner array
		$this->links[$page] = array(	'url' 		=> $page ,
						'title'		=> $title ,
						'description' 	=> $desc );
 
		// write into database		
		$this->ci->db->query('INSERT INTO web.map VALUES("",' .
					'"' . $page . '",' .
					'"' . $title . '",' .
					'"' . $desc . '",' .
					'"' . $parent . '") ');
 
		$parent = $this->ci->db->insert_id();	
 
		$this->_printLine($title,$desc,$page,$depth);	
 
 
		// avoid recursion for external domains
		if (!in_array($this->_getDomain($page),$this->domains))
		{
			return;
		}
 
		// now get links and launch recursively
		$links = $this->_getInnerLinks($doc);
 
		foreach($links as $link) 
		{ 
			if ($depth < $this->max_depth)
			{
				$this->_crawl($link,$parent,$depth+1);	
			}				
		}
 
		return;		
	}
 
	/**
	 * Prints the current links array
	 * 
	 *  
	 */
	public function printResults()
	{
		ksort($this->links);
 
		foreach ($this->links AS $link)
		{
			$this->_printLine($link['title'],$link['description'],$link['url'],0);
		}
	}
 
	/**
	 * Preloads the existent links in database 
	 * 
	 */
	private function _loadLinks()
	{
		$result = $this->ci->db->query('SELECT url,title,description FROM web.map ORDER BY url ASC');
 
		if ($result->num_rows())
		{
			$results = $result->result();
 
			foreach($results as $r)
			{
				$this->links[$r->url] = array(	'url' 		=> $r->url ,
								'title' 	=> $r->title ,
								'description' 	=> $r->description );		
			}
		}		 
	}
 
	/**
	 * Reads a document for html links
	 * 
	 * @param string $doc [optional]
	 * 
	 * @return array of links
	 *  
	 */
	private function _getInnerLinks($doc = '')
	{
		$return = array();
		$regexp = "]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
		$match  = null;
 
		if(preg_match_all("/$regexp/siU", $doc, $matches, PREG_SET_ORDER)) 
		{ 	
			foreach($matches as $match) 
			{ 
				$return[] = $match[2];
			} 
		}
 
		return $return;
	}
 
	/**
	 * Prints a line for the current url
	 * 
	 * @param int
	 * 
	 */
	private function _printLine($title = '', $desc = '', $page = '', $depth = 0)
	{
		// do not indent at this time
		$depth = 0;
		//
 
		echo '' .
				'
. ($depth*$this->indent) . 'px">' . 
					stripslashes($title) . 
				'
' . 
				'. ($depth*$this->indent) . 'px">' .
					substr(stripslashes($desc),0,100) .
				'' . 
				'. ($depth*$this->indent) . 'px">' . 
					$page . 
				'
' .
			'
';	
	}	
 
	/**
	 * Analyzes a page to get meta information
	 * 
	 * @param string $doc(cument)
	 * @param string $page
	 * 
	 * @return list($title,$description);
	 *  
	 */
	private function _analyzePage($doc, $page)
	{
		//
		$title 	= '';
		$desc 	= '';
		//
 
		if (eregi ("(.*)", $doc, $out)) 
		{
			$title = addslashes($out[1]);	
 
			if (strlen($out[1]))
			{
				$title = substr($title,0,50);	
			}			
		}
 
		$out 	= @get_meta_tags($page);
 
		if (isset($out['description']))
		{
			$desc 	= addslashes($out['description']);
		}
 
		return array($title,$desc);
	}
 
	/**
	 * Completes a page link with the domain if needed
	 * 
	 * @param string $page
	 * @param int 	 $parent
	 * 
	 * @return string corrected url
	 *  
	 */
	private function _buildUrl($page,$parent)
	{
		// prepare url if relative
		if (!eregi('http://',$page))
		{
			// root relative path
			if (strpos($page,'/') == 0)
			{
				$page = 'http://' . $this->domains[0] . $page;	
			}
			// page relative path
			else
			{
				if (strpos($page,'mailto:')<0 && strpos($page,'javascript:')<0)
				{
					$page = $this->_getUrlByID($parent) . '/' . $page;	
				}				
			}			
		}
 
		// trim final slash
		$page = trim($page,'/');
 
		return $page;
	}
 
	/**
	 * Gets url for a given ID
	 * 
	 * @param int url id
	 * 
	 * @return string the url
	 *  
	 */
	private function _getUrlByID($id)
	{
		$results = $this->ci->db->query('SELECT url FROM web.map WHERE ID = "' . $id . '"');
 
		if ($results->num_rows())
		{
			$result = $results->result();
 
			return $result[0]->url;
		}
 
		return '';
	}
 
	/**
	 * Gets domain name from a URL
	 * 
	 * @param string url
	 * 
	 * @return string domain name
	 * 
	 */
	private function _getDomain($url = '')
	{
		// get host name from URL
		preg_match('@^(?:http://)?([^/]+)@i', $url, $matches);
 
		if (isset($matches[1]))
		{
			return $matches[1];	
		}
		else
		{
			return NULL;
		}
 
	}
 
 
	/**
	 * HTTP helper function.

	 * Loads an http request and returns result.
	 *
	 * @param string $url to request
	 *
	 * @return string the result
	 *
	 */
	private function _getHTTPRequest($url = '')
	{
		// vars
		$html = '';
		$old  = '';
		$file = '';	
		//
 
		//configuration
		$timeout 			= 15;
		//
 
		// execution
		try
		{
 
  			$file = @fopen($url, 'rb');
 
			if ($file)
			{
				$html = stream_get_contents($file);
				fclose($file);
			}						
 
		}
		catch(Exception $e)
		{
			$html = 'error';
		}
		//
 
		return ($html);
	}
}

Scrum / agile development

Jorge Albaladejo — Thu, 29 Jul 2010 19:53:25 +0000

Some resources and useful information about this development technique:

About Scrum (Spanish) http://www.proyectosagiles.org/
Agile Software Development http://www.agile-software-development.com/
Good Agile Project Management software, community / free version available http://www.rallydev.com/
Scrum project management software, for Windows (Desktop) http://www.scrumdesk.com
MoSCoW method to prioritize tasks http://en.wikipedia.org/wiki/MoSCoW_Method

Basic concepts:

Sprints (iterations) last from 2 to 4 weeks, and they are released so that the client can test and provide feedback
- Every iteration is divided into small tasks planned to be achieved in less than 2 days work
The specifications for a started sprint should not change, the client has the responsibility to define carefully the priorities; they are defined as user stories
There is a project owner which interacts with the client, and a facilitator which helps the development team to achieve the goals
From 5 to 9 people in every development team
Daily meetings (about 15 min) with all the team to discuss about:
- The work done the day before
- The problems faced when doing it
- The work to be done the same day
- Only ‘pigs’ can talk, ‘chickens’ can listen, though
‘Pigs’ are: developers, project owner and project facilitator; chickens are: clients, users, marketing people
Finish every new feature (sprint) at 100%; it is better off having 80% features at 100% than 100% features at 80%

MySQL clean reinstall

Jorge Albaladejo — Tue, 27 Jul 2010 19:08:14 +0000

To perform a clean reinstall of the MySQL server (which regenerates start scripts, databases and config files) use:

^?View Code BASH

sudo apt-get install mysql-serverli>
sudo apt-get remove --purge mysql-serverli>
sudo apt-get autoremove mysql-serverli>
sudo apt-get install mysql-serverli>

Some times is useful!

FTP server on Ubuntu

Jorge Albaladejo — Tue, 06 Jul 2010 20:12:11 +0000

I write this post more as a reminder for myself than as a generic tutorial. Anyway, it is about a precise FTP server configuration on Ubuntu that might or might not be useful for other people. It is just useful in the environment I am using (web server, just one Unix user, some FTP accounts). Here we go:

The server that I’ve used in several cases, and which I prefer just because it is simple enough to have it running quickly, is VSFTP (Very Simple FTP). It works in integration with the machine’s users so that we only need to add users to the server in order to have them available for the FTP daemon. Once the server is installed, a few configurations must be done. In file /etc/vsftpd.conf:

^?View Code BASH

#
# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# You may restrict local users to their home directories.  See the FAQ for
# the possible risks in this before using chroot_local_user or
# chroot_list_enable below.
chroot_local_user=YES

So, let’s take a look at this configuration. First of all, it makes no sense having anonymous users on this machine. Then, we allow local users to connect to the FTP server, and give them write permissions. Finally, they are going to be confined to their local home directory, to make the server safe enough. But, what’s the point in that? After all, this is a web server, so we want the users to be able to update some web sites.

The solution to this is to mount the specific folders of the Apache webroot directory into the user’s home page, depending on the right access it should have. A symbolic link is just not enough, since the system prevents FTP clients to follow them as they are outside the home root. A command line like:

^?View Code BASH

mount --bind /home/username/mountpoint/ /var/www/user_website/

would do the trick.

A potential security risk is related to the use of this server. As long as FTP clients have to have a user in the machine, they potentially can log in into the server and execute code. First of all, one must ensure that these users do have just the essential access rights, but in addition, we can go further on protecting the server. In my case, the web server is a virtual machine in a cloud of computers, so there is not possible local access. Then, about SSH access, it is enough to specify the users that can log into the machine, so that we can have no matter how many users – allowed to access their home page through FTP – which cannot just execute anything on the machine. To do this, let’s edit /etc/ssh/sshd_config:

^?View Code BASH

AllowUsers user1 user2 user3 ...

With this setting, only the selected users can log in the machine via SSH access; the others are just confined to their home folders via the FTP server. And that’s all, just remember to restart both FTP and SSH services.

It is not the best FTP server I can have installed, but it is simple enough

Start a service or execute a script on boot on Ubuntu/Fedora

Jorge Albaladejo — Thu, 27 May 2010 19:25:14 +0000

I have a couple of servers with different Linux distributions (Ubuntu/Fedora) on my hands, which normally work continuously well. However, from time to time they reboot as they are hosted in a shared farm of virtual servers. It was in the first of those reboots when we noticed that neither Apache, neither MySQL servers were automatically launched on start-up, which should be the default configuration in an environment like ours.

So, I’d like to add another new trick to my the knowledge pool, which I expect to be useful for others as well: how to configure scripts and services to run after the system boots?

Run levels

First of all, we need to know something about run levels in Linux systems. The numbers and their meaning differ from one system to another, but their main purpose remains:

Single-user mode
Multi-user mode
Multi-user mode with networking
Reserved for special purposes
Adds display manager to level 3

Thus, the ‘normal’ run level we should use in a server environment is 3, which starts all the services, but does not care about X11 – which makes sense. This can be a first source of error if the default run level is not correctly selected, or it is tweaked to mimic another one which is the one that should be used instead. Anyway, to change this parameter we can go through: /etc/inittab and edit the line

^?View Code BASH

id:3:initdefault:

After verifying this setting, we can go for the unit configurations for every service on every level.

Services, daemons and servers

We can use the tool sysv-rc-conf under Ubuntu to easily mark or unmark the installed services that should run automatically with every run level. Normally, these configurations are correct, but it is no harm to double check, right? Under Fedora we can use chkconfig for the same purpose.

Custom scripts and applications

There is still another way to configure the services and software that we want to be auto-executed on system start. Under /etc/rcX.d (where X is the run level) there are some simbolic links to all the commands, scripts and applications that should run with every run level. Then, an advanced way to configure the servers that we want to launch with the machine, or even better, the custom scripts that we want to execute with a wide range of purposes (maintenance, a specific folder mount, whatever), is to create a symbolic link under one of these folders.

To create the link, the naming Sxx or Syy must be respected, where xx and yy are numbers which complement 100 (like K36mysqld and S64mysqld). The number is the order in which the command will be executed, and the letter (well ‘S’, well ‘K’) tells the system whether is a Start or a Kill action, that’s to say, if the command specified should be executed when entering or when exiting the selected run level.

As an example:

^?View Code TEXT

/etc/rc3.d/S99custom_mount -> /etc/init.d/custom_mount.sh

Alternatives

There are other possibilities to get the same result, like editing /etc/rc.local file, where we can write the commands we want to execute when the system has finished booting. Taking a look to the run level bootstrap default configuration, we may find:

^?View Code BASH

/etc/rc3.d/S99rc.local -> ../init.d/rc.local

Which means that the commands stated in this file will be executed at the very end of the system load in run level 3, that’s to say, after MySQL, Apache and the rest of servers and services. Despite the simplicity of this alternative, I prefer the one exposed above, since it gives a full control on what is being executed, where, and when. And, of course, it gives a better comprehension of Linux environments, which is one of the primary targets of this article.

Did you enjoy it? Leave a comment!

Implementing a database cubes system on MySQL

Jorge Albaladejo — Tue, 11 May 2010 09:22:01 +0000

I’ve already talked about how I solved the problem of managing huge amounts of data in my last post. Now, I’m going to explain how to implement one of the solutions found in order to comfortably face this continuously increasing avalanche of information.

Let’s imagine that, as I explained before, I have separated data input into tables coded by a station ID. All of them are stored in the same database for maintenance simplicity sake. So, we have the ‘database’ in which there are hundreds of tables called ‘Station_XYZ’. Every table has the same structure: to simplify, SensorID, Time (UNIX), Value. All right then, time to make cubes!

First of all, I define a MySQL stored procedure to extract maximum, minimum and average values from these tables and to save them in a second ‘database_cubes_1h’, named so because this process will run every hour. Also, there is a table Stations in ‘database_main’ (a third database to keep other application’s configurations), where all the stations ever installed are registered. We will use this table to know if a station exists and therefore its related table in database. A first draft would look like this:

^?View Code MYSQL

-- -----------------------------------------------------
-- Procedure `fill_cubes_1h`
-- -----------------------------------------------------
 
DROP PROCEDURE IF EXISTS fill_cubes_1h;
 
DELIMITER //
 
CREATE PROCEDURE fill_cubes_1h($StationID INT)
BEGIN
  IF Station_ID IN (SELECT StationID FROM `database`.`Stations`)
  THEN
    SET @strSQL = CONCAT('
      INSERT INTO `database_cubes_1h`.`Station_',$StationID,'`
      SELECT
        SensorID,
        UNIX_TIMESTAMP(DATE_FORMAT(FROM_UNIXTIME(Time),"%Y-%m-%d %H:00:00")) AS Hour,
        AVG(Value) AS Value,
        MAX(Value) AS ValueMax,
        MIN(Value) AS ValueMin
      FROM `database`.`Station_',$StationID,'`
      WHERE
        Time >= UNIX_TIMESTAMP(DATE_FORMAT(NOW() - INTERVAL 1 HOUR,"%Y-%m-%d %H:00:00"))
      GROUP BY
        HOUR, SensorID
      ORDER BY
        HOUR ASC
    ');
 
    PREPARE statement FROM @strSQL;
    EXECUTE statement ;
    DEALLOCATE PREPARE statement ;
 
  ELSE
    SET @foo = "";
 
  END IF;
 
END;//
 
DELIMITER ;

Basically, it composes a query to extract averages from one table and insert into another at once. This query is run as a prepared statement so that we can reuse the function for all the stations in our database, as long as tables are always named as Station_XYZ. But, what does exactly it do?

Use a statement like ‘INSERT INTO table SELECT * FROM another_table. It will copy data automatically if the number and format of columns of the SELECT output and the INSERT INTO input are the same.
As Time is stored in UNIX EPOCH time, it is converted to ISO time, then only the hour is extracted, then it is converted back to UNIX. The result of this process is be able to group the result set by hour, getting rid off minutes and seconds. “UNIX_TIMESTAMP(DATE_FORMAT(FROM_UNIXTIME(Time),”%Y-%m-%d %H:00:00″)) AS Hour”
A WHERE condition filters the results for the last hour: ‘Time >= UNIX_TIMESTAMP(DATE_FORMAT(NOW() – INTERVAL 1 HOUR,”%Y-%m-%d %H:00:00″))’.
Finally, the whole result set is grouped by Time, to be able to calculate averages, maximums and minimums. This statement does the grouping by respecting the different sensors that might have sent data on the same hourly interval “GROUP BY HOUR, SensorID” and the following ones perform calculations “AVG(Value) AS Value, MAX(Value) AS ValueMax, MIN(Value) AS ValueMin”.

Up to now we seem to have resolved the performance problem. Hourly cubes can be constructed and we only would need to add a sort of cron job. But, it is not so easy… I haven’t mentioned yet that data is not received synchronously. That means, within a time frame of three or four hours we could receive the data from a lazy or out of range station. That may be problematic, so…

I’ve changed my stored procedure to do as follows:

Calculate the whole last day in hourly tranches, while keep running every hour. “WHERE Time >= UNIX_TIMESTAMP(DATE_FORMAT(NOW() – INTERVAL 1 DAY,”%Y-%m-%d %H:00:00″)) AND Time <= UNIX_TIMESTAMP(DATE_FORMAT(NOW() – INTERVAL 1 HOUR,”%Y-%m-%d %H:59:59″))”
Modify the query to allow update in case that a hourly cube was already calculated. “ON DUPLICATE KEY UPDATE `Value` = VALUES(`Value`)”

Please note that in order to get it correctly running, there should be an effective way to detect a duplicate key. In my case, I’ve used all fields but the value as primary keys, instead of defining a new artificial key field. Thus, SensorID and Time are primary keys, so there should never be more than one value for each combination of both. Doing so, MySQL detects duplicate and updates the third column, value, by throwing no errors. The whole stored procedure would look like this:

^?View Code MYSQL

-- -----------------------------------------------------
-- Procedure `fill_cubes_1h`
-- -----------------------------------------------------
 
DROP PROCEDURE IF EXISTS fill_cubes_1h;
 
DELIMITER //
 
CREATE PROCEDURE fill_cubes_1h($StationID INT)
BEGIN
 
  IF Station_ID IN (SELECT StationID FROM `database`.`Stations`)
  THEN
    SET @strSQL = CONCAT('
      INSERT INTO `database_cubes_1h`.`Station_',$StationID,'`
      SELECT
        SensorID,
        UNIX_TIMESTAMP(DATE_FORMAT(FROM_UNIXTIME(Time),"%Y-%m-%d %H:00:00")) AS Hour,
        AVG(Value) AS Value,
        MAX(Value) AS ValueMax,
        MIN(Value) AS ValueMin
        FROM `database`.`Station_',$StationID,'`
      WHERE
        Time >= UNIX_TIMESTAMP(DATE_FORMAT(NOW() - INTERVAL 1 DAY,"%Y-%m-%d %H:00:00")) AND
        Time <= UNIX_TIMESTAMP(DATE_FORMAT(NOW() - INTERVAL 1 HOUR,"%Y-%m-%d %H:59:59"))
      GROUP BY
        HOUR, SensorID
      ORDER BY
        HOUR ASC
      ON DUPLICATE KEY UPDATE `Value` = VALUES(`Value`)
    ');
 
    PREPARE statement FROM @strSQL;
    EXECUTE statement ;
    DEALLOCATE PREPARE statement ;
 
  ELSE
    SET @foo = "";
 
  END IF;
 
END ;//
 
DELIMITER ;

The next step is to get this procedure running every hour for all the stations in the database. Now we are going to use our Stations table. The steps to follow are:

Request the highest station ID in Stations table
Since station ID’s are not sequential, get a way to determine if related table should exist
Tell the procedure to keep going if any exception raises
Run the loop until reach final station ID, and call the previous procedure passing current index as a parameter

The final result looks like:

^?View Code MYSQL

-- -----------------------------------------------------
-- Procedure `climaps_main`.`fill_all_cubes_1h`
-- -----------------------------------------------------
 
DROP PROCEDURE IF EXISTS fill_all_cubes_1h;
 
DELIMITER //
 
CREATE PROCEDURE fill_all_cubes_1h()
BEGIN
  DECLARE v INTEGER;
  DECLARE m INTEGER;
 
  DECLARE CONTINUE HANDLER FOR SQLEXCEPTION BEGIN END;
 
  SET v = 0;
  SET m = (SELECT StaID FROM climaps_main.Stations ORDER BY StaID DESC LIMIT 1);
 
  WHILE v &lt;= m DO
 
    CALL fill_cubes_1h(v);
    SET v = v + 1;
 
  END WHILE;
 
END;//
 
DELIMITER ;

We’ve nearly finished! We have a way to build hourly cubes for a variable number of stations, and solving the problem of asynchronous data. We just need to create an event to run this last procedure, as we have stated above, every hour. This should work on MySQL 5.1

^?View Code MYSQL

-- -----------------------------------------------------
-- Event `database_main`.`fill_cubes_1h`
-- -----------------------------------------------------
 
DROP EVENT IF EXISTS fill_cubes_1h;
 
DELIMITER //
 
CREATE EVENT fill_cubes_1h
  ON SCHEDULE EVERY 1 HOUR
  DO BEGIN
      CALL fill_all_cubes_1h();
  END //
 
DELIMITER ;

If for any reason we are not allowed to create events (all this stuff should be run as a user with whole access to the tables involved), or we are running a MySQL version lower than 5.1, a cron job should be run instead. Just get EXECUTE access on the database where the procedures have been created (on my case, database_main), and add this line to crontab:

^?View Code BASH

1	0 * * * * mysql -e 'CALL database_main.fill_all_cubes_1h()'

Please note that to get this statement running without asking for a password, you need to create a .my.cnf file in your home directory (UNIX systems)

And that’s all!

So, what do you think? What’s your method? Comments and feedback are much appreciated

Managing big data sets with MySQL

Jorge Albaladejo — Wed, 05 May 2010 15:23:31 +0000

I’m currently working on a database which stores, literally, a million of new data registers every day. They come from several hundred weather stations spread along the world, and need to be processed on real time. That’s a huge amount to be managed with a simple – but powerful – MySQL server, so I’ve had to design and implement a few tricks in order to succeed.

In this post I’d like to talk about the main three techniques we’ve used to engage our high performance database, and the reasons to choose these solutions.

First of all, MySQL innodb tables start degrading at some million registers (7-10M on my current server, but not tested further), so the first though I had was: we need to divide if we want to conquer. Then, after some thinking about, I decided to split the big table into small chunks, and use a table for each station. That made the whole problem more affordable, and tables smaller enough not to become scary: the new problem was how to store about 5 M registers per year per table in our database.

Of course, some logic in both the application controller and the data input server was needed to correctly place data registers. Additionally, tables get often fragmented so some cron jobs were needed to keep the database running smoothly. But there was still the problem of data growing. As long as registers were separated depending on the station, the scalability problem was solved: more tables can be added, even more servers with ranges of stations and a hash function to calculate what server is a given station hosted in. Once again were we talking about several million registers per year per table, and with a standart MySQL 5.1 server (the main problem with an start-up company tends to be funding, so we didn’t have anything better).

Then a second solution came out: since 80% of queries are performed over the last few years of data, let’s move oldest registers to depot tables! This way, data used most often will be kept on relatively small (affordable, at least) tables, and the rest will be just stored. So a big problem had turned into a matter of designing some queries over a bunch of tables, in a way that can be easily automated. Great! And it’s been so for a while, until we’ve realized that even with small tables, we are handing huge sets of data to represent evolution of physical variables along time. I’m talking about high resolution data, let’s say a measure every minute for some types of sensor, which makes sense to study minor variations along the day.

It’s now when we start thinking about cubes – resumes. The real headache with big data sets starts when comparing a whole month range of values, which will produce several thousand registers to represent on a 800px wide plot graph… that’s basically a waste of server processor and memory resources, since we are providing a resolution which is not adapted to the circumstances.

So now we have a third solution which is based on averages. MySQL – as other database engines – allows creating averages on the fly which is further better than relying these operations on the application layer. The problem beneath is that a big data set must be evaluated on demand, and this for every user on the website (we barely can cache MySQL responses because values change every 10 minutes). To fix it, we’ve decided to run these operations just once every hour, and store the results on cubes – tables with resumes – which would be consulted instead of the high density original ones. The algorithm to determine the table to look is simple: if the range of data is larger than several days, just use the hourly resolution instead. To automate this, we’ve added some naming convention, let’s say, all data tables are named the same way (based on the station ID), and only the database name (based on the data resolution) changes.

To finish with, I’d like to advance the construction of this cubes generation process:

A stored procedure calculates averages for a given period for a given table
Another stored procedure launches the first one for all the tables; it is defined to be called from whatever time event handler we use
The event handler is defined, well in MySQL engine, well as a cron job

I’ll talk about the implementation details in further posts…

Tunneling a multipart POST message through PHP and CURL

Jorge Albaladejo — Wed, 05 May 2010 14:50:51 +0000

In this post I’m going to cover a particular solution to a given problem we’ve faced up in our last project. Let’s set the background: there is a complex web application under development on a shared virtual machine, installed on a subdomain of a public network that will change when going to production. We have also a commercial website hosted in a shared server under a domain that should not change, since it is our first step out of this virtual farm.

For this project, we require to integrate some cameras which send pictures via GPRS connections to the server, and we want to provide some user authentication to avoid having anybody posting whatever to this web service. So, we have developed a controller in Code Igniter which gets a camera ID / password from POST, and also the bytes for the image, and proceeds to do some filtering. For instance, camera ID and password should match with the values previously registered in the database, and the picture should be MIME image/jpeg, with a limit of 2MBs. Up to now, nothing special, tested and working with POSTS from the within the website.

The problem came when we found out that cameras only allow to send their information to a root domain, to a particular file, let’s say upload.php. So, we would only need to put this file in the root path and it’s done. Unfortunately, doing so in the application virtual server would result on flashing again all the cameras once we move to a new server and set up a final domain name. We might be able to install this receiver in the commercial website root path, but then we would require to send the message back to the application server in any way, so it is not the best solution we want to solve this issue.

‘Just add a domain that points to this subdomain and you are done’ would you say. Well, yes, that would have done the trick. Unfortunately, we cannot host this domain in the machine since we have no administrator rights on the servers farm. Nor can we just use this external domain because it produces an HTTP redirection which, to make things worse, is not supported by the camera. We’ve even tried with cloaking the url of the receiver with an external subdomain, with no success – since cloaking wraps the real content inside an iframe and it breaks the POST message.

What a headache! it seemed there was no way to implement the system with the full functionality we wanted…

At thas moment we were a bit puzzled because we had to choose between two main options, which in any case took us away from the initial implementation design:

hosting the receiver script in a subdomain that would change
hosting it in a fixed external domain (the commercial website) and try to send the messages back to the application server
in any case, we would have to implement the features planned for the Code Igniter controller in this new stand-alone script, which would produce extra work and remove some built-in capabilities

How to fix this problem? Well, he have chosen a variant of #2: try to send the messages back to the application server. But how exactly? Well, by tunneling the multipart POST message with CURL.

To achieve our goal we have written a script which is hosted in the domain root of the commercial website, with does the following:

Reads the serial number sent by the camera
Reads the temporary upload file, and moves it to a specified folder
Initializes and configures PHP-CURL to generate a new POST message with the data recovered and the image content as multipart form
Deletes the temporary file created

And the code:

^?View Code PHP

/**
 * @file upload.php
 * This pipeline grabs a POST message and sends it back to a processing server
 * It is meant to solve a particular problem where the sender device has limited
 *  HTTP functionalities like being unable to deal with Apache redirections
 *
 * This code is just an example, you may use for learning purposes
 *
 */
// camera serial number, it is barely filtered because the real processing
//  is to be done in the final server
$csn	= ( isset( $_POST['CSN'] ) ? $_POST['CSN'] : 0 );
 
// we need to locally store the file to resend it
$file 	= $_FILES['userfile'];
$tmp 	= "tmp/" . $file["name"];
move_uploaded_file( $file["tmp_name"], $tmp );
 
// post information is rebuilt
$data = array( 'CSN' =&gt; $csn, 'userfile' =&gt; '@' . $tmp );
// note the @ which tells CURL that this field is a file
// this will be automatically interpreted as a application/multipart form
 
// init and configure curl
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, 'http://the-application-server/path/controller/method' );
curl_setopt( $ch, CURLOPT_POST, true );
curl_setopt( $ch, CURLOPT_POSTFIELDS, $data );
 
// finally, send
curl_exec ($ch );
 
// and delete the locally stored file
unlink($tmp);

Managing a web development project

Jorge Albaladejo — Thu, 07 May 2009 12:02:16 +0000

There are many types of websites, as well as sizes and complexity. At the beginning, most of my creations were constructed ‘on the fly’, without much planning – in the end, a ‘news’ module is not scary enough to need a project.

But recently I’ve started working on more ambitious jobs (like http://climaps.com), and feeling the need of a strongly defined method to apply to web applications design. Some tools are almost mandatory, like Use Case Diagrams which become really helpful in order to translate what every user can do into ACL‘s, methods, controllers, etc; or like modeling tools to graphically represent the application’s database.

But the best way – according to my experience – to successfully manage a web project is following a detailed planning. Determining what the steps are and what order should they be executed in, is a good starting point to know were exactly we are on each development stage.

Thus, I want to clarify that I’m not talking about software, applications or techniques to effectively manage big web projects; I don’t have this experience to talk about. Instead, I’m referring to the main steps I’ve followed in the past to handle the web projects I’ve worked on, steps which have proved to be effective. This article does not describe a rule but a personal way of doing things, and I’ll be glad to know your thoughts about it. What tools do you use to build your websites? Comments are welcome

The following diagram shows an overview of the entire process:

Web project management flow

Chronological steps to follow

Business: Analysis of requirements
Business: Get data model
Database: Design
- get to 3FN and study going beyond
Database: Implementation
- select a database engine
- determine storage engine
- develop triggers, views, and additional features
- develop maintenance and backup scripts
Database: Life time (*)
- design plan for durability
- implement and estimate performance
- develop triggers, views, and additional features
- design libraries to make it transparent for the application
Application: Use Case Diagrams
- represent different user profiles
- review with client
PHP: Develop initial system (*)
- choose a framework
- set up initial libraries and systems
- design template’s system
- unitary tests system
PHP: Core develop for all functionalities
- programming all functions needed for cases of use
- generation of a initial HTML view
PHP: Tests and bug fixing
- use unitary tests
- use functional tests
HTML: Finish structure
- use semantic code
- define attributes for accessibility
- run Hera, eXaminator, w3c and other tools and make adjustments
CSS: Definition (*)
- design a CSS files structured system
- start filling with some initial design
- run W3C CSS test
Design: External outsource (*)
- slice design in usable blocks
- apply design through HTML images and CSS
Standards compliance: Tests and adjustments
- multi platform: check browsers compliance: check IE, Firefox, safari
- run again W3C tests for HTML and CSS
- run again Hera, eXaminator and other accessibility checks
- layout adjustments
Ajax: Analysis of requirements
- cases of use diagram, with different user profiles
- determine functionalities to be available
Ajax: Setting up (*)
- determining a framework to work with
- establishing folder distribution
- determine unitary and functional tests systems
Ajax: Implementation
- develop all functionalities
- add necessary PHP back ends in XML / HTML format
Ajax: Test
- apply test cases
- make adjustments
Summary: Final review
- presentation of website to client
- getting feedback
- taking notes for a new wheel’s spin

Note: sections marked with (*) can be avoided when adding new modules to a previously developed application.

Jorge Albaladejo

ActionScript 3 and cross-domain problem

Basic web mapper

The architecture

The code

. ($depth*$this->indent) . 'px">' . stripslashes($title) . '

. ($depth*$this->indent) . 'px">' . $page . '

Scrum / agile development

MySQL clean reinstall

FTP server on Ubuntu

Start a service or execute a script on boot on Ubuntu/Fedora

Run levels

Services, daemons and servers

Custom scripts and applications

Alternatives

Implementing a database cubes system on MySQL

Managing big data sets with MySQL

Tunneling a multipart POST message through PHP and CURL

Managing a web development project

Chronological steps to follow