Subscribe to GameFromScratch on YouTube Support GameFromScratch on Patreon

31. July 2012

 

I had intended to end this series on part 4, all I had left to do was add a layer of persistence because Heroku didn’t keep files for more than a few hours, which was a pretty heavy limitation.  So I looked into various storage options, and I ended up going with CouchDB.  The process was a bit more involved ( and interesting ) than I suspected, so I decided to add another post in the series.  This part covers setting up cloud based database, and changing our code to persist to the database.

 

First off, I went to IrisCouch.com and signed up for a free database.  They provide (free!) cloud hosted CouchDB installs, including a full Futon management system, pictured below:

image

 

You can use this interface to create new databases, configure security etc.  It’s a bit tricky at times to navigate, but it gets the job done.

 

I created a new database called firstthis, then immediately secured it ( by default your website is publically accessible to everyone! ).  CouchDB works by storing information in documents instead of tables in a traditional database.  Instead of updating individual fields, you update the data in your document then replace the entire thing.  Tracking the newest revision becomes incredibly important when working with CouchDB.  Perhaps most key of all, CouchDB adds two fields to your data, _id and _rev.  _rev represents the newest revision, while _id represents the unique key.  In our case, for our user settings, we are going to use their email address as the key.  We simply store the files variable from our script to the server.  Here is a sample of files stored in CouchDB ( shown in the iriscouch admin page ):

 

image

 

As you can see, its simply our JavaScript variable, with an _id and _rev added. 

 

When you sign up for IrisCouch, you are given a URL where your database server is located, in the form of yoursite.iriscouch.com.

 

Now let’s take a look at the code differences.  It is pretty thoroughly documented ( combined with the above explaination ), so I won’t go into a lot of detail.  CouchDB is accessed using REST requests, but I instead used a node library Nano for access.  This was a bit of a double edged sword, as it took away a great deal of the complexity and grunt work, however it also removed me a step away from the well documented CouchDB. 

 

All of the code changes are in server.js

var express = require('express'), server = express.createServer(), im = require('imagemagick'), nano = require('nano')('http://Serapth:meat32ball@gfs.iriscouch.com'), db_name = "firstthis", db = nano.use(db_name), userEmail = 'mike@gamefromscratch.com', fs = require('fs'), files = { files:{}}; // Helper functions for getting and inserting docs in couchDB using nano function get_doc(docName,res){ db.get(docName,function(err,body){ if(!err) { res(body); } }); }; // There is no update in CouchDB. Just inserts, inserts and more inserts. // If you don't have the most current rev ( or it isn't a new insert ), an error (HTTP409) will occur. // TODO: Real error handling, attempt to get latest file, get it's rev, then try inserting again function insert_doc(doc,docname, tried) { db.insert(doc,docname, function (error,val,newval) { if(error) { return console.log(error); } // The insert will result in an updated rev, update our local files var to the most current rev. return files._rev = val.rev; }); } // Setup server static paths server.use('/cocos2d', express.static(__dirname + '/cocos2d') ); server.use('/cocosDenshion', express.static(__dirname + '/cocosDenshion') ); server.use('/classes', express.static(__dirname + '/classes') ); server.use('/resources', express.static(__dirname + '/resources') ); // Install the bodyParser middleware, which enables form data to be parsed when an upload occurs. server.use(express.bodyParser()); // Handle requests for / by returning index.html server.get('/', function(req,res){ res.sendfile('index.html'); console.log('Sent index.html'); }); // Handle requests for /settings by returning settings.html server.get('/settings',function(req,res){ res.sendfile('settings.html'); console.log('Send settings.html'); }); // Handle requests for images will be the form of site.com/image/imagename.png // Fetchs the image data from CouchDB and returns to user server.get('/image/:name', function(req,res){ if(files.files[req.params.name]) { res.contentType(files.files[req.params.name].contentType); db.attachment.get(userEmail + "/" + files.files[req.params.name].name,files.files[req.params.name].name).pipe(res); } }); // Uses ImageMagick to get image dimensions and return them as JSON data // This is to work around the requirement for Cocos2D sprites to know dimensions before loading image server.get('/imageSize/:name',function(req,res){ im.identify(files.files[req.params.name].path,function(err,features){ if(err) throw err; else res.json({ "width":features.width, "height":features.height }); }); }); // This gets the photo data, which is contained in our files variable. Simply return it JSON encoded server.get('/getPhotos', function(req,res){ res.json(files.files); }); // Erase all images : TODO: Remove images from database as well!!!! server.get('/clearAll', function(req,res){ files.files = {}; res.statusCode = 200; res.send(""); }) // Unfortunately there is no easy way to tell when a multi file upload is complete on the server, On('end') isnt called // Therefore we call /doneUpload from the client once we are done uploading. // Once we are done uploading files, we save our updated Files var up to couchDB, then get it again so it again immediately to have the most current rev server.get('/doneUpload', function(req,res){ insert_doc(files,userEmail,files._rev,0); get_doc(userEmail,function(res) { files = res; }); res.statusCode = 200; res.sendfile('settings.html'); }) server.post('/upload',function(req,res){ // This method is going to be called for each attached file. // Add the file details to files.files[] with the key set to the filename files["files"][req.files.Filedata.name] = { "name":req.files.Filedata.name, "path":req.files.Filedata.path, "size":req.files.Filedata.size, "contentType":req.files.Filedata.type, "description":req.body.description }; // Now read the file from disk and insert the file data in our CouchDB as an attachment named "emailAddress/filename.png" fs.readFile(req.files.Filedata.path,function(err,data){ if(!err){ db.attachment.insert(userEmail + "/" + req.files.Filedata.name,req.files.Filedata.name,data,req.files.Filedata.type , {}, function(err,body){ console.log(body); res.statusCode = 200; res.send(""); }); } }); }); server.listen(process.env.PORT || 3000); // Check the couchDB for an entry keyed to the users email address. If one exists, copy it into our files var get_doc(userEmail,function(results){ if(results){ files = results; } });

 

Now when you upload images, they will be stored to your CouchDB database on IrisCouch.  When you restart Node, it will automatically grab and populate files from the database. Note, the application from parts 1-4 aren’t updated to use this new code.

Keep in mind, this code isn’t production ready… error handling is sparse, there is no authentication, it would be easy to exploit with a DoS attack for example.  However, if you are interested in storing data from your Node App in a CouchDB database, I hope this sample was useful.

Programming , , ,

29. July 2012

 

I recently completed work on a tutorial series A Simple JavaScript Game using Node, cocos2D, YUI and Heroku, which followed the creation of a simple app for my own use.  It was a pretty complete application with one glaring fault…  there was no persistence.  The application was hosted using Heroku’s free tier which doesn’t keep files for more then a few hours.  This obviously leads to a bit of a problem.

 

So I have been looking into the myriad of options for persistence with Node. There are a few options, all of which have various advantages and disadvantages.

 

The easiest solution would probably be some persistent storage like Amazon’s S3 or even my local file system ( and move the application from Heroku to my servers that are running on Windows Server ).  There is nothing wrong with either solution, but I don’t really want to add more load to my servers with a technology I am still learn learning, least of all to minimize security issues.  Also, I started thinking I wanted a bit more database functionality as I may be adding more functionality.

 

Once I start thinking Database + Node, that changes the landscape quite a bit.  I am already running SQL Server, and amazingly enough Microsoft has been embracing and contributing to Node, including a driver for MS SQL Database with Node.  However a) it is extremely early in development b) the syntax looks… wordy and crude, hopefully this improves massively, because accessing data values by offsets seems so very… retro.

 

In the world of Node, there seems to be 3 front runners:

 

Redis

MongoDB

CouchDB

 

All of them have strengths and weaknesses.  All three are part of the NoSQL movement, but each approaches things quite differently.

 

Redis is stored entirely in memory ( but syncs to disk ), and works with key/value pairs.  It is not ideal for file storage, but is wonderful for quickly storing away JavaScript objects.  Also, Heroku has Redis support as an addon.  Now the downside, Redis isn’t available on Windows, at least not in a supported capacity.  As I develop using both Windows and Ubuntu, this was pretty much a deal breaker for me.

MongoDB is another NoSQL option and to be honest, I forget why I didn’t go with it, at least not initially.  I know I didn’t particularly want to install the underlying DB server, but it was at least supported on Windows.

CouchDB is what I ultimately went with.  It is another NoSQL database, but it could probably be best considered a document store, that stores JSON documents ( and other files ).  Given the nature of my application ( serving lots of files that don’t often change ), this is actually a very good thing.  That said, my SQL trained brain is having a whole lot of difficulty dealing with the change in mindset.  Storing “data” in documents that aren’t in fact documents seems horrifically unnatural to me.  Worse, I am really having trouble coming to grips with the idea of not being able to delete versions!  The idea that every time I change data it creates a new document, there is no update, only inserts.  These seems horrifically inefficient, but I have to assume I am thinking about things wrong.

 

What ultimately sold me on CouchDB was the low barrier of entry in the form of Iris Couch, which is a cloud hosted Couch DB, with a very generous free option.  Like Heroku, having someone else handle the heavy lifting is always enjoyable.

 

Being new to NoSQL, I am still going through the learning curve, so there is nothing to say I will stay with CouchDB, but I will say, I have gotten some impressive results very quickly.  I really wish Redis was available on Windows, as I would probably use redis for “data” and Couch for documents.  Anyone have alternative suggestions?

Totally Off Topic ,

27. July 2012

As you can probably tell if you have visited before, I have made some changes to the layout of the site.  This is one of those items that has been on my to-do list since… well basically since I created this site.  I originally used slightly modified existing theme when I created the site, then just sorta hacked at it as I needed to make changes.  That said, over and over I found things “pinched”, especially code samples.  Looking at stats, 95% of you are using screens wider than 1024 wide, so designing for that resolution made very little sense.

 

I am not quite done making changes, I still need to work out the menu contents, as well as give the sidebar a face lift, but I hope you enjoy the new layout.  In time, hopefully the menu at the top right will help with finding content on the site.

 

No doubt though, I made a number of mistakes, so if you have noticed any major mistakes, please let me know!  Any feedback and suggestions are appreciated as well.

News

27. July 2012

 

One area of Blender that is often over looked, or people are downright unaware of, is its video editing capabilities. Hidden behind the 3D functionality we all know and love, under a layer of yes… sometimes confusing UI, lives a remarkably capable NLE ( non-linear editor ).

 

In this tutorial I am going to look at some of the simples and most common features Blender offers when it comes to video editing. We are going to look at a couple extremely common tasks, and how you would accomplish them using Blender. Without further ado, let's jump right in.

 

Setting Blender up for video editing

 

The first step is to fire up Blender. I personally am using Blender 2.63, but any version after 2.5x should follow more or less the same steps. NLE functionality was also available in 2.4, but the UI has changed extensively.

 

Once in Blender, click the Choose Screen Layout button and select Video Editing in the dropdown, like this:

 

 

Your Blender should change to the following layout:

 

The NLA editor sequence area is where we add all of our video elements, such as video, audio and images. In the top right it is in preview mode ( they are both the same window types ), showing a preview of the video we are creating.

 

The timeline view is where your timeline controls are, as well as controls for setting keys, video duration etc.

 

The graph editor is for fine tune control using f curves and will not be used much in this tutorial.

 

Alright, we now are in editing mode, let's start with something simple.

 

Adding a movie title screen

 

 

As is pretty common, we want to have a static "Title Screen" of sorts. In our case it is just a simple png image we are going to display for 5 seconds. We are recording at 30FPS, so that means 150 frames.

 

First we need to add an image. I created a 1920x1080 PNG image in GIMP for this. This is because I am working in 1080p output. Use whatever you wish.

 

In the NLA Editor sequence window, locate the menu and choose Add Image:

 

Locate the file you want as a title screen. In the left hand side, locate start frame and set it to 0, and end frame and set it to 150:

 

Finally click Add Image Strip. The image will now appear on Channel 1 of the NLA sequence, like such:

 

 

Let's take a look at the key parts that make up the sequence window.

 

The left hand side, the vertical axis represents the different channels composing your video. In the crudest sense you can think of these like layers in a graphics program ( with obvious exceptions, such as the fact they can overlap, and contain sound ).

 

Across the bottom, the horizontal axis represents elapsed time.

 

Then in this case, the purple object represents your image. Different components have different colours as we will see. The length of this bar represents the duration that the image is visible in the timeline.

 

You can navigate this window using many familiar Blender hotkeys. The most important for now is G)rab to move the active item around. The middle mouse button pans around, while the scroll wheel zooms in and out. While the "." Key zooms to fit the selection/sequence. With only a single channel, these two features aren't really all that useful yet.

 

So, that's the sequence view, but you should also notice at the top right corner, the other NLA window is showing an active preview, like so:

 

As you move from frame to frame, this area will update.

 

Alright, so we now have a title screen that displays for 5 seconds, lets add some video:

 

Adding a video

 

 

In the NLA Sequence view, click Add, then Movie.

 

Just like before, select the movie file you want to add. Set the start frame to 151, and set channel as 2.

 

Now you will see the following results in the sequence:

 

As you can see, the video starts right where the image ends. Depending on the video file you uploaded, you may have gotten two channels like I have… what is going on here? Well, the blue strip represents the video portion of the video file we added, while the green portion represents the audio track. In this case we want to keep the audio. If you didn't, getting rid of it is as simple are right clicking it, then hitting X to delete. There is one potential problem though, the audio isn't synced by default. Let's correct that right away.

 

In the Timeline window, select Playback, and make sure AV-sync is selected.

 

Now audio and video will be synchronized.

 

At this point, we have one other issue… Our newly added video may be longer than our timeline.

 

Right click your video in the NLA Sequence window and take a look in the properties to the right ( hit N if the properties Window isn't visible. ). Locate the Length:

 

This added another 567 frames to our total length, for a total of 717 frames. We need to update our video length. In the timeline, locate End, click it in the middle and update to 717:

 

Now you can go ahead and preview your handy work up until this point. Just to the right of the end frame you just specified, are a set of VCR like controls:

 

The field to the left ( currently valued 191 ) represents the current frame, while the buttons are for controlling playback. The preview window at the top right should update as you got from frame to frame.

 

Now we have a title screen and a movie with audio, synced and playing back. Let's look at one of the next most common tasks…

 

Adding a watermark

 

 

Adding a watermark or signature to a video is one of the most common tasks you need to perform when editing a video, and fortunately it is remarkably simple. In fact, it works exactly the same as when we added the title screen. All we are doing is adding a mostly transparent image over top of our scene for the entire duration of the film. Create a 1080p image in whatever image editor you prefer, just make sure everywhere except your watermark is transparent. I used this image:

Nothing really exciting. Just like when we added the title screen, in the NLA Editor window, select Add->Image, select your file. On the left hand side we want it to start at Frame 0, go until Frame 717 and be on channel 4, like so:

 

If you look in your preview window, it probably just went all black except the watermark, like so:

This is because the image is obscuring the strips below. Don't worry, this is easily fixed.

 

In the NLA Editor sequence, make sure the watermark image is selected by right clicking it, then in the properties window ( hit N if not visible ), select the Blend drop down and choose Alpha Over, like so:

 

Then voila, in your preview Window everything is back to normal, just now with a watermark in the bottom right corner!

 

This process thus far assumed that your video was perfect and continuous, something that is rarely true. So now we look at…

 

 

Editing a video clip

 

 

Just like back in the days of physical film, cutting up and reordering video is a remarkably common task. Fortunately it is also quite easy. We are now going to split our image in half, and put an interruption in the between the splice.

 

First thing you want to do is select the point where you want to make the cut. You can do this using the vcr style controls, by left clicking anywhere in the timeline, left clicking anywhere on the VLA Editor sequence, or directly entering a frame in the current frame box. The green line indicates the currently selected frame:

Once you have your video where you want to perform the cut, hold down SHIFT and right click both the video and audio portions of the video strip ( the blue and emerald bars ). With both selected, hit 'K' to perform a hard cut. At this point, you have essentially "cut" the film and audio tracks into two separate entities.

 

Let's put a pause of 60 frames ( 2 seconds ) in between our cut. SHIFT select the audio and video strip on the right, press G to move them and move them 60 frames to the right, like so:

 

Now let's insert an image to be displayed during the gap. Using the same method we did before, add an image. In my case I want it to start at frame 231, end at frame 291 and be added on channel 4, like such:

 

Now if you play the video you will have a 2 second pause in both video and audio, while your image is displayed.

 

We are getting to the end, now let's add a simple effect. Let's fade our video to black over the final 160 frames.

 

Adding a video effect

 

 

We now want to select 100 frames from the end of our video. The easiest way to do this is probably in the timeline, although it is still only going up to 250! To resize the timeline to show our entire timeline either select View->View All, or hit the Home key. You will see the timeline now goes all the way up to 720, now left click around the 620 mark. This will move the current frame to 620, in the timeline, the VLA Editor window and in the current frame box:

 

Right click the video strip ( the blue one ), then choose Add->Effect->Color

 

In the properties window, make sure the resulting effect starts at 620 and has duration of 100. We then want to set the opacity to 0 ( making it completely transparent ), set Blend to Alpha Over like this:

 

Now in order, SHIFT Click first the video strip, then the color strip we just add, then select Add->Effect Strip->Gamma Cross. This will add another effect the results in the color getting slowly drawn over the movie. In the sequence it will appear like:

 

We now have a ( rather crappy ) fad to black to end our film.

 

Finally, lets fade out our audio file. In the timeline, move to around the 560 frame mark. In the properties window ( N if not on screen ), scroll down and hover over the volume button, like so:

 

Hit the "I" key. This will set a keyframe. The keyframed value ( volume ) will turn yellow:

 

Now we want to advance to the end of our sequence (frame 720 for me), and this time change the value to 0, and set another keyframe. Like so:

 

Now if you look in the graph editor window ( which has been collecting dust until now ), you can see a curve representing the audio falloff we just set:

You can control the rate the audio drops off by manipulating this curve. We however are going to leave it as it is.

 



And sometimes… you encounter a bug. After cutting the audio strip, I was unable to keyframe the second portion of the audio strip. It gives the error:

Could not insert keyframe, as RNA Path is invalid for the given ID (ID = SCScene, Path = sequence_editor.sequences_all["20120723_221045.002"].volume)

It is annoying, but easily fixed. Right click select the strip causing the problem, then select the menu Strip->Duplicate Strip. Now delete the original ( X ), and move your duplicate in it's place. Keyframing should now work again.

 



And we are now complete with the editing, let's go ahead and render it. This is where the Blender UI kind of falls on it's face, as this process is nowhere near as intuitive as it should be!

 

Rendering your video to file

 

 

First thing, we need to open up the Properties view. We can either open up a new Window, or repurpose an existing one. Since we aren't really using the graph editor, I am going to repurpose it. Click the window type icon and select Properties, like so:

 

First thing we want to do is set up our dimensions. Since we are outputting at 1080p, lets start with that default. Drop down the dimensions and pick HDTV 1080p.

 

In my case though, my source video ( and desired output ) is actually 30FPS, not 24. So lets change that next. Drop down FrameRate and select 30:

 

Now scroll down to the Output section, choose your output location and output format:

I am going with H.264 for this example.

 

Now scroll down and expand the encoding section. This part will be entirely dependent on your computer, the codecs you have installed as well as your desired results. I ultimately will be uploading to Youtube or Vimeo, who will both re-encode anyways, so I am going to encode at a fairly high level. The Bitrate is what will determine file size vs quality. Also by default it will not encode audio, so make sure you select an audio codec if you want your audio encoded. If you select an audio codec that isn't compatible with your video codec, it will give you an error when you try to render. Here are the settings I've used.

 

 

Now we are finally ready to output our video file. Scroll back up and click the Animation button:

 

And now Blender will churn off for a period of time creating your movie. If you are running Windows 7, a progress bar will update over the Blender application icon. You can press ESC to cancel the render at any time.

 

 

The Results

 

 

And here are the (rather lame) fruits of our labour!

 

Blender edited video results

Art , ,

26. July 2012

Unreal have just release the July update of the popular Unreal Engine.

New features include:

  • Perforce version control integration
  • Normal map workflow improvements
  • Numerous Unreal Editor refinements including
  • Nvidia Open Automate integration

 

Mobile improvements including

  • DrawTimer for indicating busy status
  • Better save game encryption
  • Better ipod background music support
  • SMS and Mail dialog support
  • Better memory management on lower end devices

 

Over all, if you aren't all that excited about Perforce or Open Automate, it's a pretty unremarkable release.

 

You can download it here and read the complete release notes here. You can read about a rabbit with a pancake on its head here.

News

Month List

Popular Comments