Things i've made

Looking for more? See a list of everything i've made.

Saturday, December 3, 2011

High performance NCSA alternative for Nagios

Ive worked with some non-trivial nagios setups: ~5 nagios nodes with each performing 20-50 service checks per second. If you have never worked with distributed nagios deployments then you would be oblivious to the pain that NCSA can cause. First off I should start be saying that NCSA is responsible for reporting the results of service checks on remote nagios nodes back to the master nagios nodes, essentially allowing the setup to be distributed. What no one tells you when you are deploy NCSA is that it send service checks in series while nagios performs service checks in parallel. If your nagios node is already operating at capacity and you try and have it report to a master node, you are going to see service check results queue up and then services start to appear as if they are flapping.

Not being able to find a simple solution to the NCSA backlog problem I went and created a micro daemon that replaces NCSA with something a lot more high performance so instead of NCSA being the bottle neck, that role is back on nagios itself. Another fantastic feature of this NCSA replacement is that it emits service checks for itself so you can monitor how many service checks per second are being proxied and how many bytes/second that is - great if you have little bandwidth.

Qt-Wrapper for Gtk nearly done

If you have been following my progress with the Gtk to Qt porting wrapper I am working on then you will be pleased to know that it has reached to %90-%95 functionality milestone. The large application I am porting works quite well for most workflows. Along the way I ended up creating a Gtk wrapper for the Qt wrapper on Scintilla so applications using the Gtk interface to Scintilla will compile and run without modification.

The most recent completed challenge was to perfectly emulate Gtk's hide/show mechanism. In Gtk, widgets are created hidden, you can then show them, or use show all which will show all child widgets as well. Additionally, Gtk widgets have a dont-show-all flag which will prevent a widget from being shown if its parent is part of a show-all operation. In Qt, a widget can be shown which will implicitly also show its children widgets unless they have been explicitly hidden in which case they will only become visible if explicit shown.  To emulate Gtk I simply hid each widget on creation and added a flag to indicate dont-show-all. When show is called, children widgets won't be shown because they were hidden. When showing all, it iterate through all child widgets and if the dont-show-all flag is not set, then I show all on those.

With the currently implemented in the wrapper, it is pleasing to take a Gtk application, compile it against the wrapper, and actually be able to use the compiled application.

Some nagging issues remain like event ordering or exact widget state which is not documented in Gtk. Its not possible to know every detail since some are specific to how Gtk is implemented rather than its intended functionality. This causes some headaches but with good debugging tools and some patience, the number of these issues is being reduced.

Want to see your Gtk application running in Qt?



Tuesday, September 13, 2011

Gtk to Qt Port - Work in Progress

Not to long ago I started on a project to port a large application from Gtk to Qt. So far the progress has exceeded my expectations and I have even been able to spend some time on achieving pixel perfect layout of widgets, something I had previously relegated to the later stages of the project.

Tuesday, August 9, 2011

Smooth Curves for Gtk Chart Component

I recently upgraded my Gtk Graph/Charting component to draw smooth curves between data points when requested. This minor updates uses cubic splines to draw curves that touch every data point in the series being drawn. This is different from plotting a trend or curve fitting, that would mean trying to fit a curve of some degree to the data points.
Smooth curves

Smooth curves applied to bounding area plot
The above images show that not only are smooth curves applied to line plots, but also area, and bounding area plots.

So what about the more complex task of curve fitting ? I've laid some groundwork for that in response to some requests so expect to see it in the future.

Sunday, July 10, 2011

Convert a Gtk Application into a Qt Application ?

So, it has been tasked upon me to take a very large Gtk application, say millions of lines of code, and plan a path for it to exist without Gtk. Why move away from Gtk? Well this application work extremely well on linux and solaris, yet it seems to have continual issues regarding Gtk on windows. I would go into the exact details here, but I do still like Gtk for what it does, I just wouldn't consider writing a large application with Gtk if Windows was going to be one of the target platforms.

Wednesday, June 22, 2011

Gtk Chart Annotations

If you haven't already seen, I wrote a charting widget for Gtk/Cairo. Over the last fews days I have been adding annotations into the package. Annotations allow you to add arbitrary lines, circles, stars, images, text, etc to a chart. Annotations can be used to add non-data information to a chart like trend lines, and resistance and support, or, you can add purely visual elements like images to enhance the readability of the chart.

Thursday, May 26, 2011

Cairo/Gtk Charting Widget

Updates! :
Gtk has no built-in charting capabilities. There are also no existing, fully functional charting packages for Gtk. I needed to add flexible charting capabilities to a trading platform, using Gtk, that would work on Windows, Linux, and Solaris. The goals of the charting package was to be able to chart data from anywhere in the same chart, and use the chart for user input, ie allow point to be moved round, zoom, navigate, etc.
 Gtk uses cairo as its rendering backend these days. Cairo is a vector graphics rendering package that works on a large number of platforms and is accelerated where hardware backends exist. Using cairo to render the chart makes it integrate nicely with the core of Gtk and also allow the charts to be used outside of Gtk applications, simply linking against cairo.

The flexibility in charting comes from the overall design. Each chart can have zero or more axis. Each chart can have zero or more series, with a series being tied to the axis it is plotted against. All text labels can be controlled through specialization. With this design at the core, adding new functionality, such as a new series type, or an irregular axis is a simple task.
A collage of some charting features
A non-exhaustive list of features supported by this charting package include:

  • Series types :Lines, bars, points columns, areas, candle sticks, HLOCs, bounded regions, bubbles, pie
  • Full control over text labels for axis, values, hover popups on points with the most common formatting built-in.
  • Unlimited X and Y axis on an combination of sides/top and bottom of the chart
  • Full Dynamic updates allowing changing points, adding and inserting, removing points. Changing or adding axis, series, etc
  • Saving to PDF, PS, SVG, or any other format supported by cairo.
  • User interaction so you can drag points, zoom in on regions, select points by region or clicking, hover popups showing details for the nearest point
  • Support for Gtk versions 2.4.x to 3.x, and generic applications by including cairo.
Availability and Source Code
I designed and implemented this charting package in its entirety as well as many actual application uses for trading. The code is of commercial interest and thus not available for download.

Feel free to contact me to discuss the availability of this package.

Stunts Remake

Have you ever played Stunts, aka 4D Sports Driving ? Well, if you haven't, let me tell you, its an old racing game where you could design your own tracks from a set of race track tiles, and race on them against the clock or one of the computer AI racers, complete with far-out personality. There is also a replay feature which allows you to save your best (or worst) races so you can show them to your friends.

To me, Stunts is a high point in racing games. It had everything else other racing games had at its time, plus the track builder. This makes the lifespan of the game longer than any other. There have been other racing games to come out with track builders, but the tracks you could build in stunts always seemed exciting whereas other games just ended up giving you a lot of freedom to make boring tracks.

Anyway, I liked this game so much that I decided it needed a remake. When I started I wasn't aware that there was already an open source remake in progress, know as ultimate stunts. Had I actually looked to see if there was a remake, I may have ended up just contributing to that instead of creating my own.
The loop is easily drivable with some practice
M remake consisted of a few logical parts put together to cover the functionality of the original game. There is a track editor, a racing simulation, a replay viewer, and a game state management interface. Deciding that supporting the original trace format gave easy access to hundreds of ready-made tracks. Actually implementing the support, while upgrading the graphics presented a few challenges. The original track format allowed you to put water sections in your tracks. These water sections rendered at flat blue sections on the track. When I upgraded the water section to true 3D tiles that have gradually deeper water, the edging of the water tiles' underwater sections made the total number of water tiles jump from nine to twenty seven. Equally as important, I had to convert to and from this new "edged" terrain format so that the saved files still matched the original track format but looked perfect in-game. 

Early development shot showing some track elements

The actual graphical improvement I provided came mostly from the use of modern rendering techniques while the geometry remained mostly the same to preserve the original feel of the tracks. Each track piece was constructed using a set of geometry construction functions which allowed for a very concise description and mathematically perfect representation. I had experimented with using modeling packages but found it difficult to keep the geometry numerically perfect, which ultimately resulted in a rough driving experience. Non-road segments show this history with scenery imported from 3d studio max.

I spent some time applying some important techniques I had learnt in my PHD to give a greater feel for the speed you are traveling at. Imagine that this game had no textures on the road, no grass texture, and few scenery objects. Under these circumstances it would be difficult to tell how fast you were going on a straight stretch of road. This was not the case in the original stunts since, even though there was no texturing, the broken white line on the road gave you a good sense of speed. Adding textures improves this but even more, adding grass on the side of the road seems to give an even better experience of speed.  As simple as it may seem, it made a huge difference. You will often see space or undersea game use space junk, dust, seaweed, anything that can fly at you just to make it feel like you are moving. Anyone remember NOX ?

Time of day is customizable going through a whole 24 hour cycle 
An interesting challenge I encountered while implementing the racing simulation was to determine when someone had taken an illegal short-cut. To solve this problem i wrote a simple recursive track tracer. I implemented it in such a way that you could either run the tracer in exploration mode, for the track editor, and find any issues with the track such as broken track, dead ends, junctions that put you going the wrong direction on the track. Another mode was a guided mode where the tracing was guided by external interaction, or in this case, driving the car round the track. If any unexpected detours were taken, a penalty time could be asses based on the shortest track path skipped, and wrong driving direction could be detected. Using the same tracer for two purposes allowed me to focus on correctness and handle a large number of exceptional cases, as is the way when you can design you own track that can have turn offs, jumps over roads or obstacles, even cycles in the track.

Debugging view of some more hairy track sections showing polygon edges and normals
 The graphical sections of the game were programmed to use OpenGL. Using OpenGL gave the benefit that the game could be run on Windows, Linux, and OSX with no differences in the rendering code. I teamed this with SDL and basically had a code base with no ifdefs or files for separate platforms, achieving identical functionality. Being somewhat of a computer graphics person, I spent some considerable effort ensuring the rendering achieved high performance. Even though the geometry is simple, a naive implementation could be outperformed one hundred times or more by an optimized implementation. The end result is over 100 frames per second at high resolution on a GeForce 2MX and processor from same era.
Car taking a corner
The physics of the original game were quite basic yet very fun and predictable to race. I achieved a similar feel with a more modern twist by use the Open Dynamics Engine. With some fine tuning I was able to get a car that could fly through the air, land without bouncing off the ground, and still have an accurate feel when cornering and transitioning between traction and losing traction.





Availability and Ownership
The source code for this project is entirely owned by myself and unreleased. The project is mostly complete, just with some polishing to do. I abruptly stopped working on this project when I had other projects start to make an income and was well aware that deriving any income from this project with some major diversion away from the original game would be immoral and possibly illegal.

If you are interested in the project, feel free to contact me directly with you queries.