Firestrand's Blog

So while looking into a javascript grid framework, i thought i would see what jqGrid had to offer.

As I am browsing the site I noticed a commercial demo section which featured ASP.NET the very technology I am planning on using the grid with. So I bounce over to that site, click a few things then ran into an error. My immediate thought was, "If this is the quality of the demo what is the quality of the rest?".

Just like a resume should have no spelling or grammatical errors, a demo meant to sell software should be pretty bullet proof. Take it slow, make sure every piece works and is tested. If you plan to sell a piece of software your demo should be perfect, it is your customer's first impression of your software. You always want first impressions to be good.

Blog, javascript javascript, demo, framework

Ran across an excellent synopsis of YouTube lessons learned in scaling their application over at HighScalability.

Four of the "Lessons" really struck me as things every architect on a large software application should learn. Copied for convenience:

Approximate Correctness - Cheat a Little

Another favorite technique. The state of the system is that which it is reported to be. If a user can’t tell a part of the system is skewing and inconsistent, then it’s not.

A real world example. If you write a comment and someone loads the page at the same time, they might not get it for 300-400ms, the user who is reading won’t care. The writer of the comment will care, so you make sure the user who wrote the comment will see it. So you cheat a little bit. Your system doesn’t have to have globally consistent transactions. That would be super expensive and overkill. Not every comment is a financial transaction. So know when you can cheat.

Expert Knob Twiddling

Ask, what do you know about your consistency model? For comments is eventually consistent good enough? Renting a movie is different. When renting there’s money so we’ll do the best we can to never lose that. Different consistency models are needed depending on the data.

Jitter - Add Entropy Back into Your System

Hot word in their group all of the time. If your system doesn’t jitter then you get thundering herds. Distributed applications are really weather systems. Debugging them is as deterministic as predicting the weather. Jitter introduces more randomness because surprisingly, things tend to stack up.

For example, cache expirations. For a popular video they cache things as best they can. The most popular video they might cache for 24 hours. If everything expires at one time then every machine will calculate the expiration at the same time. This creates a thundering herd.

By jittering you are saying  randomly expire between 18-30 hours. That prevents things from stacking up. They use this all over the place. Systems have a tendency to self synchronize as operations line up and try to destroy themselves. Fascinating to watch. You get slow disk system on one machine and everybody is waiting on a request so all of a sudden all these other requests on all these other machines are completely synchronized. This happens when you have many machines and you have many events. Each one actually removes entropy from the system so you have to add some back in.

Cheating - Know How to Fake Data

Awesome technique. The fastest function call is the one that doesn’t happen. When you have a monotonically increasing counter, like movie view counts or profile view counts, you could do a transaction every update. Or you could do a transaction every once in awhile and update by a random amount and as long as it changes from odd to even people would probably believe it’s real. Know how to fake data.

Video: Scalability at YouTube

Originating Article: 7 Years Of YouTube Scalability Lessons In 30 Minutes

Optimization, Scalability Scalability

So I got frustrated with the sytax for parsing a string into an Enum value in C#. This is the generic method I came up with after reading how some other people solved the problem. It's here in case anyone else finds it useful.

        public static T EnumParse(this string value)
        {
            return EnumParse(value, false);
        }

        public static T EnumParse(this string value, bool ignoreCase)
        {
            if (String.IsNullOrWhiteSpace(value))
            {
                throw new ArgumentNullException("value");
            }

            Type t = typeof(T);

            if (!t.IsEnum)
            {
                throw new ArgumentException("Type provided must be an Enum.", "T");
            }

            var enumType = (T)Enum.Parse(t, value, ignoreCase);
            return enumType;
        }

.Net, C# C#, .NET, Enums, Extension Methods

Ars has a great article about Etsy and some of the architecture problems they have had to overcome. i chuckled when i read about their architecture and reliance on stored procedures. I have been advocating for years to avoid that type of architecture like the plague because of how hard it is to scale. That isn't to say that database sharding and stored procs can't work, but in my experience object relational mapping is much more likely to be successful.

Development Process Architecture

Just picked up the Developer Preview of the next version of Windows. So far it's interesting that it isn't branded Windows 8 etc. Just "Windows".

I'm also not sure I like the new look, I'll give it a little time though. I'm wondering if this will be Microsoft's hit out of the park, or epic fail ala Windows Millenium.

Windows Windows, Developer Preview

Anyone who knows what my hobbies are would not be surprised to see me excited that a developer magazine has published an article about Particle Swarm Optimization. The article just covers the basics and doesn't even get close to the state of the art with Particle Swarms, but it's still nice to see.

Here's a link to the article: MSDN-Particle Swarm Optimization

My open source PSO projects and a project I have worked with:

Standard PSO

Variable PSO

SwarmOps

.Net, C#, PSO Computational Intelligence, PSO

My laptop is a 2010 Macbook Pro with a 3rd party SSD, specifically a Samsung 256 GB SSD. Yesterday I upgraded my OS to the latest from Apple name OSX Lion, almost solely for the Trim support. So you can imagine my surprise when I found that Trim wasn't enabled for my SSD. After some cursing and google searches I found that Apple has specifically only enabled Trim on APPLE designated ssd hardware. Luckily I am not alone in wanting to enable Trim on my drive and I found someone who had created a nice little solution which edits the ktext file to get Trim support working on any 3-rd party drive. So if like me you have an SSD, Lion, and want Trim support head over and install the Trim Enabler.

OSX OSX, SSD

Recently I was working with some code which was using Tanh in order to squash values to the range [-1,1]. This is especially common in artificial neural network transform functions. So out of curiosity I thought I would look and see how efficient the built in Math.Tanh function is. Imagine my surprise when I found it was 3x slower than what it typically used for most neural network transform implementations.

//This version 3x faster than Math.Tanh
public static double Tanh1(double x)
{
     return 2.0 / (1.0 + Math.Exp(-2.0 * x)) - 1.0;
}

I added an estimated tanh version which is accurate to roughly 6 decimal places which is 10x faster than the Math.Tanh version.

//An estimate that is 10x faster than Math.Tanh
public static double Tanh2(double x)
{
    //Based on Lambert continued fraction taken to 7 divisions and optimized
    var x2 = x * x;
    var ax = (((x2 + 378.0) * x2 + 17325.0) * x2 + 135135.0) * x;
    var bx = ((28.0 * x2 + 3150.0) * x2 + 62370.0) * x2 + 135135.0;
    return ax / bx;
}

The moral of this story is when you need to apply a standard library function, like Math.Tanh, millions of times don't always assume that the built in function is the fastest available. If you can relax the accuracy requirements an estimate can boost performance even further.

*Special thanks to the Variety of Sound blog for the Tanh estimate writeup.

C#, Benchmarking, Optimization Standard Library, C#, Math, Neural Networks

It’s 2011 and I continue to see people dealing with XML serialization and deserialization in very different and in some cases painful ways. I was recently modifying an open source application that was walking through the XML nodes and manually matching and building objects based on those nodes, I put that in the painful category. So here I will present my take on how basic xml serialization and deserialization should be done in .Net.

The first thing we need is an object, for this I dug deep to create a generic Person object:

public class Person
{
     public string FirstName { get; set; }
     public string MiddleInitial { get; set; }
     public string LastName { get; set; }
}

Now that we have an object I want the ability to serialize and deserialize it to and from a nice compact XML string. An initial trivial implementation could look something like:

public static Person DeserializePersonFromXmlString(string personXml)
{
    Person result;
    var xmlSerializer = new XmlSerializer(typeof (Person));
    using(var reader = new StringReader(personXml))
    {
	result = xmlSerializer.Deserialize(reader) as Person;
	if(result == null) 
	    throw new SerializationException("Deserialization of Person failed.");
    }
    return result;
}
public static string SerializePersonToXmlString(Person person)
{
    var xmlSerializer = new XmlSerializer(typeof(Person));

    using (var stringWriter = new StringWriter())
    {
	using (var writer = XmlWriter.Create(stringWriter))
	{
	    xmlSerializer.Serialize(writer, person);
	    return stringWriter.ToString();
	}
    }
}

The only problem with this implementation is that the serialization step injects an xml declaration and xml namespace which results in something like:

<?xml version="1.0" encoding="utf-16"?><Person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd=http://www.w3.org/2001/XMLSchema><FirstName>John</FirstName><LastName>Smith</LastName></Person>

So let’s tackle the xml declaration first, in order to remove it we want to specifically omit it using XmlWriterSettings OmitXmlDeclaration. That will require that we use one of the overloads for the XmlWriter.Create method which accepts the settings object.

Now let's remove the namespace, this takes two steps. First we need to add an XmlRoot attribute to our object. I chose something like [XmlRoot( Namespace = "")]. Then we create a XmlSerializerNamespaces and pass it into our Serialize call. The method would then look like:

public static string SerializePersonToXmlStringFragment(Person person)
{
    var xmlSerializer = new XmlSerializer(typeof(Person));

    var settings = new XmlWriterSettings { OmitXmlDeclaration = true };
    var ns = new XmlSerializerNamespaces();
    ns.Add("", "");

    using (var stringWriter = new StringWriter())
    {
	using (var writer = XmlWriter.Create(stringWriter, settings))
	{
	    xmlSerializer.Serialize(writer, person, ns);
	    return stringWriter.ToString();
	}
    }
}

This results in what I was looking for a nice compact XML representation.

<Person><FirstName>John</FirstName><LastName>Smith</LastName></Person>

Enjoy!

.Net, C#, XML

I was reading The Clean Coder: A Code of Conduct for Professional Programmers recently and ran across the concept of coding katas. This isn't something I have come across before so I took a closer look. I like the idea, I think it is a great way to asorb concepts and refine the craft of software development. I have adapted one of the authors katas and made it available. Enjoy!

BowlingScoreKata.pptx (350.11 kb)

.Net, C#, Development Process, Agile, kata c#, development, kata