The last few weeks have been "interesting" for me. I have some problems with my right shoulder causing me a great deal of pain and immobility.
But nothing comes from nothing, and not being able to use my right hand for much led me to write a typing trainer to help me learn to type with my left hand.
The tool had several keyboard layouts including a left-hand only layout.
Comments/improvements welcome.
Thursday, 27 November 2014
Tuesday, 18 November 2014
Big Data is not just about size...
The term "Big Data" gets used a lot these days, and some people think you only have Big Data problems if you have billions of records and petabytes or even zettabytes of information in your system. In fact most articles I've seen start off by saying how much data modern systems have to deal with.
However Big Data is not only the domain of the googles of this world. You can have a Big Data problems in projects that don't have that much data.
In 2001 a paper on data management coined the 3 V's: Volume, Velocity, and Variety.
Volume is simply the size of the data, in raw bytes or number of records. A million records is not that many if they are simple and small, if they large and hard to index however multimillion record databases can become troublesome.
Velocity is the speed that the data is being collected. Website logs are a great example of this they can generate thousands and thousands of new records every hour even for relatively small sites.
Variety is the complexity of the data, anyone who has written SQL knows that even small databases can become sluggish if you try to link too many tables in a single query.
Today some people like to add other "Vs" to the mix:
Veracity is about biases and "noise" in the data, how meaningful is the method of analysis to the data. Any statistical analysis will have the odd result that falls way outside the line of best fit for the rest of the data. How often this occurs and how you have to deal with it can effect your data solution
Validity is similar to veracity and tends to deal with the applicability of the data to the question being asked, like web proxies and caches mean server logs are not a true reflection of actual page views.
Volatility may sound like speed but rather than how fast it's coming in, volatility deals with how fast the data gets out of date. Trading prices for shares are out of date almost before you get the data, with some companies doing micro trades and relying on millisecond differences in buy and sell times.
and finally...
Value is a measure of how much the data is worth in real world terms. This is often the most important factor because it drives the business.
Now if any of these factors reaches the point where it requires developers to introduce special strategies, processes, or systems in order to manipulate and store the data, then you have just crossed into the world of Big Data.
So the "big" in Big Data refers more to the requirements of the data than the data itself.
That's it, Big Data explained. Your welcome.
Now as far as solutions to the problems of Big Data...
You can take your pick because frankly there are as many Big Data solutions are there are Big Data problems.
The technologies that most projects encounter first are storage and retrieval specific, like clustering, sharding, distributed storage, cloud based infrastructure, search based applications, key-value stores and other NoSQL approaches to data storage.
Then as you approach the more complex tasks of analysis and manipulation you start to hit distributed computation, genetic and machine learning algorithms, signal processing, time series analysis, wavelets and other semi predictive techniques.
In the end Big Data is simple, it happens when the requirements of you're data application go beyond the systems you're currently using and the solution is that you need to evaluate those systems, starting with how you store and organise your data.
However Big Data is not only the domain of the googles of this world. You can have a Big Data problems in projects that don't have that much data.
In 2001 a paper on data management coined the 3 V's: Volume, Velocity, and Variety.
Volume is simply the size of the data, in raw bytes or number of records. A million records is not that many if they are simple and small, if they large and hard to index however multimillion record databases can become troublesome.
Velocity is the speed that the data is being collected. Website logs are a great example of this they can generate thousands and thousands of new records every hour even for relatively small sites.
Variety is the complexity of the data, anyone who has written SQL knows that even small databases can become sluggish if you try to link too many tables in a single query.
Today some people like to add other "Vs" to the mix:
Veracity is about biases and "noise" in the data, how meaningful is the method of analysis to the data. Any statistical analysis will have the odd result that falls way outside the line of best fit for the rest of the data. How often this occurs and how you have to deal with it can effect your data solution
Validity is similar to veracity and tends to deal with the applicability of the data to the question being asked, like web proxies and caches mean server logs are not a true reflection of actual page views.
Volatility may sound like speed but rather than how fast it's coming in, volatility deals with how fast the data gets out of date. Trading prices for shares are out of date almost before you get the data, with some companies doing micro trades and relying on millisecond differences in buy and sell times.
and finally...
Value is a measure of how much the data is worth in real world terms. This is often the most important factor because it drives the business.
Now if any of these factors reaches the point where it requires developers to introduce special strategies, processes, or systems in order to manipulate and store the data, then you have just crossed into the world of Big Data.
So the "big" in Big Data refers more to the requirements of the data than the data itself.
That's it, Big Data explained. Your welcome.
Now as far as solutions to the problems of Big Data...
You can take your pick because frankly there are as many Big Data solutions are there are Big Data problems.
The technologies that most projects encounter first are storage and retrieval specific, like clustering, sharding, distributed storage, cloud based infrastructure, search based applications, key-value stores and other NoSQL approaches to data storage.
Then as you approach the more complex tasks of analysis and manipulation you start to hit distributed computation, genetic and machine learning algorithms, signal processing, time series analysis, wavelets and other semi predictive techniques.
In the end Big Data is simple, it happens when the requirements of you're data application go beyond the systems you're currently using and the solution is that you need to evaluate those systems, starting with how you store and organise your data.
Thursday, 13 November 2014
Back in the saddle...
After what seems like ages I finally have the time to sit down and blog again without running out of time or covering topics that are sensitive to my clients... Lets hope I get a solid run before circumstances change again :)
Tuesday, 18 June 2013
Awesome 3d printed prosthetic hand!
Those who know me also know that one of my favourite gadgets is my 3d printer. It's old now and not up to the standards of todays models, but I still love and use it to build, invent, and fix things around the house.
When I got several years ago I kept trying to tell people how wonderful they were and what they could do for the world - Sometimes I got polite interest and other times I got blank indifference. People just could not think of anything they would do with one.
Today I saw one of the most awesome stories ever (on NPR).
Back in 2011 A carpenter in Johannesburg, South Africa lost several fingers on his right hand, he recovered and hooked up with a visual effects designer in the US (gotta love the internet right?) to build himself a prosthetic.
That alone is a fantastic story, but they got approached to build a hand for child born without fingers on one hand. They accept the job and built a prototype for the child out of machined metal. Then it hits them "Why not use 3d printing?"
Makerbot industries (one of the bigger players in home 3d printing) donated 2 printers to the project and our heros where able to design a hand that could be printed and assembled for about $150 in parts instead of the thousands that a normal prosthetic device would cost.
You can follow the project here or on the Robohand Facebook page.
At this point I was already overcome with awesome, but it keeps coming.
Because the design was released under creative commons, people could make their own improvements, and that what the guys at Makerbot did, they modded the design so it can be snapped together with no bolts, bringing the cost down to about $5.
I'll say it again:
This is not the first prosthetic designed to be made on a 3d printer - for children this type of thing is amazing because the designs can be resized every few months as the child grows without breaking the bank and the non printed parts can even be re-used. Also the printed plastic is much lighter than some of the metal alternatives, it wont last as long, but given the growth rates involved it doesn't need to.
For me a 3d printer is a way to fix blinds, kitchen fittings, dishwashers, and print up prototypes for ideas I have, but for some incredible people out there its a a tool they can use to make a massive difference in someones life.
When I got several years ago I kept trying to tell people how wonderful they were and what they could do for the world - Sometimes I got polite interest and other times I got blank indifference. People just could not think of anything they would do with one.
Today I saw one of the most awesome stories ever (on NPR).
Back in 2011 A carpenter in Johannesburg, South Africa lost several fingers on his right hand, he recovered and hooked up with a visual effects designer in the US (gotta love the internet right?) to build himself a prosthetic.
That alone is a fantastic story, but they got approached to build a hand for child born without fingers on one hand. They accept the job and built a prototype for the child out of machined metal. Then it hits them "Why not use 3d printing?"
Makerbot industries (one of the bigger players in home 3d printing) donated 2 printers to the project and our heros where able to design a hand that could be printed and assembled for about $150 in parts instead of the thousands that a normal prosthetic device would cost.
You can follow the project here or on the Robohand Facebook page.
At this point I was already overcome with awesome, but it keeps coming.
Because the design was released under creative commons, people could make their own improvements, and that what the guys at Makerbot did, they modded the design so it can be snapped together with no bolts, bringing the cost down to about $5.
I'll say it again:
$5 FOR A PROSTHETIC HAND FOR A CHILD!!!I love my 3d printer and I love that people design things for themselves and then make the intellectual property open to benefit others.
This is not the first prosthetic designed to be made on a 3d printer - for children this type of thing is amazing because the designs can be resized every few months as the child grows without breaking the bank and the non printed parts can even be re-used. Also the printed plastic is much lighter than some of the metal alternatives, it wont last as long, but given the growth rates involved it doesn't need to.
For me a 3d printer is a way to fix blinds, kitchen fittings, dishwashers, and print up prototypes for ideas I have, but for some incredible people out there its a a tool they can use to make a massive difference in someones life.
Getting it wrong the right way!
One of my regular subscription emails turned up today open with the words:
What really caught my eye was the message I got 4 hours later:
It really highlights that there is a right way to fail, especially when people will know that you messed up... trying to hide it at that point would just make you look dishonest (and yes I've seen people try it).
Dear {FIRST_NAME|Colleagues and Friends},Like most people, I've seen this happen more times than I can count (and yes I've even caused it once or twice). With the best intentions in the world mistakes happen especially when transitioning between mail providers/software.
What really caught my eye was the message I got 4 hours later:
Dear Ben,There message was short and simple: "it was human error, we're sorry." This is the right way to deal with a mistake. By taking control of it they stop (or at least reduce) complaints and demonstrate that they have already caught the mistake and corrected it.
You will have just received our XXXXXXX XXXXXXX Newsletter which has been sent out with an obvious error...
It really highlights that there is a right way to fail, especially when people will know that you messed up... trying to hide it at that point would just make you look dishonest (and yes I've seen people try it).
Wednesday, 12 June 2013
Use Gmail aliases to keep your inbox clean
If you use Gmail and you're active on Google+, Facebook, LinkedIn, Twitter, etc then you could be getting flooded with notifications you rarely read that make it hard to find the important stuff in your inbox.
You could just turn off notification but sometimes you dont want to, you just want to keep it to one side while you read the other inbox stuff. In that case you should be using gmail filters.
The only problem with filters is that you have to figure out what to filter on, sometimes it's easy and you can use the sending address or a subject, but thats not always a perfect fix.
A better solution is to use an email alias! Configure your notifications to go to the alias and then build a filter based on the to address of the email.
If you're not familiar with this Gmail alaises then you should be, its one of those must-use features when signing up for services, notifications, and subscriptions:
Google doesn't offer a traditional alias function so you dont get to just pick 5 or 6 variations of your name, but you can add as many suffixes to your regular email account as you like making as many different email addresses as you need - best of all they always make their way to your inbox without you even telling google about them.
Say for instance you gmail account is john.doe1234@gmail.com and you want an alias set up to use for your daily new subscription.
You add something like +news to the name part of your email so it becomes: john.doe1234+news@gmail.com and thats it! Anything after the plus (assuming it's still allowed characters) will get through to your account.
Because you can have multiple such alaises you could create one for each service you use:
If you want to get creative you can use it with friends and in business by creating different emails that will automatically get tagged for action:
This is going to be old hat to many of you but it's really powerful and worth sharing with your non-tech friends to help them sort their mail for easy processing.
You could just turn off notification but sometimes you dont want to, you just want to keep it to one side while you read the other inbox stuff. In that case you should be using gmail filters.
The only problem with filters is that you have to figure out what to filter on, sometimes it's easy and you can use the sending address or a subject, but thats not always a perfect fix.
A better solution is to use an email alias! Configure your notifications to go to the alias and then build a filter based on the to address of the email.
If you're not familiar with this Gmail alaises then you should be, its one of those must-use features when signing up for services, notifications, and subscriptions:
Google doesn't offer a traditional alias function so you dont get to just pick 5 or 6 variations of your name, but you can add as many suffixes to your regular email account as you like making as many different email addresses as you need - best of all they always make their way to your inbox without you even telling google about them.
Say for instance you gmail account is john.doe1234@gmail.com and you want an alias set up to use for your daily new subscription.
You add something like +news to the name part of your email so it becomes: john.doe1234+news@gmail.com and thats it! Anything after the plus (assuming it's still allowed characters) will get through to your account.
Because you can have multiple such alaises you could create one for each service you use:
john.doe1234+news@gmail.comThe best part of using an alias is that if you unsubscribe you can just change the filter to delete emails coming into that alias - ideal for those times when a website "forgets" that you don't want to see their emails any more. Also if a company does sell you email for spam you'll know, because the spam will come to that email alias.
john.doe1234+twitter@gmail.com
john.doe1234+facebook@gmail.com
john.doe1234+linkedin@gmail.com
john.doe1234+googleplus@gmail.com
etc,
etc,
If you want to get creative you can use it with friends and in business by creating different emails that will automatically get tagged for action:
john.doe1234+urgent@gmail.comIt's easy to set up a Gmail filter:
john.doe1234+party@gmail.com
john.doe1234+biz@gmail.com
john.doe1234+family@gmail.com
- Click the gear icon in the top right of the Gmail window
- Choose "settings" from the menu
- Go to the "Filters" tab
- Click "Create a new filter" (it's near the bottom of the page)
- Enter your chosen alias in the "To" field
- Click "Create filter with this search"
- Check the boxes according to your needs (I normally go with "Skip the inbox" and "Apply the label [...]")
- Click the "Create Filter" button
- Finished!
This is going to be old hat to many of you but it's really powerful and worth sharing with your non-tech friends to help them sort their mail for easy processing.
Real programmers are not born they are SELF-made
There must be a thousand blog posts out there saying what makes a good developer or how to train developers or how a developer should improve their skills, etc, etc... But no matter how many of them I read I always come back to one basic requirement for a good developer - They have to want to be a developer.
There is no way to over stress the importance of choice and motivation, you can put dollar signs in front of someone and they will study programming at uni and get a job, but a job is all they will ever think of it as. Real programmers think of it as a calling, it's not what they do its what they are.
I've worked with certifiable geniuses who were incredibly knowledgable and competent developers, but they were never going to be the best developers out there because it was a only a means to an end. Programming was something they did for a paycheque or to facilitate another project that actually held their passion.
A real developer (and I realise people will challenge my use of that term) is someone who writes a program to see the program run, not because they value the output.
If you have that drive, if you have the will to program for its own sake, if it's as much an interest, hobby, and life style as it is a job. Then you are a real developer - You will seek out and read the blog posts on what training you should have, on what books you should read, on what skills you should develop because you want to be trained, you want to read and you want to have the skills.
A real developer (and this applies to almost any field) is someone with the passion to drive their personal development, their career path and their life along the path towards the goal of being a developer first and foremost.
I can compress every "real programmer" test on the internet into a single question:
If you answer "No" then maybe programming is just something you're doing until something better comes along.
There is no way to over stress the importance of choice and motivation, you can put dollar signs in front of someone and they will study programming at uni and get a job, but a job is all they will ever think of it as. Real programmers think of it as a calling, it's not what they do its what they are.
I've worked with certifiable geniuses who were incredibly knowledgable and competent developers, but they were never going to be the best developers out there because it was a only a means to an end. Programming was something they did for a paycheque or to facilitate another project that actually held their passion.
A real developer (and I realise people will challenge my use of that term) is someone who writes a program to see the program run, not because they value the output.
If you have that drive, if you have the will to program for its own sake, if it's as much an interest, hobby, and life style as it is a job. Then you are a real developer - You will seek out and read the blog posts on what training you should have, on what books you should read, on what skills you should develop because you want to be trained, you want to read and you want to have the skills.
A real developer (and this applies to almost any field) is someone with the passion to drive their personal development, their career path and their life along the path towards the goal of being a developer first and foremost.
I can compress every "real programmer" test on the internet into a single question:
Q: If you won the lottery today and would never have to work again as long as you live, with enough money to do everything you ever wanted - would you be writing code in 12 months time?If the answer is "Yes" then you are a real programmer. You program for the joy of the activity as much as the outcome.
If you answer "No" then maybe programming is just something you're doing until something better comes along.
Subscribe to:
Posts (Atom)