Natural Language Processing, Health Records and the Developing World.

Case Investigation Team

The Veterans Administration will be using Natural Language Processing (NLP) for their medical records. It can be a powerful tool for searching for trends and getting the right people to the right treatments in a timely manner. That’s a gross oversimplification.

I know a bit about medical records1. I also happen to know quite a bit about Natural Language Processing, since I’ve worked with it in the context of documentation management.

And, as it happens, I know a bit about the developing world – the Caribbean and Latin America. And I know a bit about the hospitals in the region, where hand written records are kept, but lack the rigor and discipline necessary for them to truly be useful. I recently looked at the medical record of someone in Trinidad and Tobago, if you could call it that, since I found it odd that the Doctors and Nurses didn’t seem to communicate not only with each other but their own subgroups. I saw why.

I know of one doctor who keeps patient records in Microsoft Word documents – a step in the right direction.

There is an opportunity here for the developing world in general, but it’s a technology leap that must be undertaken with the discipline of good medical records in the first place. These delapidated medical systems, despite new buildings, need to have medical records that enable good care in the first place.

There’s no reason that medical care in the developing world should suffer; it can be done much more cheaply than in the developed world and with the advancements such as NLP already being implemented, it’s vacuous to build shiny buildings when the discipline of the medical records themselves should be paramount.

But then, maybe implementing electronic medical records properly would be a good start to building that discipline. 

1Medical Records have interested me from my days as a U.S. Navy Corpsman, where we were assiduous about medical records – Doctor’s orders, nursing SOAP notes, lab results – all had their place within a folder. It was just on the very edge of the medical databases that the U.S. Navy rolled out. When I was at my first USMC command, myself and other corpsmen’s first job was  to get the medical records ready enough to allow us to deploy – and it was an onerous task, with those who had gone before not having taken the records as seriously as they should. Later, I would work with a Reserve USMC unit at Floyd Bennet Field where I would be commended for my database work as related to their medical records.

The AI Future On Mankind’s Canvas

Doctor Leia.I met her and the young Brazilian woman on the flight from Miami to Orlando, this young Doctor who had an interview in Ocala. She was to drive across to Ocala, to the East, to see if she would get the job. She didn’t look old enough to be a Doctor, but I’ve passed the age threshold where doctors were younger than myself years ago. We talked about medicine and medical administration for a while even as I checked up on the nervous Brazilian high school graduate. I sat, a thorn between two roses, all the while thinking:

What sort of world were they entering? Doc Leia, a graduate from The University of the West Indies, off to Ocala, and the young woman to my right, off to see the sights as a reward for having survived so many years of schooling. They were both easily younger than most of my nieces. The Doctor had already become heavily invested in her future – medical school was a daunting path and might have been one I would have pursued with the right opportunities. The other was about to invest in her future and it bothered me that there wasn’t as clear a path as there used to be.

Artificial intelligence – diagnosing patients on the other side of the world – is promising to change medicine itself. The first AI attorney, ‘Ross’, had been hired by a NYC firm. The education system in the United States wasn’t factoring this sort of thing in (unless maybe if you’re in the MIT Media Lab), so I was pretty sure that the education systems in the Caribbean and Latin America weren’t factoring it in. I’ve been playing with Natural Language Processing and Deep Learning myself, and was amazed at what already could be done.

The technology threat to jobs – to employment – has historically been robotics, something that has displaced enough workers to cause a stir over the last decades – but it has been largely thought that technology would only replace the blue collar jobs. Hubris. Any job that requires research, repetition, and can allow for reduced costs for companies is a target. Watson’s bedside manner might be a little more icy than House, but the results aren’t fiction.

What are the jobs of the future, for those kids in, starting or just finished with a tertiary education? It’s a gamble by present reckoning. Here are a few thoughts, though:

  • A job that requires legal responsibility is pretty safe, so far. While Watson made that diagnosis, for legal reasons I am certain that licensed doctors were the ones that dealt with the patient, as well as gave the legal diagnosis.
  • Dealing well with humans, which has been important for centuries, has just become much more important – it separates us from AI. So far.
  • Understanding the technology and, more importantly, the dynamic limits of the technology will be key.

Even with that, even as fast food outlets switch to touchscreens for ordering their food (imagine the disease vectors off of that!), even as AI’s become more and more prominent, the landscape is being shaken by technology driven by financial profit.

And I don’t think that it’s right that there’s no real plan for that. It’s coming, there is no stopping that, but what are we as a society doing to prepare the new work force for what is to come? What can be done?

Conversations might be a good place to start.






Reinvention, Recursive.

Art evolvesWarning: This is kind of long and is a rant-ble. The short of it is that I’m not on the market anymore.

It’s time to evolve again.1

No, this is not the announcement of some Silicon Valley startup that will make you better elbows to stick in your ears or, heaven forbid, something useful.

No, this is about the site, myself, and the career path. To cut to the chase, I’m no longer looking for work or contracts in technology.

There’s a few reasons for this.

  • After 2 and a half decades, it gets boring when done right and annoyingly exciting when done wrong. More often than not in most companies, it’s being done wrong and it’s no fun getting excited for the wrong reasons.
  • Everyone wants a specialist and I’m a generalist.
  • Management doesn’t like me wandering around outside the building. They don’t think I’m working just because of the GIS coordinates of my body during thought.
  • AI is gonna take over at least some programming jobs (advances in programming in the past have had the reverse effect, broadening the field – something else for another time). It will only take one programmer who will because s/he can, and then an ecosystem to evolve it.
  • Did I mention I’m bored?
  • I have other options.

Plugging tech together can only be done in so many permutations. It’s a mathematical fact if you factor in that the geometric progression is necessary for evolution through the permutations.  

I’m not sure I like how the ecosystem is plugging tech together. Frankly, while it’s nice that the iFart application created a few jobs (don’t be the guy with the microphone), and while it will be seen as invaluable to those who pay for it, it’s crap and really doesn’t advance anything but a paycheck. Because, really, money got mistaken for something of value somewhere in the history of mankind.

Because I don’t like the way things are getting plugged together, to work means to evolve again, and the value of working on things I increasingly don’t like is… silly in a human and financial perspective. I’ve always believed that people should do what they want to, then later understood that people should do what they want to only if they’re good at it. I’m still good at it, but I don’t want to think about that too much.

There are other things I’m good at, and it’s time to go do them. It’s not that I’m becoming a Luddite – far from, you should see this heap of silicon I just bought – but that it’s not a career for me, at least for a few years. I’ll be using tech in other endeavors, and a great way to spend time waiting on others is to solve problems: Write code, design systems, or make a better mousetrap. But it’s not my main thrust, and oddly, I’ve been telling kids starting college not to do tech but to do other things with tech.

And in the meanwhile, things that I put my own sweat equity into over 5 years ago are paying, and require some attention.

1 Now there’s a marketing line…

Pulling My Photos From Flickr: Lessons Learned, More.

nsb SunriseI wrote ‘Risk and the Photo Cloud‘ as a first stab at identifying the problem I am having with mitigating risk with my Flickr stream as well as new needs I have from my collection of images.

Bear in mind, this is a ‘one off’ problem for me, and I approached it as such.

I was a bit more focused on what I wanted to do with them – I have people willing to buy prints – and I may not have made the best choice by paying $25 for Bulkr Pro. The trick was to get the tags, etc, and in doing the initial research on the Flickr forums, I saw nothing better. In writing this today, I found FlickrDownloader which I probably would have chosen given that it’s open source, available at no cost, etc. I may give it a spin anyway. Try it out and let me know how it differs. 

When using Bulkr Pro,  you can opt to have the full information being saved to text files.  This is generally not a bad idea, but it’s certainly not a great idea when you have over 19,000 files on Flickr as I do (only 18,000 or so are public). So, the general idea is to get them into a database.

I opted not to directly import them into the database after some thought because I need to do some manual editing of which files I want to keep, etc – and in doing that, a spreadsheet would be easier for that part of the process. So that’s what I did.

The text files generated by Bulkr are mildly annoying in this regard because their format isn’t meant for what I wanted to do. However, the line numbers for the specific information is constant, as is the labeling, so I was able to whip together a Python 3.x script that reads all the text files and writes them all to a CSV. I tossed it up on my GitHub; you can check out

So now, I have the information in a CSV and, as time permits, I’m working on choosing which images I am getting rid of (with Flickr, I was lazy about space). Let’s just be clear and call it ‘dirty data’; when I uploaded I did it as a hobby and I now have a professional use.

From there, it’s a simple matter of uploading the CSV to MariaDB, which is extremely fast.

Having spoken to a few professional photographers I know, there are professional tools out there that do what I want with image management, etc, but I’m not too pleased with them. I may end up spinning my own image and file management system in Python, or not – I’m undecided at this point because I’m moving from a ‘one off’ to a more consistent system that will suit my personal needs.

Why Python? Honestly, I like coding in Python, but the job market has always been more interested in my C, C++, C#, VB, VB.Net, PHP/MySQL, etc. This is a low priority project for me, and it’s something I want to be fun while getting used to Python 3 a bit more. 

At this point, the system I’m thinking of will create resized images for the web, as well as allow me to edit based on tags. Pillow will allow for much of that, and more.

The Why and Why Not Of Documentation

FAIL stampMy core pet peeve of the last 10 years is the lack of documentation I’ve encountered with Software Projects. Sometimes it’s as the consultant that’s called in after someone already charged a lot of money for a substandard job. Sometimes it’s that legacy project that sticks around through generations of developers who leave a company. Sometimes it’s the “we don’t have time” factor.

Documentation is a value-added part of any software project for a variety of reasons:

  • Reference: The ability of software engineers involved with a project to look back and see why things were done the way they were, and to allow for knowing the limitations and potential of a project.
  • On-boarding: Bringing new software engineers up to speed quickly without having to bounce around the company finding answers.
  • New Projects: Proper documentation of old projects, including estimates and other data, is of great use for estimating similar projects. (Project Management and Business Analysis)
  • Complicated code reviews are simplified.
  • If someone gets hit by a bus, someone else can take over. Quickly.
  • Legal: If a project is supposed to meet certain requirements, those requirements should be traced to the actual implementation.
  • Teams: Proper documentation in an Agile or DevOps process allows team members to see what others are doing in similar areas of code, and allows easy identification of problem areas quickly.

These are just some of the reasons documentation is important. More often than not, I’ve ended up writing documentation at different companies because it simply did not exist before.

Why didn’t it exist? Here are the standard excuses:

  • No Time: We didn’t have any time to write the documentation because we’re busy working on the next thing!
  • “Me no write so good”: Fair enough, but there’s only one way to get better at writing.
  • “It’s not my job, man”: Erm – no. Documentation is a part of software engineering. A big part.
  • We don’t know how to organize it: Fair enough. Spend some time and figure it out and implement it.

At the base of all the excuses are 2 things that are really simple:

  1. Software Developers generally don’t like writing documentation.
  2. Management fails to have developers write documentation.

These 2 things are understandable, wrong, and amazingly easy to deal with if a company wants to.

How To Hire A Software Developer (2012)

Drupal Code MonkeyOne of the fun and interesting things that has been cropping up with some job interviews for contracts and employment is the potential employer wanting to know whether someone is actually a good software developer or programmer. The trouble is that there are flaws in just about any way of doing it. Having been on both sides of the desk, I’ve helped hire good programmers and bad – even over the phone – and I’ve interviewed for positions that fit me like a glove or didn’t.

Here’s some things I’ve seen recently – but if you want, just skip to the ‘How To Hire A Good Software Developer‘ part at the bottom.

How Did You Learn?

This is a legitimate question for anyone without a year of experience – to be safe, lets say 2 years. I recall interviewing for a position around 2003 in Trinidad and Tobago – a position for some oil related company about a VB 5.0 position – and I was asked this. With close to two decades of experience on my resume at the time, I explained that it was learned on the job and that I had references on it. But that’s not what they wanted. That people learn stuff in the real world seemed to escape them.

Recently, I applied for an Open Source Consultation position. It’s a natural position for me, really – open source advocacy, familiarity with open source projects and not just how the code works but the communities around them as well as their directions. I’ve even spoken at 4 open source conferences I can recall off the top of my head (not counting smaller group meetings). But somehow, someone in that company thought… If they don’t have a degree in something, even underwater basket weaving, they can’t do the job.

Does a  degree actually help with programming? I’ve seen it go both ways. I’ve met great software developers – yes, some better than me, more than I might like to admit – but whether they had a degree or not wasn’t material. I’ve also met some really poor programmers who had great GPAs. Poor programmers without degrees don’t exist. Why? The College of Experience is not as easily gamed.

Degrees don’t matter unless it’s an entry level position – and entry level positions are high risks for both employers and employees anyway.

Code Samples

There are a few companies out there that ask for code samples and, really, that might be a good way to judge how someone wrote code when they wrote the code – with or without help on a project that may or may not pertain to what the potential employer needs to have done – as opposed to what they may actually advertise for the particular position.

Another problem – one that I run into – is the ever present NDA. Believe it or not, there are companies out there that don’t seem to understand that work done these days is almost always covered by an NDA. Unless you contribute code regularly to open source projects, this may be a pit of despair. My thought: If the hiring company doesn’t understand NDAs, maybe they haven’t worked on things that work with business intelligence.


When Jeff Atwood wrote ‘Why Can’t Programmers… Program?”, the Fizzbuzz test became popular with employers unable to design their own tests. A mind familiar with solving academic problems does well with these things. Not all programmers are familiar with solving academic problems. Recently I was asked to write some code that showed the first 10 prime numbers – and in trying to recall the first 10 just as a quiz for myself, I missed a few. I did do the code on the whiteboard, but hey, the point is – the muscles exercised are the strong ones. In programming since the 80s, this was the first time someone thought the prime numbers were important enough to write code for.

Code testing is subjective. This is as it should be. But if you want to do some code testing with a candidate, why not make it subjective to the job itself? Factor in that problems as famous as Fizzbuzz can also be crammed for. Really. Just Google interview code question. Rather than throwing Fizzbuzz or an analog of it out there, why not have a test that relates to the business in the position being filled? That might be a good test. In fact, that should be the test. The candidate should be able to tackle a problem that relates to the position being filled.

What’s special about the position? Test that. It typically includes the skills that you need tested anyway.

The other part of this deals with attitude. Throw a real problem at a real software developer and you’ll get to see how they really think, code and how they approach that problem. People don’t always approach the same problem the same way and that has benefits. The interviewers have to be able to understand that. When trying to make a good first impression sometimes people don’t.


Personality tests are used here and there and the results are almost never released to the candidate. As someone who has tested consistently as an INTJ1 for decades, I’ve made it a hobby. To an INTJ, a hobby is a serious thing. I don’t claim to be an expert on personality tests but I’ve read a lot on the topic and have taken a lot of tests.

Some things to remember about personality tests:

  • They are snapshots. A personality test is affected by ‘where’ the person is when they are taking the test; stress and other factors change the results.
  • They are incomplete.
  • They create a great industry for people who make money testing personalities and telling you that you need to do them.

How To Hire A Good Software Developer

So here it is. The magic of hiring a good software developer can be summed up in point form.

  • Why do they want the job? Save the ‘World Peace’ beauty contestant speech. If a software developer has everything you need for the position, they can’t grow. A good software developer grows. If a developer has nothing to aspire to then they aren’t a good developer.
  • Do they understand your business? A developer who understands the business, even in the broad strokes, is better than someone who can do the Fizzbuzz test in one line (yeah, it can be done in one line in at least one language I know of). Some of the hardest and best interviews I’ve had involved ‘walking the floor’ and being shown the business and the role the developer would be playing within it.
  • Can they problem solve? The best way to check is to go over problems related to the business. You could hand them a Rubik’s cube – if your business is about solving Rubik’s cubes. You could have them write code, too, but try to make it at least related to the job.
  • Interviews can be nerve-wracking for people on both sides of the desk. A good interviewer harnesses this and can choose to disarm or build pressure – typically both. This is why some companies play ‘good cop, bad cop’ in interviews – but the interviewer has to understand people to do any of this. If you’re pulling someone kicking and screaming away from their keyboard and their social skills are limited to Internet interactions, their body language will likely throw things off. If the interviewer doesn’t interact with people on a social level very well… things can go bad.  And if the software developer being interviewed is a social misfit, the question is whether this misfit will fit in with the other misfits.
  • Curiosity. Good developers are curious.
  • Thoughtful – not in the ‘brings flowers to funerals and weddings’ thoughtful, but a good developer will sometimes take the time to think through something rather than rushing off on tangents. Rushing off on tangents is typically what less experienced developers do.
  • Confident. This is not to be mistaken for false confidence.
  • Interested. If they’re yawning, they’re not interested.

Clearly, everyone will approach hiring a software developer differently and it’s no question that they get different results. The trick is not finding the perfect software developer.

The trick is finding the right software developer.

Feel free to comment!

1Myers-Briggs Type Indicator and Socionics, thank you very much.

On Quality In Production Code (2012)

So I’ve been playing StarCraft II: Heart of the Swarm

Once upon a time in a universe resembling this one at a younger stage, I dreamt of game programming. I don’t think that there’s any coder of my generation that didn’t have similar aspirations – perhaps getting published in Compute! magazine, perhaps getting one’s name on the box, or maybe even just writing a really cool game. I stand with the majority of developers who have never done that sort of thing. Times changed.

Consider what the person who wrote Galactic Adventures and Galactic Gladiators had written on the Well (apparently before they thought of putting a name to it):

To me, the most amazing thing about doing these games (and I’m about to date myself) is that I did virtually everything myself. I designed the games, programmed them, drew the graphics (quite primitive), wrote the rule book I did get help from friends and people at SSI in testing and making suggestions. Compare that with today where new games are developed with a million dollar budget and use teams of 30 or more people, programmers, artists, tests, story consultants, etc, etc. And you know what — the games of today are better (Gee, why is that a surprise?)

With that in mind and considering that I knew about Blizzard when the original Warcraft was a freeware game where you paid a bit more to get a full version – no, not the MMORPG you kids play now, but the Warcraft that started a new genre that Starcraft continues – Blizzard has come a long way. Since then, Battle.Net has come online. Since then, numerous titles have come through Blizzard and, while I haven’t played Diablo or messed with World of Warcraft, not one title hasn’t been popular. Blizzard has a kind of magic that works for them. I just like a particular genre and have found MMORPGs to be disappointments when it comes to content; I dislike leveling a character by what is properly termed grinding.

All of that said, I’m sure the budget for the Heart of the Swarm expansion was over a million dollars. I’d wager that it was at least 10 times that amount. The detail is great, the storyline is good (though how a prisoner in a high security prison has a pistol escapes me) and it’s playworthy – though I keep mixing up Q and A on the keyboard. But here’s the thing: I played through the game that took so long to develop in about 8 hours of game time. Hard difficulty, much the same. Brutal – well, that’s still in progress. There’s always the PvP maps and the challenge maps. I figure that, all things considered, the game will last me well over 1,000 hours of playtime.

It sold 1.1 million copies in 48 hours. At $40 per expansion, that’s $40.4 million in 48 hours.

Yeah, we’ve come a long way from typing in games from magazines. We’ve come a long way from a single developer writing a game. There was a magic to that era, though – we didn’t yet expect a great game. We expected a game and hoped it would be good. Now people get their sodden underwear twisted when they start talking about game balance and other things on the forums. As a fuddy-duddy, I see that as an odd form of entitlement – people expect a lot more for $40 and have the privilege of time to gripe about it in poor writing.

Game development has come a long way. Humanity? Not so much.

Thank you, Blizzard, for what I consider to be another great game. As a developer, I wish I had a hand in it. I’ll wait for the Protoss section. Take your time.