CYA security

I read this article when it came out a few years back;

http://www.schneier.com/blog/archives/2007/02/cya_security_1.html

Really good coverage on why the existing security measures where ineffective against the threats encountered until *after* the threat had already been encountered, which is too late.

reading this article now:

http://www.rollingstone.com/politics/story/26793903/the_big_takeover/print

reminded me of it, especially this bit:

"In the final three months of last year, (AIG) lost more than $27 million every hour. That's $465,000 a minute, a yearly income for a median American household every six seconds, roughly $7,750 a second. And all this happened at the end of eight straight years that America devoted to frantically chasing the shadow of a terrorist threat to no avail, eight years spent stopping every citizen at every airport to search every purse, bag, crotch and briefcase for juice boxes and explosive tubes of toothpaste. Yet in the end, our government had no mechanism for searching the balance sheets of companies that held life-or-death power over our society and was unable to spot holes in the national economy the size of Libya (whose entire GDP last year was smaller than AIG's 2008 losses)."

So ok, terrorists haven't managed to fly anymore planes into buildings since 9/11, thats a good thing. In the meantime personal liberties in western countries have been eroded by an enormous degree, the US has invaded a country (that still shocks me!) on false pretences, and become a party to illegal detention and the use of torture. What price your soul?

At the same time, the greatest threat to US (perhaps the world?) National Security since the end of the Cold War has been brewing, in the form of the current freeze-up of the financial markets triggered by the sub-prime mortgage collapse. So we successfully defended against future repeats of the last threat (arguably some might say), while completely missing the next great threat. If you tally up financial cost and increased mortality due to financial-related suicide & murders during this period, I dare say the cost in dollars and human lives will outstrip 9/11.

Yes we need to guard against attacks similar to what's been encountered in the past, but we also need to have open eyes in seeing dangers from new and unexpected directions. Prior to 9/11 there was intelligence about a potential attack using planes that was ignored, leading up the current crisis there were warning signs that were ignored. We need to stop ignoring good intelligence just because we haven't seen it before.

Sheeple or the wisdom of crowds?

This is disturbing:

http://brainz.org/ten-most-revealing-psych-experiments/

But on the other hand,what of the wisdom of crowds?

http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds

Whats going on here? It seems that on the one hand individuals can be easily manipulated using peer pressure from groups, and on the other crowds seem to make better decisions than individuals? Perhaps these are complementary points of view, not contradictory? Perhaps the conscious action of the majority of people can be easily swayed, but the unconscious action of groups (displaying emergent behavior?) tends to cluster to optimal points of maxima far faster and more accurately than the conscious efforts of experts?

Could it be that while human psychology is deeply flawed, the behavior that emerges from interactions between individuals and environments can be insightful?

How do we engineer society to avoid, or at least minimise the harmful effects of our own nature? Sometimes we need to follow the crowd, sometimes we need to ignore the crowd and travel against the flow, how do we decide when to do which?

A lot of questions, no real answers.

(edit)
Coming back to this post later, it occurs to me that perhaps we should observe the crowd and be aware of what is causing it to cluster or disperse, but not necessarily follow the crowd. Know thyself...

The joy of refactoring

Imagine you have an infrastructure that allows you to run batch jobs asynchronously. However, you can't specify exactly when you want it to run (à la cron) but rather all you have is a polling interface, so your "service" will start up every x mins, do something, then sleep again. Finer grained control is up to you.

This is our situation, and we usually specify a start time, eg - "23:00", an end time "24:00", and have a bit of logic somewhere to abort the run if the current time is outside that window. I finally noticed the bit of logic that does this, and realised that it was duplicated in every service has a *copy* of the same code. behold:



Yuck.

Needless to say, I didn't like the duplication, or the code itself. I didn't have scope to retrofit the other services that used similar code, but I could fix this one and build a foundation for later.

First step was to refactor that mess out to a common util class, leaving us with this:


...
and this:



The first cut was just a raw refactor extract, the next step was to not use ugly Calendar code and handle the midnight cutover correctly (which is actually quite hard!). After a few false starts with Dates and dealing with GMT and daylight savings offsets, we ended up with this (convenience methods and javadoc not shown), which is unit-tested and guaranteed to work (except maybe during the daylight savings cutover itself..)



Much nicer, well we think so anyway.

If you can see a bug in there please let me know!!

The joy of Spring!

Had the opportunity to Springify a co-workers project while he was on leave the other day. It already sort-of used Spring, but he had inverted the inversion-of-control (yeah... think about it, passing the context everywhere and getting single beans out, scary) and because of the architecture we were variously running into "got minus one from read call" errors (Database-side connections exhausted) or "No ManagedConnections available within configured blocking timeout" (JBoss Connection pool exhausted) depending on whether you used his oracle config (multiple connection pools with max-size of 100) or mine (max-size 10 per pool).

So! Time to throw out cruft, delete code and do proper dependency-injection! always great fun. I once did a project where most of the risk was in the business algorithm, but the data requirements started fairly simple. So I rolled my own database layer and got going. Later in the project the data requirements started to grow, transaction demarcation and correct handling became an issue so I took the time to Spring-ify it, and the joy of going from raw JDBC sql to JdbcTemplates and Hibernate HQL was huge! dozens of lines of verbose JDBC try-catch-finally evaporated to one-liners. just great!

I do love taking well-written OO code and re-wiring it with Spring, you can throw out huge amounts of code and eliminate bugs that you might have never known about! Connection pooling issues, correct rollbacks, host of junk like that. A lot of the ugly casting and boilerplate code just evaporates, and you're left with get-setters (which is another issue entirely, we should have properties!)

At the moment I tend to use a Builder pattern to boot-strap Spring from the init code (not a full spring stack yet...) and get a handle to a Spring bean configured with all the resources (it knows how to orchestrate the DAOs and usually contains most of the bus. logic) which then takes over processing. I could probably do better, but it works pretty well for the moment and is easy enough to unit test, so I'm happy.

No more garbage collection pauses!!

I was listening to a back episode of the Java Posse the other night from the '07 roundup "Whither java?" session (around 63:10), and heard someone mention the "-Xincgc" option for the Sun JVM that changes from the default collector with pauses and all, to an incremental collector.

This changes the behavior from big, ugly, noticeable pauses for garbage collection full sweeps to an incremental model where the pauses aren't noticeable, with the trade-off that it uses more CPU overall. So for batch-type, long-running CPU intensive operations the default collector will out-perform the incremental garbage collector marginally, but for user-visible operations the big noticeable pauses go away.

Technically, this forces the JVM to use the Concurrent Low Pause Collector, as documented in Tuning Garbage Collection with the 5.0 Java Virtual Machine. Interesting reading if you have the time.

Script it Script it Script it!

this has been said many times before, but I'm gonna say it again, anything worth doing twice is worth scripting!

In our environment the development databases are refreshed from production and sanitised weekly. This is scripted, it happens on the dot, every week, without fail (excepting catastrophies!).

All of our local builds are automated, type ant or mvn compile and magic happens.

All of our server builds are automated, svn ci causes a build to kick off automatically, and notifies you if something goes wrong.

The project I'm currently working on requires some manual configuration and then proceeds to turn the database inside out, and is not easily backed out (I've tried, and it was more pain than it was worth!). So for every full integration test there's about 8 separate configuration steps before it can be run. They've all been scripted (SQL in this case). Now I know I can refresh my database from production with sanitised data, run one script to reload the config, and run a full 4hour integration test, all with complete confidence that I'm running off a fully reproducible slate.

The first time you do something, fair enough, do it without regard to rigorous scripting (but save any commands you run). When you come to do it a second time, you have a hard decision to make. If there's even the remotest chance that you'll be called on to do this again, then script it. In fact, even if you probably think you won't have to, script it anyway. The number of times I've had to repeat the thing that "is surely just this once-off and never again!" and later wished I'd scripted it to start with.

I've found in my experience that the time it takes to script something, most of the time, is only slightly slower than the time it takes to do it ad hoc. When you script something you exercise the muscle between your ears, which is worth it if nothing else. Then as soon as you have to do it the second time, it pays for itself, and every single time after that.

I once had to step in for a client and run some web statistics for them as the usual staffer who performed the role was away. This normally took him the best part of a day to perform, between copying log files from the server, setting up the config and waiting for the tool to process the logs, then copying to results to the web directory. The first time I did it, I took about 2 days, maybe three, this time included figuring out the process as well as scripting it while waiting. The second time it took me the 5mins to update the links on the stats page to point to the new stats, as everything had run automatically before I got to work. Time well spent.

I'm sure I'm preaching to the choir here, and everyone reading this blog will be nodding their heads as if I'm ranting about how rocks fall down instead of up, but it's something we all too easily forget in the rush of everyday, to take the time to keep our minds and our tools sharp.

Got a large log file to parse(1GB+?)? Instead of wrestling with some editor, try writing a perl/python script to parse it for you, or even a shell script with a grep pipeline.

Let the machine work hard while you work smart. You'll save time, be more relaxed for it, and who knows, you might even enjoy it!

Two Random things I like about Java...

1) the fact that each and every class file needs to have 0xCAFEBABE in the first 4 byte positions

This actually identifies the file as a valid Java class file in addition to the file extension, no non-class file is likely to have this number in that location.

2 the fact that there is a class called java.text.DontCareFieldPosition!!

The name actually makes sense, it's used by the DecimalFormatter and is package-local, so clients of the API should never be aware of it, and is a null implementation of the FieldPosition interface. Cudos to the developer that argued that name past the API police!!

The importance of trim()... or how much I hate whitespace!!

So... One of my colleagues went on leave last friday, and Murphys law dictates that that is the perfect time for the application he maintains to suddenly and precipitously cease working in production. Of course this problem fell into my lap to resolve so I set to it around about lunchtime...

The situation is that a third party regularly sends us a variety of data in plain text or spreadsheet format, in this particular case it's spreadsheet. Once we get this, we read it off the filesystem, look up data from the spreadsheet and compare it to data in our systems and perform various actions based on it. This has been in production for about 8months and has worked perfectly the whole time. Until the maintainer went on leave... :-)

Digging in, the first problem was that it wasn't reading dates properly. Hmmm, ok, so we grab the last spreadsheet that worked and compare with that, ah, they'd changed the date format from YYYY-MM-DD to DD/MM/YYYY for no reason at all. Ok, not a biggie, one excel macro later it looks fine, reload but still no workie... what tha?

After more digging we were being bitten by Excel date handling. A cell may look like a date, but if the cell is formatted as a date then internally it's actually just an integer, which needs to be converted. In my checked-out version of code this was handled transparently, but in prod and QUAL this broke as numeric dates were being missed, resulting in nulls. Ok, another excel macro and we have columns of strings that look like dates instead of numbers that look like dates. Now loading this file into QUAL the dates are picked up fine, at which point I noticed that a lookup between the file and our system wasn't working. WTF mate? Each row has several cells that contain codes that need to be converted into rich data structures from a lookup, this data structure then being used for calculations later.

The lookup was broken, why? Well, after several hours of checking differences in environment, checking datasources, checking versions of code and finally in desperation stepping through the lookup code, it dawned on me... The spreadsheet had whitespace in it.

Normally when I read tabular data from disk, or really any untrusted data at all, trim()'ing and validation is the order of the day. If the field is an enumeration, then validate that. If it's a number, check for alphas, dollar signs, commas etc and so on. It's the sort of paranoia that maintenance programming gives you. I had ASSumed that my colleague would have done the same by default, oops no. Which is fine, we all make mistakes.

So, a quick google, this handy KB article and some VBA later, and the spreadsheet was all clean again. Re-running the load it came up roses, which is to say it started working like billy-o where it had previously just cruised through, and all was well with the world.

Moral of the story is, Excel is a pain to work with. Not because it's the spawn of pure evil, but rather because although it *looks* like a datasource with strong data rules, it isn't, and you can never, ever assume that anything inside it is what you think it is until you've checked and made sure. Columns will move around, dates will randomly be strings or numbers, any and every field will randomly be space/tab/zero padded. You simply have to treat it as a hostile datasource.

And never, ever fail silently when something is wrong with the data!! Die loudly and with great gusto!! Please for my sanity!!!

else
{
// No record exists may be something wrong with **** in the input file
record.setVaild****Prev(false);
}

java.lang.OutOfMemoryError: PermGen space

You know those things that don't crop up very often, but when they do they're a complete pain in the a**? well this is one of them!

Developing a plugin for Atlassian Jira, I don't care about heap size, or space allocated to PermGen, all I care about is getting functionality to the business.

I've read this:

http://wiki.caucho.com/Java.lang.OutOfMemoryError:_PermGen_space

and this:

http://www.jroller.com/agileanswers/entry/preventing_java_s_java_lang

among many, enough to know that PermGen isn't as simple as "make the heap bigger", enough to know that it's probably the fact that I'm running the app server in debug mode that contributes to this problem, but still, I'd rather not have to deal with it.

I understand that there's a lot of complexity involved in the Java Memory Model, I understand that people a lot smarter than me came up with it and decided on restrictions like the fact that the JVM can't give memory back to the OS once it's freed it, to make the problem of Performant Garbage Collection solvable, I'm cool with all that!

But... I just want it to work! I wish there was a --XX:PromiscuosMemory flag that I could set to say, hey, I'm developing here, this won't be used in production, I know what I'm doing, I don't mind if I take a bit of a hit on performance, but I would like you to eat memory as needed until you reach the system limit, running the garbage collector whenever you want, and you know, if you get a whole lot to spare? Just go on and give it back to the OS, thanks.

Of course that doesn't exist, so I find myself adding "-XX:MaxPermSize=512m -Xms128m -Xmx512m" to my startup script, and not caring about the memory model anyway. fun fun.

Using Reflection to get around hard-coded dependencies

Spring famously has no cycles in it's dependency graph:

http://www.springsource.org/node/310

Obviously writing applications is different to writing frameworks, and we can't always take the time to get things so polished. Also by the nature of the beast the domain model will (should!) pervade the entire application (domain model should be the root of your graph, with services/repositories/ etc depending on them).

Sometimes though you may want to enable some functionality if an object is an instance of some special class, without introducing explicit dependencies on those classes. reflection is your friend!!

Of course the price is that you loose type-safety, so be warned here be dragons!!

simply at runtime see if the string representing the class name is the class you want to twiddle, if it is, reflectively invoke your methods enabling additional functionality, otherwise continue on in the boring type-safe way that only deals with the interface or bace class you're dealing with. The key here is that by default you deal with a standard interface or base class (such as javax.activation.DataSource), and only if the instance of the object has the extra bits you want (eg my.cool.SuperSmartDataSource), then you selectively elect to do extra. Spring does this quite a lot with it's own DataSource wrappers internally.

handy sometimes...

Classloading in Application Servers: JARmaggedon!

original issue with trying to access the tomcat security principal in IMGWS, but since class was loaded from a sibling classloader, not a direct parent was unable to cast the object to the correct class to access it, even though debugging revealed it to be the correct object. Problem: class loaded from an unavailable classloader, so unable to use any methods or fields related to the class, have to break in through reflection.

current issue: upgrading JBoss 4.0.4 to JBoss 4.2.3, except that we use spring hibernate, and JBoss tries to be helpful by providing those jars as well, thanks JBoss, now I get

java.lang.ClassCastException: org.hibernate.search.event.FullTextIndexEventListener cannot be cast to org.hibernate.event.PostInsertEventListener

Solution? remove hibernate jars from jboss server/default/lib. this works ok in dev, but will probably break advanced jboss functionality which relies on those jars, lucky I don't use it... :-)

Our deployment scenario is amenable to not having spring/hibernate etc jars provided by app server, instead provided by each applications WAR. for us this works as we have one app per app server, but as more come online this may change. On the other hand, each application providing all it's own dependencies may cause some duplication of libraries and cost disk space, but this is a cheap price to pay for applications to evolve at their own rates and not be constrained by server dependencies, eg - I need a feature in hibernate 3.3.0, but in prod we only have 3.2.2. No thanks!

Personal opinion is that applications should be as self-contained as possible to simplify deployment and dependency management (leave that to maven). the exception to that is infrastructure explicitly provided by the app server, eg servlets, ejb3 persistence maybe, JDBC drivers definitely. Big fat WAR is Not A Problem, compared to Jarmeggadon...

System Antics

I've had the 3rd edition of this book in my amazon wishlist for ages, now I finally found an online version of the first edition which is short & pithy. read the preface then if you like it check out the rest. fantastic discussion of systems in general (in the sense of systems that humans set up, including but not limited to IT systems, including organisations, governments & companies etc) especially relating to failure modes.

http://www.ece.osu.edu/~fasiha/systemantics/