I am a big fan of Test Driven Development (TDD) and tools like Hudson/Jenkins to automate the process of having a continuous integration build system are key.

On my current project we recently started moving things to Amazon EC2, and rather than put everything on one big server, I thought I’d follow the best practices in cloud computing and make a number of small special purpose servers to take care of the project’s needs.

We’ve had a Jenkins server running for a bit, so rather than reinventing the wheel, I figured I could copy my Jenkins configuration to a new server and get things up and running.

I fired up a new Tomcat server on Amazon Elastic Beanstalk, and loaded up the Jenkins WAR file, which quickly got me to a working Jenkins server. This project is written in PHP, so I had to install PHP after that, which meant logging in to the server and running through the whole PHP and PHPUnit setup.

Once that was done, I scp’d the Jenkins folders from the old server, edited the Tomcat startup files to include the environment variable to point Jenkins to the right place, changed a few permissions, and everything appeared to be working.

I could log in, I fired up the build, and it appeared to be running – very cool.

But that was when the flaw in the design of the PHP unit tests was exposed ….

I was watching the output of the phpunit tests, and noticed two things:

  1. The tests seemed to be taking a really long time
  2. Every test was failing

Watching the console, each time a test would fail, the little “E” would print, then a few seconds would go by and another “E” would appear. Finally after many minutes (because we have a LOT of classes to test) the error output appeared, and looked something like this for EVERY test:

And of course there were 5297 of these … I did some Google searches for the PHP_Invoker_TimeoutException which mostly pointed to issues with upgrade from one version of PHPUnit to another, but the versions on the old server and this one were the same.

So my next step was debugging an individual tests. Running the test from the command line gave me the same error, odd. But then I ran the test using php instead of the phpunit call, and found the problem: I was getting a timeout trying to open a database connection.

The issue as it turns out, is a design flaw in our code that hadn’t showed up before: all the classes invoke a database connection class that sets up the connection to the database as soon as they are loaded.

Since the Elastic Beanstalk server was in a different security group than was allowed to connect to the RDS database, it was unable to connect at all, and PHPUnit would simply timeout before the connection failed (by default phpunit sets 1 second as the acceptable for a test to run in order to catch endless loops).

Now in theory our tests shouldn’t be hitting the database (at least not for these unit tests since we don’t want them updating anything on the backend), so this problem turned out to be very fortuitous. Because the Jenkins server couldn’t reach the database, it exposed a flaw in our unit tests: we weren’t mocking all the things we needed to, so the tests were actually opening connections to the database.

With some refactoring of the test classes to mock the database access layer, the tests all succeeded. Next we’ll need to do the actual DBUnit tests for the database, and Selenium or HTTPUnit tests for all the front-end and AJAX stuff.

I needed to convert an existing Netbeans build to use Maven in order to stabilize the code and support Test Driven Development.

Generally this was a fairly simple process: just move all the source files and resources to the right folders in a new Maven project, and add the dependencies.

Everything was going swimmingly until I ran my first tests and got this really lovely set of errors that didn’t really tell me what was going on:

Now at this point I normally just do a few Google searches and find out how somebody else has solved this problem, but I really wasn’t having any luck at all finding an answer.

I did run across a couple of posts that mentioned seeing this error when using multiple persistence units in the persistence.xml.

Now my project does have a persistence unit for testing outside GlassFish (my chosen EJB container), so my first thought was to move that PU to the “right” place for my tests. So I copied the persistence.xml from src/main/resources/META-INF to the src/test/resources/META-INF and changed them so there was only one PU in each.

Well, that seemed to work for a minute, no more exception on the compile step. But now I had a new problem: I was getting an error that the PU couldn’t be found:

Tests in error:
testinsertTrackingCustomerUnFixed(com.omnistools.service.util.CompanyCustomerTrackingTest): No Persistence provider for EntityManager named ProjectTest-ejbPU
testinsertTrackingWithoutCustomerUnFixed(com.omnistools.service.util.CompanyCustomerTrackingTest): No Persistence provider for EntityManager named ProjectTest-ejbPU
testinsertTrackingCustomerFixed(com.omnistools.service.util.CompanyCustomerTrackingTest): No Persistence provider for EntityManager named ProjectTest-ejbPU
testinsertTrackingWithoutCustomerFixed(com.omnistools.service.util.CompanyCustomerTrackingTest): No Persistence provider for EntityManager named ProjectTest-ejbPU

So doing a bit of digging, it appears that the persistence.xml only gets copied once and only from the src/main/resources/META-INF folder.

I must have tried a thousand different permutations and combinations until I realized that I could have an ugly workaround that is not ideal.

The issue is that if both have the “exclude-unlisted-classes” set to true, the javac compiler tries to recreate the annotation classes during the compiler:compile and of course fails on the second PU because the classes have already been created.

So to get this to work, I set the “exclude-unlisted-classes” to false for one of the PU’s, which gets me the generated meta classes, and the tests fail (because the PU doesn’t include the required entities).

I then flip the flag back to true, and run the build again. Since the classes have already been compiled, the compile:compile doesn’t run, and the tests can succeed.

And then finally, I run across a bug report on the NetBeans site: http://netbeans.org/bugzilla/show_bug.cgi?id=183779. Now this bug talked about setting some compiler flags (in particular -proc:none which tells the compiler not to generate meta classes).

Workaround was proc:none to the javac compiler args as described here: http://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html

Now my Maven build works like a charm. Both persistence units are defined in my persistence.xml, and both have the “exclude-unlisted-classes” set to false, and not only do my tests work, but the deployment is now successful as well.

After getting my new Mac Mini pretty well set up with my development tools, I started playing with Amazon AWS. When trying to connect to one of my instances using their built in tool, Firefox bailed and I was presented with a prompt asking me if I wanted to install Java:

To open “Firefox,” you need a Java SE 6 runtime. Would you like to install one now?

Now this message looked very official, and I’d seen it before when starting up something on a new Mac that required Java, but since I’d already installed the latest JDK I was thoroughly confused. I thought, “OK, maybe you didn’t install Java after all”

So I popped open a terminal and

mini:~ robweaver$ java -version
java version “1.7.0_09″
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode.

OK, so that’s not it – so I changed browsers and tried the browser check at http://javatester.org again, only to get the same results …

Ouch – so I dug around a bit, and couldn’t find anything about how to fix it. And in fact the little bit I did find only confirmed that the browser plugin was active and the right version.

So finally I decided to punt and just let it install – Clicking the button to to let it start ran a package that looked like it was installing something, and when it was done another trip to javatester.org showed that somehow the plugin had been fixed and was now giving me the right version of Java in all my browsers.

Java Version tester

The only thing I can guess is that there was something that had to be set up on first use, but it did give me a start when being asked to install Java SE 6. Fortunately, whatever it did, it didn’t muck with my JVM and I’ll just have to hope that it didn’t break anything elsewhere.

 

I’m doing some work on a project that is using PHP, and have been working on setting up some continuous integration build scripts to make sure that we have a shot at catching errors before they make their way to production.

Recently some unit tests were added for the “forgot password” code, which uses the mcryp

t libraries which are not installed by default on Mac OSX, so I was seeing this error:


So some quick Google searches, and I found a couple of blogs with “how to” install the library (links at the end of this post), and proceeded to get this done.

First step was to download the mcrypt from SourceForge at http://sourceforge.net/project/showfiles.php?group_id=87941

Once I had the file, I opened a command prompt and ran:

Next I ran configure, setting the appropriate flags for my envirionment (note – I didn’t do this for the other configures, probably would have been a good idea):

This sets up the make file, so you can run the next two commands:

Next to make the PHP library, I needed the PHP source files, so I went out and grabbed PHP 5.3.15 (since that’s the version I have when I run php -version) by going to http://us3.php.net/get/php-5.3.15.tar.bz2/from/a/mirror. Note that you can simply change the version number in that URL to get the version you need.

Once I had that I did the following:

This of course gave me an error about autoconf not being installed:

From there I installed autoconf by doing the following:

Then back to the php mcrypt folder and reran the phpize step to and finish building:

Finally an edit to the php.ini file which was simply to add the extension:

And of course a quick restart of Apache and the extension shows up and I can run my unit tests successfully.

Related posts:

http://www.coolestguyplanettech.com/how-to-install-mcrypt-for-php-on-mac-osx-lion-10-7-development-server/

http://michaelgracie.com/2011/07/21/plugging-mcrypt-into-php-on-mac-os-x-lion-10-7/

Enhanced by Zemanta

Just a quick note – I was guided toward doxygen, which is a tool that does source code documentation. It has a DMG installer, that includes an executable to run doxygen with a wizard.

But since my purpose was to run doxygen in an Ant build script, I needed to install the command line version.

I simply looked inside the application, and was able to see that the executable was in /Applications/Doxygen.app/Contents/Resources/

Adding a symbolic link to /usr/local/bin (which is in my PATH) did the trick:

Enhanced by Zemanta

Once upon a time, there was a kingdom of people who were busy managing projects named Pee-Em-Eye Ess-Eff-BeeA Sea. They were busy doing this work, and they found themselves wondering if there was a better way than just telling people what to do, and then scurrying off to the next project.

A small group of people in the kingdom thought that there was a better way to manage the kingdom of Pee-Em-Eye Ess-Eff-BeeA Sea. They believed they could apply some of their project management skills to help all the people in the kingdom to be more successful and happy.

Continue reading

The court room quieted as Paulie C. Governance took the stand. He was sworn in on a stack of PMBOK‘s and the prosecutor was ready to begin.

The following was taken from the transcript and witness accounts:

Prosecutor: Please state your name and occupation for the court

Paulie: Paulie C. Governance, trusted advisor to the board.

Prosecutor: And you are a proposing a change to the way the board operates are you not ?

Paulie: Yes, Policy Governance®.

Prosecutor: Thank you, we have your name, I was asking you about the change you are proposing.

Paulie: Yes, that is Policy Governance® .. with a capital P and a capital G.

Prosecutor: Yes, yes, and a capital C. I was asking what, not who.

Paulie: No capital C, bu there is a registered trademark, like the PMP®

Prosecutor: You don’t capitalize the C?  That IS unusual. Why would Paulie C. Governance have a registered trademark?

Paulie: Well Policy Governance® is the child of John and Mirriam Carver, and they wanted the protection of the registered trademark.

Prosecutor: Your parents gave you a registered trademark ? I think we’re way off topic here. Please tell us: what is the name of the the system you are proposing.

Paulie: Policy Governance®

Prosecutor: So the system has your name ? Are you expecting to run the board yourself.

Paulie: No, the board will use an integrated set of concepts and principles that describes the job of any governing board.

Prosecutor: Now we’re getting somewhere: what is that system called ?

Paulie: Policy Governance@

Prosecutor (clearly frustrated): Now let me get this straight: your name is Paulie C. Governance, right?

Paulie: Yes

Prosecutor: And Mirriam Carver gave birth to Paulie C. Governance ?

Paulie: No, she only added to the theory, it was John Carver’s idea.

Prosecutor (grinning): It usually is the man’s idea. Let’s try something else: you are proposing the board use an integrated set of concepts and principles that describes the job of any governing board.

Paulie: Yes

Prosecutor: and that system is called Paulie C. Governance, but it’s not named after you?

Paulie: Yes

Prosecutor: then what is your name?

Paulie (now frustrated): Paulie C. Governance is my name, and the system is Policy Governance® which is an integrated set of concepts and principles that describes the job of any governing board.

At this point, the prosecutor tried to strangle Paulie, so court had to adjourn until a new prosecutor could be found …

I love Firebug, and I’m getting so I understand jQuery pretty well.

I often drop into the console and type in jQuery commands to figure out how to get things to happen on a page. For example, I was looking at a really long page of search results from Taleo that lists out all of my submissions for jobs at CACI. The problem is, that it shows the fully active submissions with the inactive ones, so it’s not very useful for figuring out what I need to follow up on.

 

So it occurred to me that if I could just write a couple lines of jQuery to look for the items that include “Active” in them, I could reduce this list in a way that would be meaningful for me.

So the first thing I did was go look at the page HTML to figure out whether there was something on the page I could use to separate out each row on the search page.

 

 

 

I fired up Firebug by right clicking on one of the items and choosing “Inspect in Firebug”.

 

 

 

 

 

 

 

 

 

 

This starts up Firebug (if it isn’t already showing) and took me to the part of the HTML that I was looking for:

 

 

 

 

 

 

So now I knew I needed to look for something that had a class of “iconcontentpanel” that contained the text “Active”.  I decided to outline it with a green dashed line, so my jQuery looked like:

So I flipped over to the Console in Firebug and tried running that:

 

 

 

 

 

But that gave me an error since there is no jQuery on the original page. But there’s this nifty little thing in the console that says “jQuerify”, and clicking that injects jQuery into the page:

 

 

 

 

 

 

So now that jQuery is available, running the script again gives me what I was looking for:

 

 

 

 

 

 

 

And of course I can add even more jQuery to strip out unwanted parts of the page, change format, etc, leaving me something that is more  actionable:

Enhanced by Zemanta

I’ve worked hard (as many others have) to eliminate credit cards, stay away from large fee charging banks, and generally protect my money, so when I recently learned that the EDD is paying people with debit cards, I got a bit angry (see: http://www.sfbg.com/2011/09/13/banking-misfortune?page=0,1.)

Debit Card

Now I think the idea of giving people an option other than a check or direct deposit is commendable, and I hope that the EDD at least did this to save some money (although I can’t imagine how this could possibly be more cost effective than an EFT, since somebody has to pay all those bank fees that B of A loves to charge). And maybe I’m just being old fashioned, like a generation ago when the move from cash to paychecks first happened, or again when direct deposit became the norm.

But I don’t think this is really the same, because when a check (or a direct deposit happened), you weren’t an involuntary customer of Bank of America. You were a customer of whatever bank you wanted to be. And if you weren’t a Bank of America customer, they couldn’t count you on their customer roles.

Overnight, B of A has a huge influx of people they can market to. They have an additional revenue stream as well, since these are debit cards, so if you use them at a merchant, that merchant has to pay fees to get their money.

Will the next step be elimination of paychecks ? Maybe the IRS would like that: your money would pass through B of A, they could directly pull taxes from there since it is the Bank of America ….

This just feels plain wrong to me.

I was struck by the power of unintended consequences while watching a show about the science of dyslexia.

What hit me the most was a segment of the show where they were talking to a successful business man who had struggled with dyslexia his entire life and found a place he could succeed in the high school shop. His business was started by buying up the equipment from that same shop as the high school did as most schools have done: cut “non-essential” programs.

What struck me was that because the schools have chosen to cut the broadening curriculum like art, music and shop, there are kids today who will potentially never know success. And even worse, the kids who do well in the remaining classes, will never see that those kids who struggle with the college prep classes have abilities in different areas.

From my high-school experience, I distinctly remember that there were kids who just didn’t do well in the standard classes, but luckily for us we had art, music, home ec and shop. Because of those classes, I got to see some of those kids do amazing things that I simply didn’t have the skill to.

I saw incredible cabinetry built by a kid who literally couldn’t read (today he’d probably have been diagnosed with dyslexia and learned to read).

I recall other kids who built beautiful hot rods from piles of junk, amazing metal contraptions, and people who were able to shine in all sorts of ways that aren’t readily available to kids today.

So as an unintended consequence of “saving money”, we’ve cost our society the chance to expand the social interconnection that happens with recognition.

Those kids may not get the chance to sing in front of the rest of the school and gain the respect that they deserve. That quiet kid in the back of the room might just be perceived as angry and sullen, instead of having the respect as the guy who can do something amazing with a piece of wood.

We weaken our community by not supporting the search for that potential. Those kids frustrated by subjects they have no talent in, who won’t as easily find the joy in something they can do well, and won’t be as likely to go on to be pillars of our community.