Office of Secret Intelligence

I've converted this blog to static HTML pages for now. I wanted to get off of digital ocean and rewrite my blog app and run it on heroku. I'll be sure to update things when they're ready.

I've finally put together a little code to generate an RSS feed.  Feedly will pick it up automatically, otherwise go to http://stonecolddev.in/feed.rss and add it.

Hi all,  back again with the second part of the play scala+slick+postgres adventure.

In part 1, I touched on some of the hurdles I needed to overcome in order to get things up and running (UPDATE:.  the docs have been updated, and Mirco has done a really good job of making this more clear: https://www.playframework.com/documentation/2.4.0/PlaySlick).  In this article, I'd like to show you how to set up a quick and snappy testing environment.

 

For starters, we will want a separate test database for postgres.  For local tests, just manually create the database and user, and grant that user privileges to create tables, etc. on that database: 

Yes, I know, you don't need 3 psql calls to do this but it works and I'm extremely lazy.

Now, let's start looking at the bigger picture.  What if we wanted to set up continuous integration so we could have our tests run every time we git push'd?  That sounds good, so let's do that with Travis CI.  I'll leave it up to you to get your repository and account set up.  Once that's done, check out the next part.

Travis CI makes setting up CI testing with all sorts of things really easy.  Basically, you just create a file named ".travis.yml" with your configuration options, add it to your repo (I've only done this with github, I don't know how other git hosts work with Travis) and it takes care of the rest after you commit it and push.  Here's the one we're going to be working off of:

Very simply, we tell Travis we're using scala, java 8, specifying a shell script which will kick off our tests (source shown below), and that we want to use postgres and have it run the commands in before_script prior to running our tests.  Here's the script that runs our tests:

Briefly, this checks for an environment variable called STEEL_TEST_LOCAL being set to 1, which, if present, tells us we should nuke the current test tables so we can start clean for this upcoming test run.  Set this on your local environment so you have fresh and clean test tables prior to each test run.  Travis doesn't care about this since it tears everything down automatically at the end of the test.

The next line uses a tool called Flyway to migrate our database changes.  I actually really like Flyway, thusfar at least, because it was extremely easy to set up and start using.  It completely blows Play's evolutions out of the water.  Let's set that up really quick.

Add the Flyway dependencies and configuration to your build.sbt, like so:

The one thing here that needs explaining is the flywayLocations key.  To specify a path on the filesystem (local to where your tests are being run out of, so most likely your app root), you pass something like Seq('filesystem:path_to_sql_files').

So, we have flyway set up for migrations, the next step is to create some migrations and have them applied.  Flyway actually has excellent and extremely simple documentation on how to do this, and since we're using sbt to manage our tests, here's the introduction to do so: http://flywaydb.org/getstarted/firststeps/sbt.html.

Backing up slightly, I want to explain one more line in the build.sbt file:

javaOptions in Test ++= Seq("-Dconfig.file=conf/test.conf")

This is all you need to specify a different configuration file for your test environment.  This is really useful because you can just copy conf/application.conf, and change the database credentials to match your test environment, and it will automatically be picked up when your tests run (example here: https://github.com/dhoss/steel/blob/207ce910e668457c3870446322d98e94f5271f88/conf/test.conf#L40)

So, back to setting up our CI environment.  In our run_tests.sh script, we:

1. check if we are running locally, and if so, nuke the test database
2. otherwise, we run our migrations with flyway + sbt, and because flyway lets us do so, we override the default database configuration parameters and set them to our test's parameters.
3. lastly, we finally call sbt to run ours tests with sbt +test

That's pretty much it.  I found it very simple to get a small, no-nonsense, test environment set up that transfers over quite nicely to a relatively robust CI solution. 

Next time, I'm planning on discussing some more application specific things dealing with getting your queries right with slick under this set up.

UPDATE: This isn't a full tutorial. I plan on doing this in pieces. The code shown here is one example of one piece of the app. I'll delve more into the code later, but I thought that the issues I encountered while getting up and going were more important to write about initially. I hope that's not too confusing.

---

 

 

It's been a while, but I finally have something interesting to write about.

 

I've been dabbling with the Play framework again after some time.  Work uses Java for a whole lot of stuff so I figured it'd behoove me to ease myself into the JVM again.  I don't know Scala very well, but after a foray into Go and C#, looking at it now makes a lot more sense than when I first looked at it.

I decided that I wanted to write a workout progress tracker.  I'm getting married in August and I made it a goal a while back to put on some muscle weight, and put on some weight on the bar.  I'd been struggling and struggling to get more weight up on the bar, and just couldn't be consistent.  Long story short, it turns out a lot of it was my diet, probably almost all of it (but that's another story).  My lifts started improving, I gained about 10-12 pounds, and I decided I wanted to see how far I've come since I started officially lifting again a few years ago.

 

A few things have changed since I last looked at Play.  For one, it uses the activator system to scaffold apps, run tests, install scala, etc.  It's a little confusing, but really it's not so bad.  I also had intended on using squeryl for my database object stuff, but it looks like it hasn't been updated in a while and I remembered the agonizing pain I went through just to get it to work at all the last time I did this. So, I went with slick, play's endorsed ORM FRM.  Specifically, I went with play-slick, which integrates slick directly into play itself.  

This was a little bit of a nightmare at first, because the tutorial for play initially sets you up with play 2.3.8 by default, and I wanted to be using the latest play-slick, as those docs seemed the most robust and up to date, which requires: 

1. Play version 2.4.x
2. Slick version 3.0.x
3. Scala version 2.10.x/2.11.x

The issue I ran into, that frustrated me to no end, was the fact that the tutorials don't specify which specific versions of each you need.  I just looked through my commit history to try to find specifics, and I started convulsing, so long story short: if you're getting a lot of errors trying to find dependencies, make sure your build.sbt has at least a couple resolvers for the typesafe repository, and google the latest version for your dependency and try using that.  Here are my plugin.sbt and build.sbt for reference (I went ahead and made a gist so that they remain tied to this post's version, here's the repo in case you are from the future and want to see something that may be more up to date: https://github.com/dhoss/steel): https://gist.github.com/dhoss/1a824ee5397f22eec0ac.

ANOTHER NOTE BEFORE I GO FURTHER: I felt like a complete idiot for this, but adding a resolver in plugins.sbt doesn't necessarily mean it's going to be available in build.sbt.  I'm sure this is incredibly naive and dumb, but I spent a lot of time yelling at my screen trying to figure out why nothing was being pulled down from the repo I had just added.  I ended up just adding the resolvers to both build.sbt and plugins.sbt.  Like they say, "shoot 'em all and let God sort 'em out."

 

The last thing I really needed to get sorted was this nagging issue where my app couldn't connect to postgres.  Postgres was running, I could connect to it using the same username and password, but the app couldn't do it. Again, this was something minor that turned out to be in front of my face the whole time, but because I wasn't able to find a lot of documentation on slick driver connection configuration.  This one line was all that was standing between me and glory:

 This needs to go before a line that looks like this:

 

This really confused me for a while, so I can help keep others from this same frustration.

Moving on to some code.

Make sure you read through this and get everything set up first: https://www.playframework.com/documentation/2.4.x/Home 

I decided to go with a DAO set up for my database access, because the initial code I was following  is set up as such and it seemed reasonably simple and flexible.  You do have to write your CRUD methods, but I found it nice to be able to control exactly what you want those doing.

For starters, let's define our model.  In this case, we basically define our table structure in relatively broad strokes.

Simple enough.  Remember, you still need to write the actual SQL to create these tables with play evolutions.

Next, let's take a look at one of our DAOs.

We define our column methods, and a * projection that acts how would expect it would with SQL.  Next, we create a class that sets up the dbConfig, our query object (exerciseTypes), a method to map ids to names for option dropdowns, and our insert and list methods.  Fairly simple.

Per TDD, here is a small test that makes sure we can insert and retrieve things:

Nothing out of the ordinary here either.

So, this concludes things for now.  I've got a ways to go on my scala skills, and this app certainly has a bit to go as well.  I'm hoping to document the app's progress and post any "gotchas" I run into from here on out as well.

Next time, I'll try to demonstrate how to add a datatable for quick CRUD and sorting (Part 2: http://stonecolddev.in/posts/playing-with-scala-building-a-small-web-app-with-play-2-4-play-slick-and-postgres-part-2-testing)

lvm pvscan
                    lvm vgscan
                    lvm lvchange -ay /dev/VolGroup00/LogVol_home
                       ^ volume group     ^ logical volume name
                         ex: kodiak-dev     ex: home, var, usr
                    lvm lvscan

                    fsck -yfv /dev/kodiak-dev/home

Rinse and repeat for the rest of the logical volumes you wish to fsck.

I've been working on a gem called fluorescent for a little while, and I just recently cleaned it up and released it.  This CMS is using it, so it seems to be working.

 

Basically, I wanted to be able to highlight search terms in search results and only show them in context for things like article bodies, which could potentially be several pages long.  Truncating them to a certain length is certainly beneficial.

 

Here's a brief rundown:

 

Add fluorescent to your Gemfile:

gem 'fluorescent'

Next, you need to pass your search results from ActiveRecord to fluorescent.

I do it like so:

Then, in my model:

Some very simple controller code:

 

How you display the results in the view is up to you.  I simply make sure the query parameter exists, then make sure there are actual results, and then iterate over the array of hashes:

 

I realize there are several points here that need to be brought up to the Rails 4 code standard, but I think these snippets get the point across.  I think it would be wonderful just to be able to call something like @results = fluorescent(@resultset) and have it just do the right thing, but that's down the road a bit in terms of development and complexity.

You'll also note that I keep track of the raw ActiveRecord resultset.  This isn't really used right now, but originally it was in case you needed to get to the non-hash serialized ActiveRecord row objects.

 

All thoughts and suggestions, feedback, and critique are welcome.  Please enjoy!