Jay Paulynice

My thoughts about life, technology, cars, music, business and more.

Author: Jay

Building A Raspberry Pi 3 Cluster

I will write a much longer post about this at some point, but these are some pictures I took while building my raspberry pi 3 cluster.

I ordered a raspberry pi 3 when it came out on February 29, but it took 2 weeks to arrive and in total coincidence arrived home on pi day, 3/14.

I went to the Micro center here in Cambridge to find some more hardware to hook up my pi3, but to my surprise, I found they had several pi3’s at hand. I grabbed a few to build a cluster and this is the end result. I haven’t decided what project I will do with them, but this is just for fun for now.

Up and running:

4-node cluster:

The raspbery pi 3 unboxing:

The supplies:

The original PI:

Fixing Apache Mahout’s Connection Pooling Datasource Warning

I’ve meant to write about this some time ago, but forgot. While using Mahout to build a recommendation engine with a datasource like MySQL or PostgreSQL, one of the warnings one usually encounters is this:

“You are not using ConnectionPoolDataSource. Make sure your DataSource pools connections to the database itself, or database performance will be severely reduced.”

This is a weird warning because even if you’re using a connection pool, the warning will not go away.

Last week, I got very annoyed and decided to fix it. I dug through the source code on github and found this code in AbstractJDBCInMemoryItemSimilarity.java:

In order to create an item similarity based recommender with data stored in a MySQL database, we would do something like this:

MySQLJDBCInMemoryItemSimilarity extends SQL92JDBCInMemoryItemSimilarity which in turn extends the abstract class AbstractJDBCInMemoryItemSimilarity where the data source type is checked to make sure it is a ConnectionPoolDataSource type as above.

The fix is rather simple, but takes some time to dig through the code and figure it out. Here I’m using Spring to autowire my dataSource object, creating the tables and populating them with some initial data SpringDataConfig.java:

This requires commons-dbcp dependency like this:

The difference is instead of returning a dataSource that is an instance of DriverManagerDataSource, we return a new ConnectionPoolDataSource that takes in our normal data source. Mahout then takes care of the connection pooling as follow:

That’s it.

The Terrible Technical Interview or (why technical interviews don’t work)

Having been on both sides of the technical interview, I have to say that technical interviews are just terrible. Here is a typical scenario: a candidate finds this really awesome looking job posting online and decides it sounds really interesting! The candidate submits a resume, and somehow the resume ends up in a recruiter’s hand and long behold someone on the team is scheduled to do a phone screen with the candidate.

Here is what I’ve noticed: While everyone says an interview is a 2-way street, the technical interview almost always ends up with the company only asking the candidate technical questions.

Sometimes, I wonder how so many smart savvy developers allow this to be the case.

Here is what is so interesting about a typical 1-hour technical interview: 55 minutes is spent on asking a developer to do xyz in their favorite programming language then the developer at the end has 5 minutes to ask questions of the interviewer. Notice sometimes no introduction, etc…

Here is another interesting thing: ever asked the interviewer a really meaningful question? Like so what would I be working on at xyz company? The typical answer? “Well I don’t know, my boss wanted me to do this interview…we have blah blah blah…If you have more questions I can get back to you…”

Wait what? Noticed the double standard? What if a candidate said: “Well this is a hard question…never thought about it…I’ll get back to you in a couple days when I find out?” Or God forbid the candidate asks: “So what’s the company’s revenue? How many customers do you have?”

I’m sure what the response would be.

Unfortunately or fortunately, a few years ago, I found myself in this exact position. I had no clue what the candidate would be working on, but someone asked me if I could interview a few people for his team. Here is the sad part of that story: we found a really good developer who was just the right guy for the team, but the company all a sudden froze hiring!

Jacoco coverage for multi-project Gradle setup

This post is about setting up a Gradle multi-project build with Jacoco to get an aggregate test code coverage report. This is something I struggled with at first, but with some free time in my hands yesterday, I set out to make it work finally.

The example is part of a Seed Project I have been working on for REST API using Jersey2/Spring.

See build.gradle for full example. The end results:

Coveralls integration:

Simple Html reports:

Building A Simple Movie Recommendation Engine

The goal of this blog post is to build a simple movie recommendation engine using Apache Mahout.

The code is on my github here: Movie Recommendation Engine.

I first came across Apache Mahout a couple years ago while researching machine learning libraries for a music application I was working on. Mahout’s goal, according to the official Apache web page, is “to build an environment for quickly creating scalable performant machine learning applications.” The latest version 0.10 seems to offer just that and more combined with Hadoop and several new features.

But today, I just want to look at a simple use case of building a movie recommendation API. The Use case:

Given:

  • A list of users
  • A list of movies
  • Similarities between movies…movie 1 and movie 2, etc.
  • Users preferences for each movie

For a given user recommend a number of movies.

This is similar to the feature we see all the time on Amazon after buying a book: “You might also be interested in that book”. This falls under the collaborative-filtering algorithms families whereby we have a database of user preferences for certain items like movies, books, etc and similarity between the items. Using that data, we can predict what a user would prefer.

Data Model:

The data model is somewhat simple:

  • users table stores user info
  • items table stores movie details
  • taste_preferences stores user preference for a movie
  • taste_item_similarity table stores similarity between movies

To start, I created an interface for the recommendation engine. While the data can be stored in a MySQL database, it can also live in a CSV file. Also recommendations can be user based or item based. With user based recommendations, the idea is to look for similar users and what items they like. Item based recommendation on the other hand mean given several user preferences for items, find similar items. Therefore, we can have multiple implementations.

Interface:

MySQL Item Based Recommendation Implementation:

Initializing the recommender:

Recommending movies:

Getting movie details:

Example Response:

Android- Change A Shape’s Color Dynamically.

This post is part of a small android jigsaw puzzle game that I have been working on for some time. I originally created the project 2 years ago when I worked on a similar python jigsaw puzzle game.

The project is on my github here: Android-Jigsaw-Puzzle

The idea is very simple: a user draws a picture then creates a jigsaw puzzle with the drawing. The application slices the image into n x m pieces which are then randomly rearranged. The user tries to reconstruct the image by dragging the slices into the correct positions.

I thought this would be a fun idea because after drawing the picture, the user would need a lot of attention to detail to recreate it from the randomly generated and rearranged pieces.

These are some example drawings:

A boat with green color selected:

Boat with orange color selected:

Looking at the lower rows, we see 6 colors and 4 brush sizes. When a user selects a color, all 4 brushes change to show the selected color.

With Android, a developer can create an image by specifying an xml resource. For example this is the small brush which resides under the Android-Jigsaw-Puzzle/app/src/main/res/drawables folder.

small.xml file:

This image can then be referenced as the source code of an image button using the @drawable/id where ‘id’ is the shape created above…@drawable/small in our case.

One problem I came across was how to dynamically change the color of the brushes when a user selects a color. I googled for a few hours and none of the solutions seemed to work for me…for example: stack overflow. Finally, I thought hard about it and came up with this solution:

It could be that the android API has changed, but this is quite simple. However, it took me a long time to get it. So I wrote this quick post in the hope that someone looking to change a shape’s color dynamically in android can find it.

Configuring Backbone+Marionette Application Layout With Database persistence using MySQL+Hibernate

This post is part of a larger code repository search application that I have been working on as a side project. The complete code is on my github here. In order to keep the post short and sweet, it assumes some familiarity with a lot of technologies from the front-end: backbone+marionette+requirejs to the middle and service tiers: java+spring+cxf+elasticsearch+hibernate+mysql.

The focus of this post is primarily this use case:

  • as a user, I want to be able to configure my application layout so that I can have my search results on the left side and my document preview on the right side or vice versa.

Examples:

1. Results on the left and preview on the right

2. Preview on the left and results on the right

My complete project setup on github:

  • owlpad-webapp: web application
  • owlpad-service: public api
  • owlpad-domain: domain objects shared by the api, service and the client
  • owlpad-service-impl: service implementation

To start, I created 4 classes under the domain project:

Configuration.java

Layout.java

LayoutType.java

Region.java

Under the owlpad-service project, I created an interface:

ConfigurationService.java

Under the owlpad-service-impl, I then created my dao/dao implementations as well as service implementations:
ConfigurationDao.java

ConfigurationDaoImpl.java

ConfigurationServiceImpl.java

applicationContext.xml for owlpad-service-impl:

There is rather a bit more involved with setting up the REST service, MySQL, and Hibernate, but this is left to the user to experiment with and figure out since the complete code is here

When the service starts, Hibernate creates the tables:

then import some default configuration data located under:
owlpad-service-impl/src/main/java/resources/META-INF/db/initialize.sql

Now that we’ve setup the REST Service, we go into our webapp and setup the spring controller and repository:

Under the owlpad-webapp project, we create:
ConfigController.java

ConfigRepository.java

servlet-context for webapp:

servlet-context.xml

Now that all the backend and middle tiers have been setup. We can look at the fun javascript, backbone, marionette parts. Again, this project assumes that the user is familiar with requirejs, backbone and marionette and has a good understanding already.

In order to make a call to get the application configuration, we create an ajax entity:
configEntities.js

then a configMediator:

configMediator.js

Then our marionette application:
app.js

The important part here is the call to load the application configuration:
configMediator.loadConfig() which returns the json:

Then using underscore’s built in functions _.object and _.map, we create a nicer object for the regions:

Configuring the application is a matter of changing which selector goes with what region in the database. In this case, we have the preview region on the left (r1) and results on the right (r2).

I will add a part 2 later to explain in more details…but for now, the github repo is a good source…

© 2017 Jay Paulynice