Manish Pandit's Tumblr

Migrating a Perforce repo into git

Thu, 11 Apr 2013 13:02:00 -0400

Here is the git version that I used for this on the Mac running 10.8.3.

$ git --version
git version 1.8.2

It is pretty much a one liner thereafter.

$ git p4 clone //path_to_p4_repo@all /tmp/path_to_git_repo

Note the @all at the end of the P4 repo, this is so that the clone can grab full commit history of the P4 repo.

Once the clone is done, you’ll cd to the git repo and notice the .git folder there. In order to add this to remote, do this:

Step 1. Create a remote repo where you’d be pushing the newly created git repo to. Lets call it some_git_project. I assume you’re in the newly created git repo folder for step 2.

Step 2.

$ git remote add origin http://user@server/path/some_git_project.git

Step 3.

$ git push origin master

You may see this error during step 3 if the code base and/or the revision history is huge.

Delta compression using up to 8 threads.
Compressing objects: 100% (427/427), done.
Writing objects: 100% (2132/2132), 15.92 MiB | 17.65 MiB/s, done.
Total 2132 (delta 1163), reused 2132 (delta 1163)
Unable to rewind rpc post data - try increasing http.postBuffer
error: RPC failed; result=65, HTTP code = 302
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly

This can be fixed by running the below, thanks the post here, and retrying Step 3.

git config http.postBuffer 209715200

Enjoy!

Cassandra : Demystifying Column Families

Mon, 18 Mar 2013 14:50:00 -0400

Last 2 posts were fairly trivial, but now comes the fun part. I am seeing a lot of UTF8Type and validation_class type of syntax, which tells me (at least initially) that there is a lot of flexibility when it comes to creating structures and types. I will find out what these things are - but without getting into a lot of details, from whatever documentation I could google, a column family is what defines the structure of the document. In other words, its a family of columns, just like a relational table is. For Mongo folks, this would be hard to grasp as there is no concept of a column or a group of columns there. All Mongo has is a collection to group a bunch of documents, which in turn, have columns (or keys). So, a family has a name, and you define the columns that form that family. This to me felt super odd till I found out that I can also have dynamic families, where I do not have to specify the columns ahead of time. We’ll get there, and we’ll also try to add/delete columns from a family that we declare.

Initial instinct would be to just declare a column family with no columns.

[default@myspace] create column family SomeFamily;
9b08e2e8-f0d8-3788-a23c-0c4a720b766c

Apparently it worked. Now, this just means that we’ve run a SQL similar to create table SomeFamily. Whats different is that the SQL wont work as we did not specify any columns.

To insert data into a column family, its similar to inserting data in an map. Lets analyze the below (it will not actually work, we’ll find out soon).

[default@myspace] set SomeFamily['lobster1234']['name']='Manish Pandit';
org.apache.cassandra.db.marshal.MarshalException: cannot parse 'name' as hex bytes

Looking at the set statement, we’re using a key (which is my Yahoo! ID, lobster1234) and setting name to the value of my full name. This looks familiar, doesn’t it? Like a map of maps or a multimap. The key is Yahoo! ID, and the value of this key is another map, with key as name and value has the name of the user. I can also add another key-value to the valuemap, this time adding a gender. Now if I were to extend it, I can keep adding more folks with their Yahoo! IDs as keys to the multimap, with the value of this key being another map, whose keys are name and gender. A point to be noted here is that this key (lobster1234) is referred to as a row key, as it identifies the key-value pairs that are mapped to it. Like a key in a multimap.

In Scala, this is what SomeFamily boils down to:

scala> var someFamily = Map[Any,Map[Any,Any]]()
someFamily: scala.collection.immutable.Map[Any,Map[Any,Any]] = Map()

scala> someFamily = someFamily+("lobster1234"->Map("name"->"Manish Pandit"))
someFamily: scala.collection.immutable.Map[Any,Map[Any,Any]] = Map(lobster1234 -> Map(name -> Manish Pandit))

Hope this clarifies or at least gives you an image of what a column family is like. The closest counterpart to this for MongoDB would be a collection. A collection contains a bunch of documents, with every document having an _id, and key-value pairs, where keys are Strings and values can be of the types supported by BSON spec (including another key-value pair).

In Mongo terms, the above would look like:

use  myspace
db.someFamily.save({_id:"lobster1234","name":"Manish Pandit"})

Adding another value to this document will be something like

 db.someFamily.update({_id:"lobster1234"},{$set:{"gender":"Male"}})

Going back to the error we saw when trying to insert a record in our column family (SomeFamily) - it appears that Cassandra is assuming that we are providing hex values instead of UTF8. So there has to be some way to tell Cassandra that the column family is expected to hold UTF8 data. Again, Google and Cassandra documentation say that we need to provide additional parameters while creating the column family. Lets try this one more time. We can delete SomeFamily column family so we can use the same name.

[default@myspace] drop column family SomeFamily;
36cac489-882a-345f-94f4-10dfb61d9395

Note that column family names are case sensitive - I learnt it by messing with the case of the column family name. SomeFamily and somefamily would be two unique column family names in a given keyspace.

So, lets change our create column family statement to account for UTF8 data as the key as well as value(s), and try to insert a record.

[default@myspace] create column family SomeFamily with comparator=UTF8Type and key_validation_class = UTF8Type and default_validation_class=UTF8Type;
27b66cfa-76ce-340c-bd31-4079873368db
[default@myspace] set SomeFamily['lobster1234']['name']='Manish Pandit';
Value inserted.
Elapsed time: 4.1 msec(s).

We added comparator and validation classes to the definition. These are often referred to as comparators and validators. The data type of the column name is called a comparator. For comparison, the comparator in a relational database is always String, because all the column names in a table have to be a String. A validator defines the data type of the column values. We used key_validation_class so that the key ‘lobster1234’ can be validated against the specified type, and the default_validation_class so that the value ‘Manish Pandit’ can be validated against the specified type.

The default value for comparators and validators is hex byte[], hence the error that we saw earlier.

Cassandra does support type-casting values, so if we did not provide the default_validation_class during the creation of the column family, the following would work:

[default@myspace] set SomeFamily['mpandit.meetup']['name']=UTF8('MPandit Meetup');

Note the UTF8(..) conversion.

MongoDB does not need any of this, as it supports the BSON types for the values, and figures the type out at runtime. As long as the provided value is one of the BSON types, MongoDB will not complain. There is no type tied to a value. In other words, a key can have different type of value in 2 documents within the same collection, like this (note the type of the key someType is an int in one document, and a String in another).

> db.test.insert({"someType":2})
> db.test.insert({"someType":"Manish"})
> db.test.find()
{ "_id" : ObjectId("5147d3f995500e98b8f7a6cd"), "someType" : 2 }
{ "_id" : ObjectId("5147d3fd95500e98b8f7a6ce"), "someType" : "Manish" }

Also, the keys (or the names of the keys) in a MongoDB document can only be Strings.

To keep the discussion lively and post length in check, I’ll discuss static and dynamic column families, CRUD operations in the next post.

Cassandra-CLI : Do not forget your semicolons!

Mon, 18 Mar 2013 03:55:00 -0400

I knew it was too good to be true. After being spoilt by Scala, and MongoDB client - I need to sober up and start using semicolons. I really hope there is some kind of switch I can use here but this gets annoying.

mpandit$ ./cassandra-cli
Column Family assumptions read from /Users/mpandit/.cassandra-cli/assumptions.json
Connected to: "Test Cluster" on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.2.2

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] help
...    
...	;
Getting around:
?                       Display this help.
help;                   Display this help.
quit;                   Exit this utility.

[default@unknown] quit
...    
...	
...	
...	;

Now lets build the databases and stick some data in. A db in Mongo == a keyspace in Cassandra. So, a Mongo show dbs is same as show keyspaces; (do not forget the semicolon!) in Cassandra. However, this produces huge output and unlike MongoDB it isn’t JSON, so you cannot wrap this into a javascript and filter it out. But, more data means more information - reading through the output, I am starting to see the various attributes that can be tuned and configured.

Creating a DB (keyspace) is explicit, unlike Mongo where its implicit. In Mongo you can say use somedb and the moment you save something in that context, like db.somecollection.save({“a”:”b”}), that db will be created for you. In Cassandra, you create a keyspace explicitly, like so:

[default@unknown] create keyspace myspace;
e5bb6d2e-36ea-365b-85cc-065d9cf15941

This returns a hash, which I am not sure what to do with at this moment. I think its the UUID for every entity that gets created within Cassandra. I could be wrong, we’ll find out. Maybe its something like MongoDB’s getLastErrorObj which is returned after every operation.

Now that we’ve created a keyspace, we use it. Similar to MongoDB’s use somedatabase, except that the keyspace has to exist before using it.

[default@unknown] use myspace;
Authenticated to keyspace: myspace
[default@myspace] use myotherspace;
Keyspace 'myotherspace' not found.

Notice the promot has changed from default@unknown to default@myspace.

You can then use show schema; to introspect the keyspace. This is very much like show create table of MySQL and I cannot imagine a MongoDB counterpart of this.

[default@myspace] show schema;
create keyspace myspace
  with placement_strategy = 'NetworkTopologyStrategy'
  and strategy_options = {datacenter1 : 1}
  and durable_writes = true;

use myspace;

Also you will notice that the create statement has a list of default settings that will show up. Since we used none, this gives an idea of what all options are available.

In the next post we will create Cassandra counterparts for Collections and Documents.

Tiptoeing on Cassandra after running a MongoDB Marathon

Mon, 18 Mar 2013 02:43:00 -0400

As mentioned in my previous post, I am weeks away from starting at Netflix Engineering. One of the things that I absolutely have never touched is Apache Cassandra. While I do not think of myself as a NoSQL n00b, Cassandra is something I really never got around to.

With this change, I decided to pay around with it, and I’ll use this blog as a series to post my thoughts with the context of MongoDB inertia.

Getting Started:

As with anything else, the best way to learn is to *not* RTFM. Lets see how far I can get with this.

I downloaded 1.2.2 and extracted it on my Mac. When running

bin/cassandra -f

I did get a boatload of errors.

xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms2048M -Xmx2048M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError 

log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/cassandra/system.log (No such file or directory)
........
........
INFO 23:54:25,132 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 23:54:25,133 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 23:54:25,141 Scheduling row cache save to each 0 seconds (going to save all keys).
ERROR 23:54:25,167 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 23:54:25,172 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 23:54:25,172 Failed to create /var/lib/cassandra/data/system/batchlog directory
ERROR 23:54:25,173 Failed to create /var/lib/cassandra/data/system/peer_events directory
ERROR 23:54:25,173 Failed to create /var/lib/cassandra/data/system/peer_events directory
ERROR 23:54:25,174 Failed to create /var/lib/cassandra/data/system/hints directory
ERROR 23:54:25,175 Failed to create /var/lib/cassandra/data/system/Schema directory
ERROR 23:54:25,176 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 23:54:25,177 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
ERROR 23:54:25,177 Failed to create /var/lib/cassandra/data/system/schema_keyspaces directory
.....

Clearly I need to fix the folder locations. Usually all servers have a conf or config or settings folder. So does Cassandra. I found the conf folder. Looking at the logs, it read the config from a yaml file. Fortunately there is only one yaml file there. I guessed the log location is in the log4j properties file. Next up, to change locations where the process has permissions to write.

README.txt    		cassandra-rackdc.properties	cassandra.yaml
cqlshrc.sample			log4j-tools.properties
cassandra-env.sh		cassandra-topology.properties
commitlog_archiving.properties	log4j-server.properties

I vi’d the cassandra.yaml file and started changing the locations. Same with log4j*.properties. Lets try using ~/cassandra/? instead of /var/? and see where we go with it.

mpandit$ ./cassandra -f
xss =  -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms2048M -Xmx2048M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError
 INFO 00:28:36,783 Logging initialized
 INFO 00:28:36,799 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_43
 INFO 00:28:36,799 Heap size: 2105540608/2105540608
 INFO 00:28:36,814 Loading settings from file:/Users/mpandit/software/apache-cassandra-1.2.2/conf/cassandra.yaml
 .....
 ......
 INFO 00:28:39,610 Binding thrift service to localhost/127.0.0.1:9160
 INFO 00:28:39,641 Using TFramedTransport with a max frame size of 15728640 bytes.
 INFO 00:28:39,652 Using synchronous/threadpool thrift server on localhost : 9160
 INFO 00:28:39,653 Listening for thrift clients...
 INFO 00:28:49,628 Created default superuser 'cassandra'

Great! looks like we got our Cassandra server working. Next post is where I’d google around to figure out Cassandra CLI and create the MongoDB equivalent of dbs, collections, and documents. To be honest I’ve no idea if these constructs are even valid, or make sense in Cassandra world - we will find out.

So long, IGN - Hello Netflix!

Fri, 15 Mar 2013 21:00:00 -0400

After a long hiatus from the blogosphere, I am back. A lot has changed. IGN got acquired by Ziff Davis, and I, along with a lot of awesome engineers and friends, were let go as a part of the restructuring during last week of February. Life happens.

After a few weeks of rigorously interviewing at several companies in the SF Bay Area, I accepted the offer from Netflix. Not only do I love the product (Netflix subscriber since 2000), but the technology brand that Netflix carries is like none other. Their culture and talent density intrigues me, and I cannot wait to start at a company with such well defined, practical values, and a track record of following them religiously.

Technology wise, Netflix is the poster child for AWS deployments, and distributed, highly available architectures. They have also started the Netflix OSS initiative. The initiative involves open sourcing parts of Netflix’s tech stack. The OSS roadmap is very exciting, and so are the projects that have already been open sourced. The number of forks and stars tell the same story. Too bad I was 80 miles north (in SF) when the OSS meetups happened in Los Gatos, but now I’ve all the reasons and more to be a part of them.

One of the first things I would like to do is hiring - if you want to be a part of Netflix Engineering Team, and are as intrigued about the company and the culture as I am - please let me know. Take a look at the Netflix Technology Blog, Netflix OSS repos - and drop me a line!

On Thursday, 10/04/2012 : Evolving IGN’s API on Scala

Wed, 26 Sep 2012 13:37:37 -0400

On Thursday, 10/04/2012 : Evolving IGN’s API on Scala

Happy Birthday to my daughter, Tanvi!

Mon, 02 Apr 2012 05:05:00 -0400

Happy Birthday to my daughter, Tanvi!

Lone Hydrant at Yosemite View Lodge in Yosemite, CA

Sat, 17 Mar 2012 04:05:18 -0400

Lone Hydrant at Yosemite View Lodge in Yosemite, CA

Also: BD is the new SEO

Fri, 16 Mar 2012 17:20:21 -0400

Also: BD is the new SEO:

roybahat:

Or, the return of the human.

There was (once upon a time, in the early 2000’s) a generation of Internet products built on the premise that if you mastered Google’s search rankings, you could grow traffic. Services like IMDb, About.com, Wikipedia, etc. provided value to their users, but were…

Anorm rules!

Fri, 16 Mar 2012 02:48:00 -0400

Tried a few lines of code in Anorm within a PlayFramework 2.0 App. Very impressed.

      
DB.withConnection { implicit conn =>
  SQL("select * from person_accounts limit 0,20")
  .apply().foreach(x => println(x[String]("account_type")))	
}

How cool is that?

Keep your QA close...and your users closer

Fri, 24 Feb 2012 18:15:00 -0500

Building great products is every engineer’s passion, more so when the product is going to get millions of visits every day. At IGN, we build such products and work hard to make sure that we have quality, scale, predictability, and stability with our systems.

One of such systems is our Social API, which gets about 20K requests a minute. Recently we started noticing issues with this API farm going down for no obvious reason. Newrelic showed a massive spike getting data from memcached, but thats about all we had. Memcached hit ratio kept dropping whenever this happened - all the way to zero. This started happening at a much greater frequency while we started debugging critical pieces of code that could potentially fail to scale. We started looking at traffic data to see if there is anything unusual. Nothing stood out of the ordinary.

It was not until one fine morning when I received this from one of our top users, AngryMrBungle (read from bottom -> top)

This DM woke me up at 7AM, and I thought he could be talking about his browser crashing and not MyIGN. Later did I realize that he never mentioned a browser. Time to get more info.

That cleared up the entire situation. On my way to work I was able to link this problem to the exact classes and method that needed refactoring. We had a fix in place after a bit of whiteboarding.

Testing was again a challenge, as I did not want to bother AngryMrBungle, but at the same time we could not figure out to test the site in the right circumstances, fast. Thats one of the challenges with testing “scalability” with the right mix of requests. I asked him if he could help us test this and he gladly agreed. We had the team with eyes pinned on NewRelic. Not a blip on NewRelic. In a few minutes I got a DM from him that he did not face any problems like he did earlier.

This was good news for the team, and the refactored architecture will get rolled out to other API layers as well.

In summary, it is very critical for consumer facing sites to be in constant touch with the users. The users are way more passionate than we as engineers think, and very helpful in charting the feature set they want. We use uservoice and it does not surprise me to see users write detailed feedback and requests, and telling us how we can make the U|X better. We are refactoring quite a bit of technology platforms at IGN and moving to a more API-based, Open Architecture on the back-end, and Widget-based on the front-end. The idea is for us to build a platform to scale for user-base and content growth, and the only way to do this better is to engage with our audience. We take pride in being on top of this, and I thought this incident was something worth mentioning in a blog. Between engaging with our users on boards, MyIGN and twitter - UserVoice is something that helps us a lot with getting the pulse of the users, and pretty much have our product featureset “direction” on an auto-pilot.

I’ll write a separate blog explaining the technical side of this “scalability bug” that AngryMrBungle helped uncover.

IGN Engineering launches code.ign.com

Fri, 24 Feb 2012 16:42:00 -0500

IGN Engineering launches code.ign.com

FML : Java how much I love thee..

Wed, 22 Feb 2012 01:20:00 -0500

private List<String> longList2StringList(List<Long> list){
    List<String> toReturn = new ArrayList<String>(list.size());
    for(Long l : list) toReturn.add(l.toString());
    return toReturn;
}

Vs.

implicit def longList2StringList(x:List[Long]) 
       = x.map(_.toString)

Re-architecting how the activity streams work in IGN’s social platform. Should be interesting...

Sat, 18 Feb 2012 04:14:14 -0500

Re-architecting how the activity streams work in IGN’s social platform. Should be interesting next few days. Challenges around push vs. pull, aggregation and sharding top the list for now.

February bloom in the backyard!

Mon, 13 Feb 2012 04:24:43 -0500

February bloom in the backyard!

Dandelion growing near my mailbox.

Mon, 06 Feb 2012 02:12:03 -0500

Dandelion growing near my mailbox.

Tip-toeing node.js

Wed, 01 Feb 2012 04:06:00 -0500

While writing a pubsubhubbub implementation at IGN (which we’re going to open source!), I decided to write the publisher and the subscribers (callbacks) in node.js. One big reason was that this seemed to be an area where node would really shine, and I was sick of writing Scala for days on end (the “hub” is in Scala) and wanted a break.

What better way to mix it up with a little node.js. I find Javascript pretty cute (having dealt with it as JScript in the 90s and giving up!) and intuitive. I do not (nor will I) have the chops to dish out a killer front end in JS, but server side JS is a match made in heaven. I was able to roll out a simpleton producer and the callback handlers in a few hours and could not have been happier. Keep an eye on IGN’s github repo, as thats where the hub will be published.

Named Orinoco, it is destined to solve quite a few problems that a lot of systems face, mostly around polling and batch jobs.

Scala, Maven and Tests..

Sun, 22 Jan 2012 03:18:36 -0500

I moved my sbt project to maven recently and discovered that none of my tests were getting detected when I ran mvn test.

After looking around a bit, it was a missing annotation on the test class which fixed the problem. I am sure you can google around and its right there, but thought of making a note here as well.

package com.ign.orinoco.test
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FlatSpec
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner

@RunWith(classOf[JUnitRunner])
class OrinocoTest extends FlatSpec with ShouldMatchers{

  it should  "make sure that 2 ones make a 2" in {
    1+1 should equal (2)
  }
 }

And this gets detected, and run just fine!

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.ign.orinoco.test.OrinocoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Play!Framework 2.0 Beta : Not happy

Thu, 12 Jan 2012 19:47:00 -0500

Being a huge fan of PlayFramework, I decided to try the 2.0 beta today. However, the good ol’ SBT came to haunt me again. I feel really bad for having to say that this is not something I expected from Play!. They should have stayed with the Python tools they had for build management rather than picking such a platform and version dependent tooling system. Or better, they should have bundled a version of SBT that actually works with the distribution. My first few minutes with Play! 2.0 Beta were nowhere close to the first few with Play! Java a few years ago. I was expecting to download, eclipsify and run a sample app in under 5 minutes, just like with Play! 1.2.3. What a disappointment.

mpandit-mbp:tmp mpandit$ ~/software/play/play new myapp
       _            _ 
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/ 
             
play! 2.0-beta, http://www.playframework.org
The new application will be created in /private/tmp/myapp
What is the application name? 
> myapp
Which template do you want to use for this new application? 
  1 - Create a simple Scala application
  2 - Create a simple Java application
  3 - Create an empty project
> 1
OK, application myapp is created.
Have fun!
mpandit-mbp:tmp mpandit$ cd myapp/
mpandit-mbp:myapp mpandit$ ~/software/play/play eclipse
[info] Loading global plugins from /Users/mpandit/.sbt/plugins
[info] Updating {file:/Users/mpandit/.sbt/plugins/}default-31fbe2...
[warn] 	module not found: com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3
[warn] ==== typesafe-ivy-releases: tried
[warn]   http://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbteclipse/sbteclipse-plugin/scala_2.9.1/sbt_0.11.0/2.0.0-M3/ivys/ivy.xml
[warn]   -- artifact com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3!sbteclipse-plugin.jar:
[warn]   http://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbteclipse/sbteclipse-plugin/scala_2.9.1/sbt_0.11.0/2.0.0-M3/jars/sbteclipse-plugin.jar
[warn] ==== local: tried
[warn]   /Users/mpandit/software/play/framework/../repository/local/com.typesafe.sbteclipse/sbteclipse-plugin/scala_2.9.1/sbt_0.11.0/2.0.0-M3/ivys/ivy.xml
[warn]   -- artifact com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3!sbteclipse-plugin.jar:
[warn]   /Users/mpandit/software/play/framework/../repository/local/com.typesafe.sbteclipse/sbteclipse-plugin/scala_2.9.1/sbt_0.11.0/2.0.0-M3/jars/sbteclipse-plugin.jar
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/typesafe/sbteclipse/sbteclipse-plugin_2.9.1_0.11.0/2.0.0-M3/sbteclipse-plugin-2.0.0-M3.pom
[warn]   -- artifact com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3!sbteclipse-plugin.jar:
[warn]   http://repo1.maven.org/maven2/com/typesafe/sbteclipse/sbteclipse-plugin_2.9.1_0.11.0/2.0.0-M3/sbteclipse-plugin-2.0.0-M3.jar
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn]   http://scala-tools.org/repo-releases/com/typesafe/sbteclipse/sbteclipse-plugin_2.9.1_0.11.0/2.0.0-M3/sbteclipse-plugin-2.0.0-M3.pom
[warn]   -- artifact com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3!sbteclipse-plugin.jar:
[warn]   http://scala-tools.org/repo-releases/com/typesafe/sbteclipse/sbteclipse-plugin_2.9.1_0.11.0/2.0.0-M3/sbteclipse-plugin-2.0.0-M3.jar
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Some unresolved dependencies have extra attributes.  Check that these dependencies exist with the requested attributes.
[warn] 		com.typesafe.sbteclipse:sbteclipse-plugin:2.0.0-M3 (sbtVersion=0.11.0, scalaVersion=2.9.1)
[warn] 
[error] {file:/Users/mpandit/.sbt/plugins/}default-31fbe2/*:update: sbt.ResolveException: unresolved dependency: com.typesafe.sbteclipse#sbteclipse-plugin;2.0.0-M3: not found
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? q

Compare this mess to Play! 1.2.3. As easy as it can get, which is why I am a huge fan of Play!

mpandit-mbp:tmp mpandit$ ~/software/play-1.2.3/play new myapp
~        _            _ 
~  _ __ | | __ _ _  _| |
~ | '_ \| |/ _' | || |_|
~ |  __/|_|\____|\__ (_)
~ |_|            |__/   
~
~ play! 1.2.3, http://www.playframework.org
~
~ The new application will be created in /private/tmp/myapp
~ What is the application name? [myapp] 
~
~ OK, the application is created.
~ Start it with : play run myapp
~ Have fun!
~
mpandit-mbp:tmp mpandit$ cd myapp
mpandit-mbp:myapp mpandit$ ~/software/play-1.2.3/play eclipsify
~        _            _ 
~  _ __ | | __ _ _  _| |
~ | '_ \| |/ _' | || |_|
~ |  __/|_|\____|\__ (_)
~ |_|            |__/   
~
~ play! 1.2.3, http://www.playframework.org
~
~ OK, the application is ready for eclipse
~ Use File/Import/General/Existing project to import /private/tmp/myapp into eclipse
~
~ Use eclipsify again when you want to update eclipse configuration files.
~ However, it's often better to delete and re-import the project into your workspace since eclipse keeps dirty caches...
~
mpandit-mbp:myapp mpandit$

Edit: After following the instructions here, I was able to generate an Eclipse project. I will keep blogging about my experiences with Play! 2.0 beta. For now this sure has not been a breeze and playful as Play! < 2.0 was.

mpandit-mbp:myapp mpandit$ rm -rf ~/.sbt
mpandit-mbp:myapp mpandit$ rm -rf ~/.sbt.cache.lock 
mpandit-mbp:myapp mpandit$ pwd
/tmp/myapp
mpandit-mbp:myapp mpandit$ ~/software/play/play 
[info] Loading project definition from /private/tmp/myapp/project
[info] Updating {file:/private/tmp/myapp/project/}default-d21f0b...
[info] Done updating.
[info] Compiling 1 Scala source to /private/tmp/myapp/project/target/scala-2.9.1/sbt-0.11.0/classes...
[info] Set current project to myapp (in build file:/private/tmp/myapp/)
       _            _ 
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/ 
             
play! 2.0-beta, http://www.playframework.org

> Type "help" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.
[myapp] $ eclipse
[info] About to create an Eclipse project for you.
[info] Please hang on, because it might be necessary to perform one or more updates and this might take some time ...
[info] Updating {file:/private/tmp/myapp/}myapp...
[info] Done updating.
[info] Successfully created Eclipse project files.

Evolving Scala Echosystem

Thu, 12 Jan 2012 05:08:00 -0500

After spending a few months with Scala, I am at a point where I’ve found quite a bit of tidbits on digging myself in a hole, and managing to get out of it. So far I’ve found that sbt is a pain in the ass, maven is not as bad as its set out to be, salat is a decent object (well case class) to DBObject mapper, and lift is not that scary to tippie-toe into.

IGN’s Video API was the 1st service to roll onto the new stack. We call it v3 (apply for a job at IGN Engineering and I’d be more than happy to go over v1 and v2, which is a very interesting story I promise).

The new stack has:

Scalatra for the services endpoints
MongoDB for persistance
Elasticsearch for..well, search
Salat for Scala <-> DBObject transformation
sbt as the build system

We use Newrelic to monitor the performance, and use memcached as our caching layer. The numbers were *very* impressive compared to where we ported the Video services from, and it was a good proof as well as a vote of confidence. Soon enough, we started to roll out this new stack to the new services, as well as porting some old ones to it.

Along the way, we found that dealing with update-merges and juggling around DBObjects was turning out to be bug-prone and labor intensive. The solution? lift-mongodb-record. No more case classes, and easy transformation to/from JSON to top it. The next set of services are using this when it comes to talking to MongoDB.

One pain in the ass was sbt, which, when we tried to upgrade, we ran into issues with Idea plugin, and the jetty-web plugin incompatibility.

mpandit-mbp:video-api mpandit$ sbt gen-idea
[info] Set current project to default (in build file:/Users/mpandit/.sbt/plugins/)
[info] Set current project to default (in build file:/Users/mpandit/work/git/video-api/project/plugins/)
[info] Set current project to default (in build file:/Users/mpandit/work/git/video-api/)
[info] Trying to create an Idea module default
[error] java.lang.NoSuchMethodError: sbt.Attributed.get(Lsbt/AttributeKey;)Lscala/Option;
[error] Use 'last' for the full log.mpandit-mbp:video-api mpandit$

The hacks to fix this (per google search) were too time consuming after many failed attempts and frankly we had to move on - we like to keep things simple, and since we were not writing custom sbt plugins, we’d rather stick to what works. At this point we decided to brush the dust off maven, and surprise surprise - it worked like a charm. We plan to stick with Maven for now, as we do not touch configuration that often anyway. Also, the ScalaIDE 2.0 from Typesafe made Scala development in Eclipse fairly easy (compared to the super buggy earlier releases) and I was able to get back to my favorite IDE for my newfound favorite programming language.

Its going great so far, and we’re very happy with the progress along the new stack. Needless to say there is a lot more we can improve on, but thats all the part of learning! I am sure the stack will change going forward as we learn more with pair programming and tech-talks. We are not wrangling implicits and other advanced (academic? cryptic?) scala capabilities yet, but given the progress the team has made so far, its not too far on the horizon.