Why Unowned Relationships Make More Sense With App Engine Datastore

A month back we started migrating an enterprise time keeping and invoicing system to GAE. Initially, the application was deployed at the enterprise on tomcat. The opportunity was to port the application to GAE and utilize the public cloud space. The choice about GAE was based on the expected ease of migration and then not having to worry about any infrastructure maintenance. As you would have read multiple times across multiple sites, the google datastore is not an RDBMS. The App Engine datastore is a schemaless object datastore, with a query engine and atomic transactions. The people behind GAE have done a good job of abstracting out the details of the underlying implementation by supporting JDO and JPA. However, the truth still remains that not all the functionality of JPA/JDO can be supported because it is not an RDBMS under the hood. Keeping this in mind, we went ahead with the migration.

Let us look at what the partial relationship model looked like before we started migrating. This is a bare minimum stripped down version of the model relevant for this post and easy to understand since it is something that we all can associate with.

Our Agile discussion board snapshot on July 31' 2010

As you would notice there is a 1:n relationship between the Department and the User. Likewise there is a m:n between the User and the UserRole. Again the User can have multiple ProjectAssignment(s) and so on. With JPA/JDO it would be easy to map these relationships and then ensure that the entities are fetched when a query is done.

So far so good with the traditional way of thinking. But, if you apply the same thinking with the datastore world, then expect trouble.

The datastore has the concept of Entity groups. Entity groups are hierarchy of relationships between entities.
To create an entity in a group, you declare that the entity is a child of another entity already in the group. An entity all by itself without a parent is a root entity. A root entity without any children exists in an entity group by itself.

Each entity has a path of parent-child relationships from a root entity to itself (the shortest path being no parent).

So for starters let us consider this, what if you want to get the list of all the active Users in the system with the traditional query like

getEntityManager.createQuery(“SELECT FROM User WHERE active=true”);

With datastore you cannot do this. Why? Because you are trying to refer to more than one Entity group within a single transaction. The beauty of Entity groups is that since datastore has to support a massively scalabale architecture, one Entity group could be present on one node and the other one miles apart on a different node.

There are other issues too with the relationships. For example if the Department has an Id which is Long, can the User also have an Id which is Long?

Yes, you would say instantly is you have just coded an application for JPA/JDO. The answer for the datastore is NO. Why?

The datastore has to store a reference of the parent-child relationship in the child. Hence it needs to have a field which is an Id in the child where it can store the Long value of the parent. So what could this field for the child be? Well, it could either be a String encoded key or the Key of the datastore.

Ok, let us look at more. If you look at the ProjectAssignment entity in the above diagram, then you would see that it has a relationship with both the User and the Project. It has a 1:n relationship with both of them. Sounds fine, right?

Not in the datastore world. In the datastore world, a child can only have one parent. What this translates to is that the ProjectAssignment can either hold the key for the parent User or the parent Project.

So how do we resolve these issues with the relationships?

The simple answer to the hard question is that you dont solve for relationships, you simply remove them. Now, if you look at the whiteboard image again, you would see a lot of places where we have a double line on the lines showing the relationship. These are the relationships that we had to break because of one issue or the other.

And then how do we maintain relationships?

Datastore provides an easy mechanism called unowned relationships where you can maintain relationships manually. You can still manage these relationships using Key values in place of instances (or Collections of instances) of your model objects. The caveat is that the datastore does not guarantee referential integrity with these Key references, but the use of Key makes it very easy to model (and then fetch) any relationship between two objects.

So you might ask what about the relationships that we have not discarded, the ones between Department and User say for example. Well we broke that too today.

The reason is that once you have associated a User to the Department they belong to the same entity group. Once you have a entity group relations you cannot change them. Aargh! So if for hypothetical reasons you are transferred from the Development department to the HR department then we cannot do that with the current mapping structure in datastore. That is the reason that we had to break that relationship too and make it unowned.

The conclusion is that we are quite certain that for a non-trivial application like the one that we are migrating, involving invoices and payments too, unowned relationships is the way to go. Our bet is that with the complex queries that you would be firing and entity groups that you would like to traverse, it would make more sense to plan your model with unowned relationships rather than owned irrespective of whether you want to use JPA or JDO.

Please leave your thoughts on this if you have been able to work with relationships in a non-trivial manner without the need to break them.

About Vikas Hazrati

Vikas is the Founding Partner @ Knoldus which is a group of software industry veterans who have joined hands to add value to the art of software development. Knoldus does niche Reactive and Big Data product development on Scala, Spark and Functional Java. Knoldus has a strong focus on software craftsmanship which ensures high-quality software development. It partners with the best in the industry like Lightbend (Scala Ecosystem), Databricks (Spark Ecosystem), Confluent (Kafka) and Datastax (Cassandra). To know more, send a mail to hello@knoldus.com or visit www.knoldus.com
This entry was posted in Agile, Java and tagged , , , . Bookmark the permalink.

8 Responses to Why Unowned Relationships Make More Sense With App Engine Datastore

  1. Pingback: Improve Performance By Using Keys Only Query on Google App Engine « Inphina Thoughts

  2. How do you break a relationship? 😉

  3. luke says:

    How do you get back collection of object for the unowned set of keys without using the “IN” operator as this produces N number of queries and is maxed out at 30?

  4. Marek says:

    I’m dealing with an entity very similar to the ProjectAssignment you described here. I’ve run into a problem I don’t know how to cope with. In the example the ProjectAssignment entity creates a relation between a user and a project (it’s an intermediate table for a many-2-many relationship between them). Though, in my model I don’t want to make user to be conscious about all his assigments – I don’t want to store the list of keys to assignments in the user entity. I don’t want to maintain the list of assignments in the project either. At the same time I want to be able to query the datastore for all projects assigned to a user. Is it possible with GAE JPA? This can be easily achieved using any relational DB, but have you achieved this with google datastore?

  5. Hiep says:

    create the entity ProjectAssignment which holds {userKey, projectKey}, you donnot need to store the Set in User or Project. Be carefull about transaction, because your ProjectAssignment entity is root in this case (it has no parent).

    how will you deal with the 1MB limit size per entity? if a Departement have to hold a set of 10,000 user? (case of my project)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s