Monday, April 15, 2013

Cloning of entity subgraphs

First of all, I have to introduce myself. I work as a programmer, analyst and a SW architect since 2003. Past experiences with JPA and Hibernate motivated me to write this short article about considerable benefits of entity cloning. I also introduce an opensource project JpaCloner. You can find it on the github link https://github.com/nociar/jpa-cloner (released under the Apache License, Version 2.0).

Deep entity clone

The cloning mechanism itself is nothing new or special. Every semi-skilled programmer knows it (copy constructor in C++ or clone() method in Java). The cloning of a simple object is common and definitely not interesting for most of us. Until... we have to deal with concepts like fetching of entity subgraphs, defensive copying or the entity serialization. It is advisable in many situations to use, pass or return a deep copy of an entity. But the question is: how to define the deep copy? It depends on many factors but mostly on the actual context and requirements. So to be more precise, we often need to clone an entity subgraph. Deep copies of entities have many benefits, some examples:

  • A deep copy can be considered as a fetched entity subgraph.
  • A copy is always detached from a persistence manager. Any change of a copy does not affect the original.
  • A deep copy can be used out of a transaction scope and prevents the throwing of the LazyInitializationException.
  • A copy allows to get rid of JPA proxy classes. A proxy free copy can be serialized by many frameworks (like GWT RPC).

I found couple of projects which deal with similar concepts like Apache Dozer, Gilead (hibernate4gwt) or JpaEntityManager#copy() in EclipseLink. But non of them fulfills my needs, so I decided to do a small project myself. Following sections describe usage of the project in more details.

Entity subgraph

JPA/ORM mapping can be considered as a graph of POJOs. Entities are nodes and relations are edges (singular, collection or map). A deep clone of an entity should also make deep clones of neighboring entities and their neighboring entities, and so on... Such strategy of transitive cloning may easily ends with the whole DB cloned and OutOfMemoryError thrown. We have to clone only a part of the object graph - subgraph. The desired subgraph can be specified by a set of paths from a root entity. After some time I found that the common dot notation (e.g. "company.department.boss.address") is not flexible enough to cover many additional needs like recursive relations and path splitting and joining. So I introduced the GraphExplorer class which generates the paths upon a string pattern. String literals inside the pattern are treated as properties. Operators resemble the syntax of regular expressions:
  • Dot "." separates paths.
  • Plus "+" generates at least one preceding path.
  • Split "|" divides the path into two ways.
  • Terminator "$" ends the preceding path.
  • Parentheses "(", ")" groups the paths.
  • String literals support wildcards: star "*" and question mark "?".
Some examples follow:
  • project.devices.interfaces
  • project.devices.inter??ces
  • project.devices.*
  • school.teachers.lessonToPupils.(key.class|value.parents)
  • company.departments+.(boss|employees).address.(country|city|street)

JPA Cloner

The JpaCloner class allows to clone entity subgraphs specified by the string patterns. After the algorithm is finished, each cloned entity contains basic properties (columns) and cloned relations. Non-cloned relations are left as null. The cloned entities are always created as raw classes (annotated by the @Entity or @Embeddable). So a cloned object will never be a hibernate proxy. The JpaCloner supports relations defined as List, Set, SortedSet, Map and SortedMap. The class contains static methods to shield the programmer from the implementation details. The usage looks like:

MyEntity e = em.find(MyEntity.class, 1234);
MyEntity c = JpaCloner.clone(e, "x.y.z", "aa.(bb.cc)*.dd");

If you need to clone a list or a set of entities, you can use overloaded clone methods:

TypedQuery<MyEntity> query = entityManager.createNamedQuery(...);
List<MyEntity> list = query.getResultList();
list = JpaCloner.clone(list, "x.y.z", "aa.(bb.cc)*.dd");


Conslusion

Working with JPA and ORM may easily become painful (and in some cases a nightmare). This applies especially when it comes to a persisting and merging of complex structures with higher-level semantics. I found that using clones makes my work with ORM again entertaining. Clones can be returned by any transactional method with fetched subgraphs (I use it in my GWT applications). I hope that the project will be useful not only for me :-)

20 comments:

  1. It's fast and easy to use.
    Thanks for this great project!

    ReplyDelete
  2. it helped very much!!
    This is exactly what i wanted!!

    Thanks Man!!

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. hi Miroslav,

    I want to know if it is possibile to say to jpaCloner to NOT copy some fields?
    We use openJpa enhancer and the enhancer adds all its logic during compile.. it adds even other fileds like:

    protected transient StateManager pcStateManager;
    private transient Object pcDetachedState;

    We do not want to copy this fields bc when we try to persist the entity this exception is throwed "Caused by: org.apache.openjpa.persistence.EntityExistsException: Attempt to persist detached object ... If this is a new instance, make sure any version and/or auto-generated primary key fields are null/default when persisting."
    We use PropertyFilter filter = PropertyFilters.getAnnotationFilter(Id.class, Transient.class);
    so the id is null but the error continue to persist since the enhancer controls the pcDetachedState... We cannot set to null this fields in our code bc it is added during compile time...

    Using
    BeanUtils.copyProperties(copy, orig);
    pcStateManager and pcDetachedState are set to null so it persist well on a single element but not on a graph...

    So can we say to jpaCloner using patterns or somethine else to not copy this fields when it clones the objects??

    Thank you
    Arbri

    ReplyDelete
    Replies
    1. Hello, you can also provide a custom PropertyFilter implementation for clone methods. For example:

      PropertyFilter f1 = PropertyFilters.getAnnotationFilter(Id.class, Transient.class);

      PropertyFilter f2 = new PropertyFilter() {
      public boolean test(Object entity, String property) {
      return !"pcStateManager".equals(property)
      || !"pcDetachedState".equals(property);
      }
      }

      You can also mix the filters like:

      PropertyFilter mixed = PropertyFilters.getComposedFilter(f1, f2);

      I hope that this answers your question...

      BR/
      Miro

      Delete
  5. Hi Miroslav,

    ti works :-)
    thank you very much, this utility of yours is a very good one. Compliments ;-)
    This spare us a lot of time.
    You need just to add some more documentation/examples

    ps: the test condition must be
    return !"pcStateManager".equals(property) && !"pcDetachedState".equals(property);


    Thank you again
    Arbri

    ReplyDelete
  6. I am trying to do exactly what is described here and I am not able to filter the id field of of sub-entity. Apparently, the filter that I am using is working on the root level entity and not it's children.
    Can someone guide here and tell me what I am doing wrong? Or is it how this cloner works.

    ReplyDelete
  7. Hello, could you please provide more details regarding the issue? Any entity field should be able to filter out during the cloning process. Do you use a custom implementation of the PropertyFilter interface?

    ReplyDelete
  8. Here is my code:

    PropertyFilter customFilter = new PropertyFilter() {
    public boolean test(Object entity, String property) {
    return !"id".equals(property);
    }
    };
    PropertyFilter annotationFilter = PropertyFilters.getAnnotationFilter(Id.class, Transient.class);

    PropertyFilter composedFilter = PropertyFilters.getComposedFilter(customFilter, annotationFilter);
    McqQuestion clonedQuestion = JpaCloner.clone(question, composedFilter, "*");
    System.out.println(clonedQuestion);

    Question is an entity object here and it has a OneToMany relationship with another entity(Choice Entity). I want to clone this question and I don't want the choices' id and other transient fields to be cloned.

    But the observed behavior is that the id and transient fields of cloned question are behaving correctly(not cloning) but the id and transient fields of Choice entity are still getting cloned.

    I want the Choices' id and transient fields to be left null.

    ReplyDelete
    Replies
    1. Hello, could you please copy a part of the Choice entity? Especially the class declaration and the primary key field. Does the class use the field access or the property access?

      Thank you

      Delete
  9. Hi,
    Here is the entities' declaration:

    @Getter
    @Setter
    @Document(collection = "QuestionCollection")
    @Entity
    @Table(name = "Question_New")
    @Inheritance(strategy = InheritanceType.JOINED)
    public class Question {

    public Question() {
    }

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "id")
    protected BigInteger id;

    @Column(name = "field1")
    protected int field1;

    @Column(name = "field2")
    protected long field2;
    }


    @Getter
    @Setter
    @Entity
    @Table(name = "Mcq_Question")
    @PrimaryKeyJoinColumn(name = "base_question_id", columnDefinition = "bigint NOT NULL")
    public class McqQuestion extends Question {

    @Column(name="base_question_id")
    private BigInteger id;

    @LazyCollection(LazyCollectionOption.FALSE)
    @OneToMany(cascade = CascadeType.ALL)
    @JoinColumn(name = "fk_question_id", columnDefinition = "bigint",nullable=false)
    private Set choices;
    }







    @Getter
    @Setter
    @Entity
    @Table(name = "Choice")
    public class Choice {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "id")
    private int id;

    @Lob
    @Column(name = "field1")
    private String field1;

    @Column(name = "field2")
    private double field2;
    }


    To answer your question about field or property access, since I am using Lombok's annotations, it should be property access.

    Reitertating the problem, the question and it's subclass McqQuestion are getting cloned properly(filters are working), but Choice is not.

    ReplyDelete
    Replies
    1. Hello thank you for the reply. I believe that the problem is caused by using the same filed name "id" in Question and McqQuestion. The "id" field in the McqQuestion class hides the "id" field in the Question class. Please try to change:

      @Column(name="base_question_id")
      private BigInteger id;

      to

      @Column(name="base_question_id")
      private BigInteger baseQuestionId;


      BTW: you mapping uses field access, as the fields are JPA-annotated - lombok adds only getters/setters ;-)

      Delete
    2. Hi Miroslav,
      I am not facing issues in question vs McqQuestion. The problem is with Choice entity. I don't want it's Id field to be cloned. How to prevent that?

      Here is my test case:

      McqQuestion question = new McqQuestion();
      question.setId(new BigInteger("1"));
      question.setAnswerDescription("ans descr");
      question.setQuestionType(QuestionType.MCQ);
      question.setClientId(3490);

      Set answerChoices = new HashSet<>();
      AnswerChoiceMCQ e = new AnswerChoiceMCQ();
      answerChoices.add(e);
      e.setChoiceId(12);

      question.setChoices(answerChoices);
      PropertyFilter customFilter = new PropertyFilter() {
      public boolean test(Object entity, String property) {
      return !"id".equals(property);
      }
      };
      PropertyFilter annotationFilter = PropertyFilters.getAnnotationFilter(Id.class, Transient.class);

      PropertyFilter composedFilter = PropertyFilters.getComposedFilter(customFilter, annotationFilter);
      McqQuestion clonedQuestion = JpaCloner.clone(question, composedFilter, "*");
      System.out.println(clonedQuestion);

      Delete
    3. Hello, could you please also paste the code for AnswerChoiceMCQ? I see there is a setter "setChoiceId()". May be this is the root cause...

      Delete
    4. It's the same Choice entity that I had posted earlier. Was trying changing the name of id field here.

      Delete
    5. Hello in that case I suppose that you have to find the problem by debugging. You can inspect the cloning of objects in the method JpaCloner#copyBasicProperties - very simple method which copies basic properties (not relations) from original to cloned objects.

      Delete
    6. I did that originally. I thought that the list of choices won't be considered as the basic property as it is a oneToMany relation. But apparently this is not happening.
      I would like to discuss more on this with you. Would it be possible for you to discuss this on some chat messenger? Skype or something?

      Delete
  10. Well this is an open source project and I have extremely limited time for support. So please try to debug your posted example - create a conditional breakpoint and try to find out WHY the @Id field of the AnswerChoiceMCQ entities get cloned:

    https://github.com/nociar/jpa-cloner/blob/master/src/main/java/sk/nociar/jpacloner/JpaCloner.java#L244

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete