java - NotSerializableException for Jackson ObjectNode in Spark closure issue -

say have following java object maps jackson full data binding:

public class student implements serializable{     private objectnode name; // 2 keys: "first_name", "last_name"      // getter , setter ... } 

and have following spark code attempts serialize closure variable student of type student due different scopes.

class a(student : student) extends serializable {     def process(input: dataframe): unit = {         val test = input.map { =>             print(student)         }     } }  

which gives following error: caused by: java.io.notserializableexception: com.fasterxml.jackson.databind.node.objectnode

i understand why getting such error. basically, spark attempt serialize out-of-scope variables a.k.a. closures , pass each executors. since objectnode not serializable, executor cannot student instances.

my question is, ways can solve this?

i have tried using map<string, string> instead of objectnode, since objectnode's put , set can have "primitives" , jsonnode value, causes error when try :

objectnode meta_info = jsonnodefactory.instance.objectnode(); meta_info.set("field name", student.getname()); 

there several options.

if need object node json serialization purpose can rewrite student class , remove objectnode. in example can subsitute object firstname , lastname fields

class name implements serializable {     string firstname;     string lastname; } 

however if not possible can custom serialization this

public class student implements serializable {     private transient objectnode name;      private void writeobject(objectoutputstream out) throws ioexception {         objectmapper mapper = new objectmapper();         out.writeutf(mapper.writevalueasstring(name));         // other fields here     }      private void readobject(objectinputstream in) throws ioexception,             classnotfoundexception {         objectmapper mapper = new objectmapper();          jsonnode node = mapper.readtree(in.readutf());         if (!node.isobject()) {             throw new ioexception("malformed name field detected");         }          name = (objectnode) node;          // read other fields     } } 

in example serialized object node json string, of course can iterate on object node fields store each field separately.

you can read more custom serialization in objectoutputstream javadoc.

also can experiment different data serializers kryo.