apache spark - Scala: Xtream complains object not serializable -
i have following case classes defined , print out clientdata
in xml format using xstream.
case class address(addressline1: string, addressline2: string, city: string, provincecode: string, country: string, addresstypedesc: string) extends serializable{ } case class clientdata(title: string, firstname: string, lastname: string, addrlist:option[list[address]]) extends serializable{ } object ex1{ def main(args: array[string]){ ... ... ... // in below, x try[clientdata] val xstream = new xstream(new domdriver) newclientrecord.foreach(x=> if (x.issuccess) println(xstream.toxml(x.get))) } }
and when program execute line print each clientdata in xml format, getting runtime error below. please help.
exception in thread "main" org.apache.spark.sparkexception: task not serializable @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:304) @ org.apache.spark.util.closurecleaner$.org$apache$spark$util$closurecleaner$$clean(closurecleaner.scala:294) @ org.apache.spark.util.closurecleaner$.clean(closurecleaner.scala:122) @ org.apache.spark.sparkcontext.clean(sparkcontext.scala:2055) @ org.apache.spark.rdd.rdd$$anonfun$foreach$1.apply(rdd.scala:911) @ org.apache.spark.rdd.rdd$$anonfun$foreach$1.apply(rdd.scala:910) @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:150) @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:111) @ org.apache.spark.rdd.rdd.withscope(rdd.scala:316) @ org.apache.spark.rdd.rdd.foreach(rdd.scala:910) @ lab9$.main(lab9.scala:63) @ lab9.main(lab9.scala) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:498) @ com.intellij.rt.execution.application.appmain.main(appmain.java:147) caused by: java.io.notserializableexception: com.thoughtworks.xstream.xstream serialization stack: - object not serializable (class: com.thoughtworks.xstream.xstream, value: com.thoughtworks.xstream.xstream@51e94b7d) - field (class: lab9$$anonfun$main$1, name: xstream$1, type: class com.thoughtworks.xstream.xstream) - object (class lab9$$anonfun$main$1, <function1>) @ org.apache.spark.serializer.serializationdebugger$.improveexception(serializationdebugger.scala:40) @ org.apache.spark.serializer.javaserializationstream.writeobject(javaserializer.scala:47) @ org.apache.spark.serializer.javaserializerinstance.serialize(javaserializer.scala:101) @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:301) ... 16 more
it isn't xstream complains, it's spark. need define xstream
variable inside task:
newclientrecord.foreach { x=> if (x.issuccess) { val xstream = new xstream(new domdriver) println(xstream.toxml(x.get)) } }
if xstream
sufficiently cheap create;
newclientrecord.foreachpartition { xs => val xstream = new xstream(new domdriver) xs.foreach { x => if (x.issuccess) { println(xstream.toxml(x.get)) } } }
otherwise.
Comments
Post a Comment