apache spark - Scala: Xtream complains object not serializable -


i have following case classes defined , print out clientdata in xml format using xstream.

case class address(addressline1: string,                    addressline2: string,                    city: string,                    provincecode: string,                    country: string,                    addresstypedesc: string) extends serializable{  }  case class clientdata(title: string,                       firstname: string,                       lastname: string,                       addrlist:option[list[address]]) extends serializable{  }   object ex1{     def main(args: array[string]){     ...     ...     ...      // in below, x try[clientdata]     val xstream = new xstream(new domdriver) newclientrecord.foreach(x=> if (x.issuccess) println(xstream.toxml(x.get)))      } } 

and when program execute line print each clientdata in xml format, getting runtime error below. please help.

exception in thread "main" org.apache.spark.sparkexception: task not serializable     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:304)     @ org.apache.spark.util.closurecleaner$.org$apache$spark$util$closurecleaner$$clean(closurecleaner.scala:294)     @ org.apache.spark.util.closurecleaner$.clean(closurecleaner.scala:122)     @ org.apache.spark.sparkcontext.clean(sparkcontext.scala:2055)     @ org.apache.spark.rdd.rdd$$anonfun$foreach$1.apply(rdd.scala:911)     @ org.apache.spark.rdd.rdd$$anonfun$foreach$1.apply(rdd.scala:910)     @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:150)     @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:111)     @ org.apache.spark.rdd.rdd.withscope(rdd.scala:316)     @ org.apache.spark.rdd.rdd.foreach(rdd.scala:910)     @ lab9$.main(lab9.scala:63)     @ lab9.main(lab9.scala)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:498)     @ com.intellij.rt.execution.application.appmain.main(appmain.java:147) caused by: java.io.notserializableexception: com.thoughtworks.xstream.xstream serialization stack:     - object not serializable (class: com.thoughtworks.xstream.xstream, value: com.thoughtworks.xstream.xstream@51e94b7d)     - field (class: lab9$$anonfun$main$1, name: xstream$1, type: class com.thoughtworks.xstream.xstream)     - object (class lab9$$anonfun$main$1, <function1>)     @ org.apache.spark.serializer.serializationdebugger$.improveexception(serializationdebugger.scala:40)     @ org.apache.spark.serializer.javaserializationstream.writeobject(javaserializer.scala:47)     @ org.apache.spark.serializer.javaserializerinstance.serialize(javaserializer.scala:101)     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:301)     ... 16 more 

it isn't xstream complains, it's spark. need define xstream variable inside task:

newclientrecord.foreach { x=>    if (x.issuccess) {     val xstream = new xstream(new domdriver)     println(xstream.toxml(x.get))    } } 

if xstream sufficiently cheap create;

newclientrecord.foreachpartition { xs =>    val xstream = new xstream(new domdriver)   xs.foreach { x =>     if (x.issuccess) {       println(xstream.toxml(x.get))      }   } } 

otherwise.


Comments

Popular posts from this blog

javascript - Thinglink image not visible until browser resize -

firebird - Error "invalid transaction handle (expecting explicit transaction start)" executing script from Delphi -

mongodb - How to keep track of users making Stripe Payments -