import java.util.Arrays;
import java.util.List;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.broadcast.Broadcast;
public class Test {
public static void main(String[] args) {
JavaSparkContext sc =
new JavaSparkContext(new SparkConf()
.setMaster("local[*]")
.setAppName("test"));
Broadcast<List<Integer>> broadcast = sc.broadcast(Arrays.asList(1,2,3));
broadcast.destroy(true);
// fails with java.io.IOException: org.apache.spark.SparkException:
// Attempted to use Broadcast(0) after it was destroyed
sc.parallelize(Arrays.asList("task1","task2"),2)
.foreach(x -> System.out.println(broadcast.getValue()));
}
}
这个代码失败了,这是因为我在使用它之前自愿摧毁广播,但事实是,在我的心理模型中,它不应该编译,更不用说运行正常.
的确,Broadcast.destroy(Boolean)被声明为private [spark],所以它不应该从我的代码可见.我会尝试看广播的字节码,但这不是我的专长,所以我更喜欢发表这个问题.另外,对不起,我太懒了,创建一个不依赖于Spark的例子,但至少你得到了这个想法.请注意,我可以使用Spark的各种包私有方法,而不仅仅是广播.
任何想法发生了什么？
解决方法
package yuvie
class X {
private[yuvie] def destory(d: Boolean) = true
}
并在Java中反编译：
[yuvali@localhost yuvie]$javap -p X.class
Compiled from "X.scala"
public class yuvie.X {
public boolean destory(boolean);
public yuvie.X();
}
我们看到Scala中的私有[package]在Java中公开.为什么？这来自于Java私有包不等同于Scala私有包的事实.有一个很好的解释in this post：
The important distinction is that ‘private [mypackage]’ in Scala is
not Java package-private,however much it looks like it. Scala
packages are truly hierarchical,and ‘private [mypackage]’ grants
access to classes and objects up to “mypackage” (including all the
hierarchical packages that may be between). (I don’t have the Scala
spec reference for this and my understating here may be hazy,I’m
using 07001 as a reference.) Java’s packages are not hierarchical,and
package-private grants access only to classes in that package,as well
as subclasses of the original class,something that Scala’s ‘private
[mypackage]’ does not allow.So,‘package [mypackage]’ is both more and less restrictive that Java package-private. For both reasons,JVM package-private can’t be used to implement it,and the only option that allows the uses that Scala exposes in the compiler is ‘public.’
