我需要计算一行的列之间的相似性,并尝试使用columnsimilarities()方法来获得结果.
public static void main(String[] args) {
SparkConf sparkConf = new SparkConf().setAppName("CollarberativeFilter").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
SparkSession spark = SparkSession.builder().appName("CollarberativeFilter").getOrCreate();
double[][] array = {{5,5},{0,10,0},{5,5}};
LinkedList但是
output in the text file was 0,2,0.9999999999999998
.
接下来,我尝试使用double [] [] array = {{1,3},{2,7}};
那么
output of the text file was 0,1,0.9982743731749959
有人可以解释我的答案格式.我不能得到矩阵的每一列对的分数.如3乘3矩阵我需要3个分数1,2列之间的相似性,3列,3,1列.
任何帮助赞赏.
最佳答案
使用如下定义的Cosine Similarity计算列相似度：
由于你包含了scala标签,我将作弊并重复你在Scala REPL中所做的事情：
scala> import org.apache.spark.mllib.linalg.{Vectors,Vector}
import org.apache.spark.mllib.linalg.{Vectors,Vector}
scala> import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix
scala> val matVec = Vector(Vectors.dense(5,5),Vectors.dense(0,0),Vectors.dense(5,5))
matVec: scala.collection.immutable.Vector[org.apache.spark.mllib.linalg.Vector] = Vector([5.0,0.0,5.0],[0.0,10.0,0.0],[5.0,5.0])
scala> val matRDD = sc.parallelize(matVec)
matRDD: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = ParallelCollectionRDD[44] at parallelize at 此输出表示在(row0,col2)处只有一个非零条目.因此实际(上三角)输出是：
0 0 .9999
0 0 0
0 0 0
这是你所期望的(因为col0和col1之间的点积为零,col1和col2之间的点积为零)
这是一个稀疏列相似性矩阵的示例：
scala> def randVec(len: Int) : org.apache.spark.mllib.linalg.Vector =
| Vectors.dense(Array.fill(len)(Random.nextDouble))
randVec: (len: Int)org.apache.spark.mllib.linalg.Vector
scala> val randRDD = sc.parallelize(Seq.fill(3)(randVec(4))
randRDD: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = ParallelCollectionRDD[123] at parallelize at 404121150599919,0.8206115742925799,0.12848224469499103]
[0.5414651842028494,0.26273347471310016,0.3139446375461201,0.351113866208812]
scala> randRowMat.columnSimilarities.entries.collect.foreach{println}
MatrixEntry(0,0.4630854334046888)
MatrixEntry(0,0.9238294198864545)
MatrixEntry(2,0.33700154742702093)
MatrixEntry(0,0.7402725425024911)
MatrixEntry(1,0.7418690274112878)
MatrixEntry(1,0.8662504236158493)
代表以下矩阵：
0 0.74027 0.92382 0.46308
0 0 0.74186 0.86625
0 0 0 0.33700
0 0 0 0
