我正在计算矩阵的spearman相关性.我发现使用scipy.stats.spearmanr时矩阵输入和双数组输入给出了不同的结果.结果也与pandas.Data.Frame.corr不同.
from scipy.stats import spearmanr # scipy 1.0.1
import pandas as pd # 0.22.0
import numpy as np
#Data
X = pd.DataFrame({"A":[-0.4,1,12,78,84,26,0],"B":[-0.4,3.3,54,87,25,np.nan,1.2],"C":[np.nan,56,143,11,np.nan],"D":[0,-9.3,23,72,-2,-0.3,-0.4],"E":[78,-1,-11,323]})
matrix_rho_scipy = spearmanr(X,nan_policy='omit',axis=0)[0]
matrix_rho_pandas = X.corr('spearman')
print(matrix_rho_scipy == matrix_rho_pandas.values) # All False except diagonal
print(spearmanr(X['A'],X['B'],axis=0)[0]) # 0.8839285714285714 from scipy 1.0.1
print(spearmanr(X['A'],axis=0)[0]) # 0.8829187134416477 from scipy 1.1.0
print(matrix_rho_scipy[0,1]) # 0.8263621207201486
print(matrix_rho_pandas.values[0,1]) # 0.8829187134416477
后来我发现熊猫的rho和R的rho一样.
X = data.frame(A=c(-0.4,0),B=c(-0.4,NaN,1.2),C=c(NaN,NaN),D=c(0,-0.4),E=c(78,323))
cor.test(X$A,X$B,method='spearman',exact = FALSE,na.action="na.omit") # 0.8829187
但是,Pandas的corr不能用于大表(例如,here和我的情况是16,000).
感谢Warren Weckesser的测试,我发现Scipy 1.1.0的两个数组结果(但不是1.0.1)与Pandas和R的结果相同.
如果您有任何建议或意见,请与我们联系.谢谢.
我使用Python：3.6.2(Anaconda); Mac OS：10.10.5.
最佳答案
当输入是一个数组并给出一个轴时,scipy.stats.spearmanr似乎没有按预期处理nan值.这是一个脚本,它比较了几种计算成对Spearman排序相关性的方法：
原文链接：https://www.f2er.com/python/438897.htmlimport numpy as np
import pandas as pd
from scipy.stats import spearmanr
x = np.array([[np.nan,3.0,4.0,5.0,5.1,6.0,9.2],[5.0,4.1,4.8,4.9,4.1],[0.5,7.1,3.8,8.0,7.6]])
r = spearmanr(x,axis=1)[0]
print("spearmanr,array: %11.7f %11.7f %11.7f" % (r[0,1],r[0,2],r[1,2]))
r01 = spearmanr(x[0],x[1],nan_policy='omit')[0]
r02 = spearmanr(x[0],x[2],nan_policy='omit')[0]
r12 = spearmanr(x[1],nan_policy='omit')[0]
print("spearmanr,individual: %11.7f %11.7f %11.7f" % (r01,r02,r12))
df = pd.DataFrame(x.T)
c = df.corr('spearman')
print("Pandas df.corr('spearman'): %11.7f %11.7f %11.7f" % (c[0][1],c[0][2],c[1][2]))
print("R cor.test: 0.2051957 0.4857143 -0.4707919")
print(' (method="spearman",continuity=FALSE)')
"""
# R code:
> x0 = c(NA,3,4,5,9.2)
> x1 = c(5.0,NA,4.1)
> x2 = c(0.5,7.6)
> cor.test(x0,x1,method="spearman",continuity=FALSE)
> cor.test(x0,x2,continuity=FALSE)
> cor.test(x1,continuity=FALSE)
"""
输出：
spearmanr,array: -0.0727393 -0.0714286 -0.4728054
spearmanr,individual: 0.2051957 0.4857143 -0.4707919
Pandas df.corr('spearman'): 0.2051957 0.4857143 -0.4707919
R cor.test: 0.2051957 0.4857143 -0.4707919
(method="spearman",continuity=FALSE)
我的建议是不要以spearmanr形式使用scipy.stats.spearmanr(x,nan_policy =’omit’,axis =< whatever>).使用Pandas DataFrame的corr()方法,或使用循环使用spearmanr(x0,nan_policy =’omit’)成对计算值.