我有6000个观测值的数据集；以下是一个示例：

job_id      job_title                                           job_sector
30018141    Secondary Teaching Assistant                        Education
30006499    Legal Sales Assistant / Executive                   Sales
28661197    Private Client Practitioner                         Legal
28585608    Senior hydropower mechanical project manager        Engineering
28583146    Warehouse Stock Checker - Temp / Immediate Start    Transport & Logistics
28542478    Security Architect Contract                         IT & Telecoms

目的是根据职位预测每一行的职位.

首先,我对job_title列进行一些预处理：

def preprocess(document):
    lemmatizer = WordNetLemmatizer()
    stemmer_1 = PorterStemmer()
    stemmer_2 = LancasterStemmer()
    stemmer_3 = SnowballStemmer(language='english')

    # Remove all the special characters
    document = re.sub(r'\W',' ',document)

    # remove all single characters
    document = re.sub(r'\b[a-zA-Z]\b',document)

    # Substituting multiple spaces with single space
    document = re.sub(r' +',document,flags=re.I)

    # Converting to lowercase
    document = document.lower()

    # Tokenisation
    document = document.split()

    # Stemming
    document = [stemmer_3.stem(word) for word in document]

    document = ' '.join(document)

    return document

df_first = pd.read_csv('../data.csv',keep_default_na=True)

for index,row in df_first.iterrows():

    df_first.loc[index,'job_title'] = preprocess(row['job_title'])

然后,我对Gensim和Doc2Vec执行以下操作：

X = df_first.loc[:,'job_title'].values
y = df_first.loc[:,'job_sector'].values

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,stratify=y,random_state=0)

tagged_train = TaggedDocument(words=X_train.tolist(),tags=y_train.tolist())
tagged_train = list(tagged_train)

tagged_test = TaggedDocument(words=X_test.tolist(),tags=y_test.tolist())
tagged_test = list(tagged_test)

model = Doc2Vec(vector_size=5,min_count=2,epochs=30)

training_set = [TaggedDocument(sentence,tag) for sentence,tag in zip(X_train.tolist(),y_train.tolist())]

model.build_vocab(training_set)

model.train(training_set,total_examples=model.corpus_count,epochs=model.epochs)   

test_set = [TaggedDocument(sentence,tag in zip(X_test.tolist(),y_test.tolist())]

predictors_train = []
for sentence in X_train.tolist():

    sentence = sentence.split()
    predictor = model.infer_vector(doc_words=sentence,steps=20,alpha=0.01)

    predictors_train.append(predictor.tolist())

predictors_test = []
for sentence in X_test.tolist():

    sentence = sentence.split()
    predictor = model.infer_vector(doc_words=sentence,alpha=0.025)

    predictors_test.append(predictor.tolist())

sv_classifier = SVC(kernel='linear',class_weight='balanced',decision_function_shape='ovr',random_state=0)
sv_classifier.fit(predictors_train,y_train)

score = sv_classifier.score(predictors_test,y_test)
print('accuracy: {}%'.format(round(score*100,1)))

但是,我得到的结果是22％的准确性.

这使我感到非常怀疑,特别是因为通过使用TfidfVectorizer而不是Doc2Vec(两者都具有相同的分类器),我的准确率达到了88％(！).

因此,我想我在应用Gensim的Doc2Vec时一定做错了.

这是什么,我该如何解决？

还是仅仅因为我的数据集相对较小,而诸如词嵌入等更高级的方法却需要更多数据呢？