2_evaluation.ipynb 3.05 KB
Newer Older
Philipp Koncar's avatar
Philipp Koncar committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from gensim.models import Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "*models_filename:* The complete path to the pickeld word2vec models.\n",
    "\n",
    "*word_pairs_filename:* The complete path to the list of word pairs used for evaluation. This needs to be a **.csv** file.\n",
    "\n",
    "*selected_model_filename*: The filename for the best performing model which will be used for the subsequent classification. You may use the **.p** extension indicating a pickled file, but you are free to use whatever you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models_filename = \"models.p\"\n",
    "word_pairs_filename = \"ready_to_use/word_pairs/French.csv\"\n",
    "selected_model_filename = \"best_model.p\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"loading models\")\n",
    "with open(models_filename, \"rb\") as handle:\n",
    "    models = pickle.load(handle)\n",
    "print(\"loaded {} models\".format(len(models)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_sim = 0\n",
    "best_pars = None\n",
    "for p, m in models.items():\n",
    "    similarities = m.wv.evaluate_word_pairs(word_pairs_filename, delimiter=\",\")\n",
    "    if similarities[0][0] > best_sim:\n",
    "        best_sim = similarities[0][0]\n",
    "        best_pars = p\n",
    "print(\"found best model with parameters: {}\".format(best_pars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save best performing model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(selected_model_filename, \"wb\") as handle:\n",
    "    pickle.dump(models[best_pars], handle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}