In {T210844} we'll be generating TSV files that live in HDFS. We want to take these TSV files and import them the m2-master database.
#Analytics is working on a solution to take data from the analytics cluster to the production servers at {T213976}.
We have import scripts that take TSVs and import them into MySQL. Where should these scripts be executed?
== Summary of discussion so far
How to import article recommendation scores to MySQL from Hadoop?
- Transfer data to a production node that has access to the MySQL
database and run the import script ...
1. From the recommendation-api service node. #Services, is it viable?
2. From some production virtual machine. Which one? #Operations which one is
a good for this use case? Should we request a dedicated VM (Ganeti
instance)?
- Write directly from Hadoop to MySQL. #Operations what are the issues here?
- Running import scripts from stats machines or database host machines
has been ruled out.
== Links
- [[ https://gerrit.wikimedia.org/r/admin/projects/research/article-recommender-deploy | import scripts ]]
- [[ https://meta.wikimedia.org/wiki/Research:Technology/Article-Recommendation-Pipeline-Overview | Production Pipeline overview ]]
== A/C
- [ ] Decide where and how to execute the import scripts.
- [ ] Implement the agreed-upon solution.