NLP Toolkit (Made for self use - but feel free to use it too :) )

A list of tools for NLP tasks.
Done mostly for self-reference... hence quite brief.

Better lists can be found out there:
https://github.com/keon/awesome-nlp


SciPy (sklearn.feature_extraction.text)
  • CountVectorizer - converting text to token-counts matrix (n-grams co-reference) which is then used by:
  • TfidfTransformer - transforms a count-matrix (CountVectorizer output) to term-frequency or inverse document frequency (TF-IDF)
  • Corpus reader - Words tokenization
  • POS
  • Chunking
  • Stemming
  • Creating parse trees out of sentences
  • Using KnowledgeBases (FrameNet, WordNet, propBank)
  • Different implementation for tagging (Senna, Stanford)
  • Corpus reader - Tokenization
  • NER, POS
  • Semantic representation (Word Vectors)
  • Labeled dependency parsing
Gensim:

  • Corpus reader / parser
  • transofrmations (TF-IDF, LSA, LDA, HDP)
  • Similarity Queries
  • Topic segmentation (LDA, LSA)
  • High-level trained models
  • Machine comprehension (QA based on given text)
  • Coreference Resolution (who/what does 'it', 'they' refers to)
  • NER
  • SRL 
  • Textual Entailment (similar to topic segmentation and similarity queries combined)

openNMT

  • Seq2seq mapping (Encoder - Decoder for translations)
  • Tagging
  • Classification
  • Also includes built-in WordEmbedder, Tokenizer
  • Based on Tensorflow

  • Attention-based encoder-decoder (machine translation)
  • Based on Theano through dl4mt tutorial

Corpus
http://universaldependencies.org (github repository: https://github.com/UniversalDependencies)
https://github.com/nlp-compromise/nlp-corpus (different categories)


GPU-based services to train and run models:

  • FloydHub (2h GPU, 20h CPU, 10GB)
  • PaperSpace (10$ credit - monthly costs start with 5$ for 50GB storage and 0.07$ per hour)
  • Kaggle (Free - Good for small, short-time training; execution are killed after ~5h; includes many cleaned datasets)

Comments

Post a Comment

Popular posts from this blog

pip install pymssql fails with 'sqlfront.h': No such file or directory

I've tried to install pymssql on Windows using command line: pip install pymssql The operation fails with an error: fatal error C1083: Cannot open include file: 'sqlfront.h': No such file or directory while looking for a solution, the first results  are not really helpful, until you arrive to this helpful one . In short: I was using python version 3.6, which currently isn't supported with pymssql. Using python 3.5 instead solves that issue, and the  pip install pymssql runs well. If you're using Anaconda Navigator , it is made even simpler: In the Environment section, you can add a new environment with your choice of python version. When you're choosing a supported version - 3.5 for example - you will find the pymssql in the packages list and install it from there directly.
Read more

ASP.NET 2.0 with MySQL 4.1.2 - unicode problem

Hey guys This is my first post and the reason i've actually started this blog. you see, it's now 3:00 AM here but i could not get sleep until I share this with the world :) during the last few days i struggled connecting MySQL database, version 4.1.2, through my ASP.NET application (using SubSonic DAL btw - great project, thanks guys!), and sending unicode utf-8 format data. The problem was that when I tried saving data (Insert or Update SQL Commands) i got question marks ('?????') saved instead of unicode characters, like hebrew for example. I've looked up at all the forums and I have tried everything. I mean really - EVERYTHING. For example, I've read that I should send, right after I'm establishing the connection, SET NAMES utf8 COLLATE 'utf8_unicode_ci'; SET CHARACTER SET utf8; So I've tried that... but no good, still got the question marks. I've downloaded the connector code and recompiled it. tried to see what does it sends. The ASP C
Read more

Natural Language Processing :: A beginner’s (brief) guide for working on NLP problems

This article  was written for the Data Science Society's Datathon held on February 2018, as an answer to questions from the participants who were new to NLP. The Data Science Society's  Datathon 2018 , presented us this time many cases which are Natural Language Processing related . One of the cases, for example, involves extracting entities' activities from unstructured documents, and determining their sentiment. So, how should one begin working on such a problem? First, let’s break this problem down: (1) we need to to detect which entities are mentioned in an article, and then (2) we need to detect the sentiment that is related, specifically to those entities. Let’s think about it logically for a moment: Entities are (normally) nouns, and in order to get the sentiment we will probably need to look at the adjectives that describe them, or the verbs that are related to them. So a first step could be (a) parsing the document into sentences, and then int
Read more