NLP Toolkit (Made for self use - but feel free to use it too :) )

A list of tools for NLP tasks.
Done mostly for self-reference... hence quite brief.

Better lists can be found out there:
https://github.com/keon/awesome-nlp


SciPy (sklearn.feature_extraction.text)
  • CountVectorizer - converting text to token-counts matrix (n-grams co-reference) which is then used by:
  • TfidfTransformer - transforms a count-matrix (CountVectorizer output) to term-frequency or inverse document frequency (TF-IDF)
  • Corpus reader - Words tokenization
  • POS
  • Chunking
  • Stemming
  • Creating parse trees out of sentences
  • Using KnowledgeBases (FrameNet, WordNet, propBank)
  • Different implementation for tagging (Senna, Stanford)
  • Corpus reader - Tokenization
  • NER, POS
  • Semantic representation (Word Vectors)
  • Labeled dependency parsing
Gensim:

  • Corpus reader / parser
  • transofrmations (TF-IDF, LSA, LDA, HDP)
  • Similarity Queries
  • Topic segmentation (LDA, LSA)
  • High-level trained models
  • Machine comprehension (QA based on given text)
  • Coreference Resolution (who/what does 'it', 'they' refers to)
  • NER
  • SRL 
  • Textual Entailment (similar to topic segmentation and similarity queries combined)

openNMT

  • Seq2seq mapping (Encoder - Decoder for translations)
  • Tagging
  • Classification
  • Also includes built-in WordEmbedder, Tokenizer
  • Based on Tensorflow

  • Attention-based encoder-decoder (machine translation)
  • Based on Theano through dl4mt tutorial

Corpus
http://universaldependencies.org (github repository: https://github.com/UniversalDependencies)
https://github.com/nlp-compromise/nlp-corpus (different categories)


GPU-based services to train and run models:

  • FloydHub (2h GPU, 20h CPU, 10GB)
  • PaperSpace (10$ credit - monthly costs start with 5$ for 50GB storage and 0.07$ per hour)
  • Kaggle (Free - Good for small, short-time training; execution are killed after ~5h; includes many cleaned datasets)

Comments

Post a Comment

Popular posts from this blog

pip install pymssql fails with 'sqlfront.h': No such file or directory

I've tried to install pymssql on Windows using command line: pip install pymssql The operation fails with an error: fatal error C1083: Cannot open include file: 'sqlfront.h': No such file or directory while looking for a solution, the first results  are not really helpful, until you arrive to this helpful one . In short: I was using python version 3.6, which currently isn't supported with pymssql. Using python 3.5 instead solves that issue, and the  pip install pymssql runs well. If you're using Anaconda Navigator , it is made even simpler: In the Environment section, you can add a new environment with your choice of python version. When you're choosing a supported version - 3.5 for example - you will find the pymssql in the packages list and install it from there directly.
Read more

ASP.NET 2.0 with MySQL 4.1.2 - unicode problem

Hey guys This is my first post and the reason i've actually started this blog. you see, it's now 3:00 AM here but i could not get sleep until I share this with the world :) during the last few days i struggled connecting MySQL database, version 4.1.2, through my ASP.NET application (using SubSonic DAL btw - great project, thanks guys!), and sending unicode utf-8 format data. The problem was that when I tried saving data (Insert or Update SQL Commands) i got question marks ('?????') saved instead of unicode characters, like hebrew for example. I've looked up at all the forums and I have tried everything. I mean really - EVERYTHING. For example, I've read that I should send, right after I'm establishing the connection, SET NAMES utf8 COLLATE 'utf8_unicode_ci'; SET CHARACTER SET utf8; So I've tried that... but no good, still got the question marks. I've downloaded the connector code and recompiled it. tried to see what does it sends. The ASP C...
Read more

C#, Entity Framework, WCF and JSON serialization

Ever tried to serialize Entity Framework with WCF? Have you tried it with the Lazy-Initialization feature? And how about the Proxy Creation Enabled? There's a problem of circular reference if one table is referencing another table - which referencing the first one. We have had a great deal of problems with it. We even tried to consult an external expert of Entity Framework who simply told us it's impossible to serialize it on WCF. oh darn... :( We tried many solutions before we gave up: like that one: http://geekswithblogs.net/danemorgridge/archive/2010/05/04/entity-framework-4-wcf-amp-lazy-loading-tip.aspx and that: http://blogs.microsoft.co.il/blogs/gilf/archive/2011/10/17/avoiding-circular-reference-for-entity-in-json-serialization.aspx We even used t4templates to create a whole new set of flattened objects (!!!) But it doesn't really make life easier as one has to load separately each and every object. But this was not what we were looking for. We needed a solution that...
Read more