Liad's little blog

Yesterday I had the pleasure to host Markus Harrer  who gave a talk about analyzing software code using data science, which he describe in details in his blog . There is often a communication gap between software developers and management. While good developers can see the big picture of the code, and timely identify a need to restructure or even rewrite the code, they often miss to see the risks of both, time and cost, of such an operation. The management on the other hand, while being able to identify risks of getting into an adventure of rewriting a legacy system, often fail to understand the outcome and the risks of lack of maintenance. The solution to overcome this 'gap of ignorance' is communicating using data science and analytics. Using Jupyter  notebooks, for example, to combine both textual explanations as well as analytics and diagrams can easily communicate the risks of a software to the management. For using data science on software code, one should decide...

This article  was written for the Data Science Society's Datathon held on February 2018, as an answer to questions from the participants who were new to NLP. The Data Science Society's  Datathon 2018 , presented us this time many cases which are Natural Language Processing related . One of the cases, for example, involves extracting entities' activities from unstructured documents, and determining their sentiment. So, how should one begin working on such a problem? First, let’s break this problem down: (1) we need to to detect which entities are mentioned in an article, and then (2) we need to detect the sentiment that is related, specifically to those entities. Let’s think about it logically for a moment: Entities are (normally) nouns, and in order to get the sentiment we will probably need to look at the adjectives that describe them, or the verbs that are related to them. So a first step could be (a) parsing the document into sentences, and then int...

A list of tools for NLP tasks. Done mostly for self-reference... hence quite brief. Better lists can be found out there: https://github.com/keon/awesome-nlp SciPy ( sklearn.feature_extraction.text ) CountVectorizer - converting text to token-counts matrix (n-grams co-reference) which is then used by: TfidfTransformer - transforms a count-matrix ( CountVectorizer output) to term-frequency or inverse document frequency (TF-IDF) NLTK Corpus reader - Words tokenization POS Chunking Stemming Creating parse trees out of sentences Using KnowledgeBases (FrameNet, WordNet, propBank) Different implementation for tagging (Senna, Stanford) Spacy Corpus reader - Tokenization NER, POS Semantic representation (Word Vectors) Labeled dependency parsing Gensim : Corpus reader / parser transofrmations (TF-IDF, LSA, LDA, HDP) Similarity Queries Topic segmentation (LDA, LSA) AllenNLP High-level trained models Machine comprehension (QA based on given t...

I've tried to install pymssql on Windows using command line: pip install pymssql The operation fails with an error: fatal error C1083: Cannot open include file: 'sqlfront.h': No such file or directory while looking for a solution, the first results  are not really helpful, until you arrive to this helpful one . In short: I was using python version 3.6, which currently isn't supported with pymssql. Using python 3.5 instead solves that issue, and the  pip install pymssql runs well. If you're using Anaconda Navigator , it is made even simpler: In the Environment section, you can add a new environment with your choice of python version. When you're choosing a supported version - 3.5 for example - you will find the pymssql in the packages list and install it from there directly.

Hi, This problem was really bothering me for few months. In earlier cases I tried to avoid it, but today I could not hide from it any longer. And the thing is that there was no post on any other place that could solve this matter... So here it goes. You're writing a directive in AngularJS , and you want to access one of the DOM elements declared within its template. Usually you do it through the linking function (post section in the compile) or in the directive controller, by injecting and using '$element', as explained here . But, in my case, I had to access an element using its ID and this ID was dynamic. So, my template contains:     < div >     < div ng-attr-id ="{{ 'something_' + dynamicValue }}"></ div > </ div >       And my directive contains: {     templateURL: 'pathToTempalte' ,     scope: {         dynamicValue: '...

Since the invention of NOSQL databases, it gets more and more attention from the developer community. One of the remarkable and unique databases exists in this topic of NOSQL is the graph database of NEO4J, an open-source database which stores the data as a Graph. Modeling a graph database is quite different than modeling the regular RDBMS database, and even from other NOSQL databases such as key-value collections. We got used to identifying the molecularity of the data and save it as the columns, joining similar data together into tables. But since Neo4J is a flexible graph database, this case does not work there. In order to model the data, we need to identify first the queries that we're about to run on the database. These queries will form the basic logical sentences which are the keys of modeling the database (more about that coming up in a second). We need to identify where to store each piece of information: - as a node - as an edge - as a property of a node or of ...

I recently had to create an animation, which obey to the laws of physics, using javascript and HTML5. Among the gravity and free falling, one of these animations had to also apply sort of bouncing and elastic laws. After creating it first with CSS3 and HTML, we realized that most of the Android devices does not support it fully yet. Especially versions prior to Ice Cream Sandwich, such as Android 2.1, 2.2, 2.3 and lower (Froyo and its siblings). So, I had to transfer everything into HTML5 Canvas, and instead of applying the css-transitions, I had to implements everything alone. The first website I encountered and would like to highly recommend is Timothee Groleau's easing generator which can be found here : http://timotheegroleau.com/Flash/experiments/easing_function_generator.htm Originally made for Flash - but can easily adopted to Javascript HTML5 too. The output of his generator is basically a function describes a certain movement of an object. Such...

Apparently, generating a treePanel from a treeStore is not 'out-of-the-box' like generating a grid or even generating tree from static json in Ext-JS. In addition, at the moment there are not a many examples out there that explains how it should be done. Most of the examples are using a static JSON (memory). But what if you are using a store with an already-made JSON which you need to customize to be a tree? listeners to the rescue The way to make it happen, is to alter the store a bit. we start by defining a store as usual: Ext.define('MyTreeStore', { extend: 'Ext.data.TreeStore', config: { someConfig: 0 }, constructor: function (cfg) { var me = this; cfg = cfg || {}; me.callParent([Ext.apply({ autoLoad: true, storeId: 'MyTreeStoreID', root: { expanded: true }, proxy: { type: 'rest', url: 'http:/...

Ever tried to serialize Entity Framework with WCF? Have you tried it with the Lazy-Initialization feature? And how about the Proxy Creation Enabled? There's a problem of circular reference if one table is referencing another table - which referencing the first one. We have had a great deal of problems with it. We even tried to consult an external expert of Entity Framework who simply told us it's impossible to serialize it on WCF. oh darn... :( We tried many solutions before we gave up: like that one: http://geekswithblogs.net/danemorgridge/archive/2010/05/04/entity-framework-4-wcf-amp-lazy-loading-tip.aspx and that: http://blogs.microsoft.co.il/blogs/gilf/archive/2011/10/17/avoiding-circular-reference-for-entity-in-json-serialization.aspx We even used t4templates to create a whole new set of flattened objects (!!!) But it doesn't really make life easier as one has to load separately each and every object. But this was not what we were looking for. We needed a solution that...

Recently we've started using the new Ext-JS 4.0 class system. one of the obstacles I came across with was using the configuration while making inheritance. On most of the examples in the web - it works seamlessly perfect. but what happens when you try to extend an existing object? for example, 'Ext.panel.Panel' ? Suddenly - weird stuff happens and it doesn't work any longer. So for example: Ext.define('ProductionMap.popup.serverExplorer', { extend: 'Ext.panel.Panel', config: { ParameterConfig: '', }, constructor: function (cfg) { this.initConfig(config); return this; }, initComponent: function () { ... } will give an error. took me an hour to figure it out - but I found the solution and it's REALLY simple. It's all about the constructor: instead of that code, use: constructor: function (cfg) { this.initConfig(config); this.callParent([config]); return this; }, ...

A bit late, I admit, but finally I've started using Sencha's ExtJS framework. The first project I was executing was involving charts with JSON data loaded from a C# ASP.NET WCF service. I have encountered several problems and a lack of proper documentation. Many of the solutions were finally found in another websites and forums but there was no blog post I have found that had everything together in one example. So I'm proud to be one of the first who does that. I will also describe the problems I've encountered and of course - the solutions. In ExtJS, the code is created in Windows , Panels or Forms . The latter, Form , can not be added to Asp.net page as it regenerates a <form> tag, which can not be placed under the already-generated <form runat="'server'"> of the asp.net. I've used the Ext.Panel then, in an external JS file, like so: Ext.onReady(function () { var myPanel = new Ext.Panel({ }); }); inside the panel, among the rest o...

While developing a CMS for small gallery websites, I decided to store only the original uploaded image and not to create a thumbnail upon uploading. I didn't want, of course, to use the html resizing feature, especially not while uploading high res images. Instead, I created a generic handler that creates the thumbnail image on the fly. Not only this handler knows how to resize the image, but also to crop it, in case I want all the images to be on a fixed ratio of width and height. So, how is it done? There are many blog posts of resizing images on the fly. where you'll find a code similar to this: public static Bitmap CreateThumbnail(Bitmap loBMP, int lnWidth, int lnHeight) { System.Drawing.Bitmap bmpOut = null; try { ImageFormat loFormat = loBMP.RawFormat; decimal lnRatio; int lnNewWidth = 0; int lnNewHeight = 0; //*** If the image is sm...

Recently I've tried to user the new version of SubSonic (3.0.0.3) with MySQL. for two days i was struggling to make it work. I've tried the new MySQL .NET connector (6.1) and every time got errors. Eventually, I've downloaded the code of both, MySQL and SubSonic and tried to understand what the heck they are doing there. I've used the MySQL t4 that was supplied with the subsonic 3, renamed it to SQLServer.ttinclude and it still didnt work. After breaking it down and solving one part after the other i've realized that the MySQL include file is missing support of Stored Procedures. So I've rewrote it out of the SQL Server template, and using the MySQL Schema. Here it is: List GetSPParams(string spName){ var result=new List (); string[] restrictions = new string[4] { DatabaseName, null, spName, null }; using(conn=new MySqlConnection(ConnectionString)){ conn.Open(); var sprocs=conn.GetSchema("PROCEDURE PARAMETERS", restrictions); c...

Recently i encountered a problem which was hard finding a solution in the internet: setting the focus to the flash object in firefox and chrome when loading the website. I tried using this idea: <body onload="window.document.FlashName. focus ();"> <object name="FlashName" ... /> <embed name="FlashName" ... /> </object> </body> but while this works in IE it's not working in firefox at all. I also tried detecting the browser and focusing the embed instead of the object in firefox, but turns out that the embed can't take focus easily. Eventually i've encountered this post: http://www.asserttrue.com/articles/2006/11/17/firefox-and-flash-swf-selection-and-focus-problems which solved this issue. In order to focus flash in both firefox and IE, there're 2 things need to be done: 1. add wmode="opaque" to the object and to the embed 2. address the flash by name and not by ID or separating the code for IE and F...

Hey guys This is my first post and the reason i've actually started this blog. you see, it's now 3:00 AM here but i could not get sleep until I share this with the world :) during the last few days i struggled connecting MySQL database, version 4.1.2, through my ASP.NET application (using SubSonic DAL btw - great project, thanks guys!), and sending unicode utf-8 format data. The problem was that when I tried saving data (Insert or Update SQL Commands) i got question marks ('?????') saved instead of unicode characters, like hebrew for example. I've looked up at all the forums and I have tried everything. I mean really - EVERYTHING. For example, I've read that I should send, right after I'm establishing the connection, SET NAMES utf8 COLLATE 'utf8_unicode_ci'; SET CHARACTER SET utf8; So I've tried that... but no good, still got the question marks. I've downloaded the connector code and recompiled it. tried to see what does it sends. The ASP C...