Analyzing Software Development using data science and analytic tools

Yesterday I had the pleasure to host Markus Harrer who gave a talk about analyzing software code using data science, which he describe in details in his blog.

There is often a communication gap between software developers and management. While good developers can see the big picture of the code, and timely identify a need to restructure or even rewrite the code, they often miss to see the risks of both, time and cost, of such an operation.

The management on the other hand, while being able to identify risks of getting into an adventure of rewriting a legacy system, often fail to understand the outcome and the risks of lack of maintenance.

The solution to overcome this 'gap of ignorance' is communicating using data science and analytics. Using Jupyter notebooks, for example, to combine both textual explanations as well as analytics and diagrams can easily communicate the risks of a software to the management.

For using data science on software code, one should decide what are the purpose of the analysis, and which features should be chosen to analyze this target.

The main targets that Markus was discussing were:

  • Un-maintained code (old code)
    Code that was not maintained more than 6 months, for example, is considered old. And as such, the company may be lacking the maintenance know-how of that modules.
  • Structure
    By analyzing the connection between different objects or function calls, one can find 'dead' parts that are not being called and can be removed from the system.
  • Performance
    By analyzing logs and execution timestamps, the bottlenecks of the software can be identified.
To illustrate these ideas, he had demonstrated it using Jupyter notebook and python.

Old code

By retrieving all the commits of a project from GitHub, in his example it was Linux, as a CSV which contains the user, timestamp, filename and line number, and by using Pandas to analyze it, he demonstrated with simple group_by statements, how to find the top contributers, the most and least maintained files (and with simple string manipulations also modules/folders) and using the delta between the current date and the timestamp column - also the files and modules which were not updated recently.

Code Structure

For analyzing the structure, he used an automated tool to convert Java OO code to a graph DB scheme (Neo4J). Every class was represented as a node, and the relation between classes as an edge.

The same can be also done for Function-oriented programming, by converting a function invocation as en edge between two nodes (which represent the functions).

When the graph is ready, it is easy to analyze it using graph-theory algorithms, such as finding isolated islands (dead code), barebones (the nodes with the most edges), etc.

Performance

To analyze the performance, he analyzed execution logs by converting them into a Graph as well. 
A similar approach for web development can be done by extracting the Performance execution from Chrome Developer's Tools, and converting it from a Tree to a Graph.

Since each of the function invocation includes a timestamp, it's easy to detect not only the heaviest functions, but also the heaviest function-chains, and tackle it later by code.

Interesting?

For more information, please visit his blog and his github repository for examples, the original talk slides are here, and feel free to feedback this post.

Keep up with our meetups, and we're also on facebook and twitter.

Comments

Post a Comment

Popular posts from this blog

pip install pymssql fails with 'sqlfront.h': No such file or directory

I've tried to install pymssql on Windows using command line: pip install pymssql The operation fails with an error: fatal error C1083: Cannot open include file: 'sqlfront.h': No such file or directory while looking for a solution, the first results  are not really helpful, until you arrive to this helpful one . In short: I was using python version 3.6, which currently isn't supported with pymssql. Using python 3.5 instead solves that issue, and the  pip install pymssql runs well. If you're using Anaconda Navigator , it is made even simpler: In the Environment section, you can add a new environment with your choice of python version. When you're choosing a supported version - 3.5 for example - you will find the pymssql in the packages list and install it from there directly.
Read more

ASP.NET 2.0 with MySQL 4.1.2 - unicode problem

Hey guys This is my first post and the reason i've actually started this blog. you see, it's now 3:00 AM here but i could not get sleep until I share this with the world :) during the last few days i struggled connecting MySQL database, version 4.1.2, through my ASP.NET application (using SubSonic DAL btw - great project, thanks guys!), and sending unicode utf-8 format data. The problem was that when I tried saving data (Insert or Update SQL Commands) i got question marks ('?????') saved instead of unicode characters, like hebrew for example. I've looked up at all the forums and I have tried everything. I mean really - EVERYTHING. For example, I've read that I should send, right after I'm establishing the connection, SET NAMES utf8 COLLATE 'utf8_unicode_ci'; SET CHARACTER SET utf8; So I've tried that... but no good, still got the question marks. I've downloaded the connector code and recompiled it. tried to see what does it sends. The ASP C
Read more

Natural Language Processing :: A beginner’s (brief) guide for working on NLP problems

This article  was written for the Data Science Society's Datathon held on February 2018, as an answer to questions from the participants who were new to NLP. The Data Science Society's  Datathon 2018 , presented us this time many cases which are Natural Language Processing related . One of the cases, for example, involves extracting entities' activities from unstructured documents, and determining their sentiment. So, how should one begin working on such a problem? First, let’s break this problem down: (1) we need to to detect which entities are mentioned in an article, and then (2) we need to detect the sentiment that is related, specifically to those entities. Let’s think about it logically for a moment: Entities are (normally) nouns, and in order to get the sentiment we will probably need to look at the adjectives that describe them, or the verbs that are related to them. So a first step could be (a) parsing the document into sentences, and then int
Read more