Enterprise Tensorflow -Python vs. Java

Why not just use Python for everything?

Tensorflow and Python go hand in hand, so when we think about using Tensorflow, we think of doing so in Python. The vast majority of the Tensorflow API is only available in Python, so when we develop our models and train them, there is pretty much no alternative to using Python (proof-of-concept or esoteric approaches not withstanding). Depending on the environment where we actually want to use our model, however, other languages might be better suited to run our model, especially when we cannot or do not want to install a Python runtime. (For reasons where a Python based approach might not be warranted, see below.) In cases where Python is not readily available, we need to remember that the actual implementation of the computation graph in Tensorflow is written in C++, i.e. it is a native binary. Hence, it can be made to run within most other languages. For many languages, Google already provides such bridging for running the graph. If the Tensorflow wrappers are not an option, there is still the possibility of using the parameterization we have trained in Tensorflow (e.g. our neural network weights) in another framework that is more readily available. In the rarest of cases where neither Tensorflow nor another framework is available for inference, it helps that inference is often much easier implemented than training, so we even have the possibility, to implement inference ourselves and just use the learned parameters like we would when using another ML Framework. In later posts we well look at each of these possibilities in turn.

When might Python not be an option?

The platform on which we want to run our model might constrain us:

• Mobile: You cannot / should not embed a complete python environment in your native App
• Native desktop app: A sleek small installer is wanted.
• Embedded: Maybe there are not enough resources to add a Python environment (though Python can run on many “small” platforms)

The project might put constraints on your options:

• Customer requirements (whether justified or not) might not allow you to use Python.
• Political reasons (you have to get the right people on board) - this is especially a case in large corporations.
• Time consuming certification / approval processes for each new component not already in use - again, a typical case for large corporations
• Key people in the decision process are against it - and all arguments fail
• Team / organizational concerns: the team building the software that runs the model is different from the developers of the model, the “Python people” might not be around for the whole lifetime of the product

There are other benefits that may not forbid usage of Python, but still make a non-Python solution more attractive in some cases:

• keeping dependencies to a minimum
• keeping the build simple
• keeping the installer simple / the deployment process simple
• keeping updates simple
• keeping artifact size to a minimum
• you may not want to use multiple programming languages in your project
• you do not like Python (preferences do differ, after all)

Why use Java instead?

The above arguments apply for many alternative environments to Python. In the rest of this post and the following series we will focus on one alternative specifically, Java for Server Side applications, especially in conjunction with “Enterprise” frameworks. In direct response to the arguments above here are reasons why Java sometimes might be a better alternative (as software development is a complicated affair, each case must of course be judged individually):

Platform constraints:

Except for Android, where we can regard Java as a first class citizen, platform constraints that apply to Python will generally also apply to Java (we need a separate runtime environment etc.). As we do focus on server side Java, we do not regard the Android case here. Also Google is heavily invested in this area (see Tensorflow Light), so there is plenty of discussion of this scenario elsewhere. So generally, Java will not help you with platform constraints. But as we focus on server side development, luckily the platform is rarely the problem for us. Servers these days have plenty RAM, Storage and CPU power.

Project constraints:

This is the area where Java often shines. The magic incantation “We can offer you a 100% Java Enterprise Solution” opens the gates to easy project approval, larger budgets and fast acceptance. (Pro Tip: Wear a nice suit while you say it). On a more serious note, Java is one of the most widely used frameworks in the business world, especially for large companies like insurances, banks, broadcasting and industry. In those environments, a tried-and-true technology with a vibrant ecosystem, commercial support and a large potential workforce will (almost) always beat the next cool RAD technology. Hence if there is a policy that enforces the use of certain technologies, chances are good that Java (and certain Java frameworks) are among them. One also often encounters entrenched teams that have taken care of an IT project for years and are hesitant to change and might torpedo your new and cool AI projects with FUD. Being able to play the Java Enterprise card may just be the kicker you need to make your hand. Being able to use the existing team to keep the new and shiny AI solution running once it is developed, even if the team that built it is not around anymore, will let managment sleep easy and make your work a lot easier.

Other Benefits

So far we have mainly argued why a Java based integration might be a good idea if existing infrastructure / teams / organizations are already heavily invested in the Java ecosystem. But sometimes Java might even make sense for a clean slate project. What follows are some points highlighting the strength of Java in general and for a server side application in particular. As always, these opinions come with a dose of individual bias and do not claim that this is the silver bullet solving all problems. There is always more than one way to do it. Java has - as far as the “coolness” is concerned - fallen somewhat out of favor in the last years. One of the reasons being that the cathedral-building over-engineering enterprise approach has often proven to be inefficient and unnecessarily complex. But as these things often do, programming fashions come and go and in response to that (well-based) criticism many things have changed in Java land that have improved the situation for your classical Java Server Enterprise Project. Java is:

• Fast. Surprisingly so, in my experience often beating hand optimized C code thanks to the JIT Comiler. While computing power is cheap, being an order of magnitude more efficient than the last cool JavaScript, Ruby etc. framework might reduce your server cost significantly.
• Mature. Especially many of the Apache libraries have matured over years, and have an impressive track record regarding stability. Often new, fancy frameworks look awesome at first glance but fail when real-world edge cases come along.
• Not as bloated as it used to be. New frameworks also allow for a RAD approach more commonly found in Ruby on Rails or Django - we do not spend one week writing XML for a “Hello World” server application anymore. Also Java 8 finally brings us FP and libraries like guava allow for well readable concise code. While we will never be es elegant as Haskell things have gotten a lot better.
• Well supported. The documentation of the language, tooling and libraries is extremely extensive.
• Statically typed. If this is a pro or a con of course very much depends on your personal programming style.

As a sidenode as of now (2017) my stack recommendation for new Java server projects would be:

• Java 8
• A tried and true SQL database (e.g. PostgreSQL)
• JOOQ for the persistence layer
• Ninja as web framework (CXF, Spring etc. are not that bad either)
• any utility libraries that make your life easier, especially Guava

Using this stack, Java development is actually fun again - try it! And of course lastly many of the above points do not only apply to Java but to many other JVM based languages. All the points we will present in the following posts will apply to most JVM based languages that allow easy Java interop, like Kotlin, Scala, Clojure and others.

Where to go from here?

If after this discussion you end up using Java for the integration of a Tensorflow model into a production environment there remains the question: how to best do this? This again depends on a number of factors. In the next few posts we will discuss the various aspects of such an integration. Specifically we will cover:

• How to get the parameterization (or the whole graph) from your Training code into your server
• How to load that data and run the inference in the server
• How to design your client side facing server interface for various inference scenarios
• How to set up a build chain to automize this integration to varying degrees
• How to organize your projects