As Electi reaches it's 9th birthday, we have gone major shifts in the industry. 

Last year marked the end of scarcity in code, now everything revolves around leveraging LLM's and AI to create something. 

However building something of value in a controlled environment with such immence speed requires having absolute control of the process. Since 2017 we have worked with banks, regulators, insurers and shipping groups on the buying side of AI. 

Follow one product from first pilot to scale inside organisations like these, and the same foundations appear in order. 

1. It starts with a job that has an owner, a deadline and a budget, like closing the month-end books. Jensen Huang points to the same place: the biggest opportunities sit inside real-world problems. The best teams stay in that one segment until their first 100 users cannot work without them. Those users outweigh 100,000 who tried the demo once. 

2. Then the buyer asks the first hard question: how do you know it is right? Evals answer it, so they come before features. They also make the architecture safe to change. Plan to swap models twice a year. With a test set that shows within a day whether the new model performs better on real tasks, a swap takes two weeks. Without one, it takes a quarter. Next to evals sits the trust layer: permissions, audit logs, human in the loop. Enterprise buyers pay for control. 

3. Next comes money. Inference is COGS, so measure the cost of each completed task before you set a price. Then price the outcome. Seat pricing shrinks as agents replace headcount, which means revenue falls just as the product starts working. Outcome pricing holds up when both sides share a definition of done they can audit. Covey's advice applies: begin with the end in mind. 

4. After that, the product starts to compound. Every completed task feeds a loop of usage data that only you hold. The same usage earns trust across your client base, and that trust becomes your distribution. A competitor can rebuild your product in 6 weeks. Earning those relationships takes years. Eric Ries put the discipline plainly: the only way to win is to learn faster than anyone else. 

Read the journey again. Every stage is an operating discipline: focus, measurement, control, pricing, learning. Grove, Covey and Horowitz wrote that playbook decades before the first LLM, and it still applies. 

build the loop <-> rent the model

 This quote hit hard : "Nobody wants to read something that you didn’t bother writing."

also : "Artisanal, hand made slop"

Not your average LI slop:

 

 

It's been a long time since i posted here.

We have moved to the agentic era, a new shitstorm of bad software is coming online.

Software that is sloppy, brittle and unrefined.

The main culprit of fragility at this current period is the agentic part. 

Your agent has one version number and four things that decide what it actually does.

> Ship a change to the orchestration code, tag it, done. That covers one of the four.

>> The instructions and policy block is the second. Also yours, also the one everyone treats as the whole system, right up until the third and fourth parts move without telling you.

>> The model runtime is the third. A vendor's pinned snapshot, sitting in Active, Legacy, Deprecated, or Retired, on that vendor's own lifecycle, with a minimum notice period before it goes away. Anthropic's is 60 days. That is one company's policy. It binds nobody else's model, and it does not ask what your regression suite thinks.

>>> The fourth is the protocol and tools your agent calls out through. MCP now stamps a dated version string on every single request, which is more discipline than most teams manage on their own code. The tool schemas sitting behind that protocol still version on their own schedule, with no such stamp, and no one upstream is watching your integration tests when they change.

Most teams have a git log for the first two and nothing for the last two, then act surprised when the agent that passed every eval last month answers differently this month with no local commit to blame.

I put the rest of it in the attached deck: the release pipeline for a system that cannot be tested to certainty, what decides whether a rollback is even possible, and the four OWASP agentic-risk entries that are this problem read as a containment question.

 

Presentation here: https://drive.google.com/file/d/1rAghJ-Wcp57gFey1caBpA8xKWenz28eB/view?usp=drive_link 

Keras/Tensorflow threshold with gradient flow

Tensorflow / Keras threshold operations break the gradient flow.

There is a way to fix this by using a combination of operations.

def threshold_min_max_value(input_layer,
                            min_value=0.0,
                            max_value=1.0):
    """
    Thresholds all the values of the a tensor that exceed value to that
    max_value, and than are lower than the min_value,
    this layer retains gradient flow

    :param input_layer: the input layer
    :param min_value: minimum value to threshold to
    :param max_value: maximum value to threshold to
    :return: threshold-ed input layer
    """

    def _threshold(_x):
        ge_max_value = K.greater_equal(_x, max_value)
        ge_max_value = K.cast_to_floatx(ge_max_value)
        lt_max_value = 1.0 - ge_max_value

        le_min_value = K.less_equal(_x, min_value)
        le_min_value = K.cast_to_floatx(le_min_value)
        gt_min_value = 1.0 - le_min_value

        tmp0 = keras.layers.Multiply()([
            lt_max_value, gt_min_value, _x
        ])
        return tmp0 + (min_value * le_min_value) + (max_value * ge_max_value)

    return keras.layers.Lambda(_threshold)(input_layer)

Fix Tensorflow Object Detection Framework taking too much disk space while training

The model_lib_tf2.py file contains all the training and evaluation loops.

In the function:

def eager_train_step(detection_model,
                     features,
                     labels,
                     unpad_groundtruth_tensors,
                     optimizer,
                     learning_rate,
                     add_regularization_loss=True,
                     clip_gradients_value=None,
                     global_step=None,
                     num_replicas=1.0):

We can see that every training iteration it saves a few training dataset images.

tf.compat.v2.summary.image(
    name='train_input_images',
    step=global_step,
    data=features[fields.InputDataFields.image],
    max_outputs=5)

There are three problems with this:

1. It takes A LOT of space

2. It actually slows up training 

3. Images look saturated.

We can fix this easily by replacing the above snippet with this:

if global_step % 100 == 0:
    # --- get images and normalize them
    images_normalized = \
        (features[fields.InputDataFields.image] + 128.0) / 255.0
    tf.compat.v2.summary.image(
        name='train_input_images',
        step=global_step,
        data=images_normalized,
        max_outputs=5)

 

Gaussian Filter in Keras (code snippet)

Very often we need to perform basic vision operations on a computational graph like building a Laplacian pyramid or filter a tensor with a specific precalculated filter. 

Below i present a code snippet for building a fixed non-trainable gaussian filter in keras.

import keras
import numpy as np
import scipy.stats as st

def gaussian_filter_block(input_layer,
                          kernel_size=3,
                          strides=(1, 1),
                          dilation_rate=(1, 1),
                          padding="same",
                          activation=None,
                          trainable=False,
                          use_bias=False):
    """
    Build a gaussian filter block
    :return:
    """

    def _gaussian_kernel(kernlen=[21, 21], nsig=[3, 3]):
        """
        Returns a 2D Gaussian kernel array
        """
        assert len(nsig) == 2
        assert len(kernlen) == 2
        kern1d = []
        for i in range(2):
            interval = (2 * nsig[i] + 1.) / (kernlen[i])
            x = np.linspace(-nsig[i] - interval / 2., nsig[i] + interval / 2.,
                            kernlen[i] + 1)
            kern1d.append(np.diff(st.norm.cdf(x)))

        kernel_raw = np.sqrt(np.outer(kern1d[0], kern1d[1]))
        # divide by sum so they all add up to 1
        kernel = kernel_raw / kernel_raw.sum()
        return kernel

    # Initialise to set kernel to required value
    def kernel_init(shape, dtype):
        kernel = np.zeros(shape)
        kernel[:, :, 0, 0] = _gaussian_kernel([shape[0], shape[1]])
        return kernel

    return keras.layers.DepthwiseConv2D(
        kernel_size=kernel_size,
        strides=strides,
        padding=padding,
        depth_multiplier=1,
        dilation_rate=dilation_rate,
        activation=activation,
        use_bias=use_bias,
        trainable=trainable,
        depthwise_initializer=kernel_init,
        kernel_initializer=kernel_init)(input_layer)

from my open source project https://github.com/NikolasMarkou/multiscale_variational_autoencoder

Tensorflow to Onnx (tf2onnx) testing with different opsets

To test tf2onnx changes with different operators set your external variable prior to calling pytest.

So for example in a windows setup this will run the tests

set TF2ONNX_TEST_OPSET=11

pytest

and in a standard linux setup

export TF2ONNX_TEST_OPSET=11

pytest

Opencv 4.3.0 options

Building OpenCV 4.3.0 from source

cmake ../ -DHAVE_MKL=ON -DBUILD_opencv_python2=OFF -DBUILD_opencv_java=OFF -DMKL_WITH_TBB=ON -DWITH_OPENCL=ON -DWITH_CUDA=ON -DWITH_CUBLAS=ON -DENABLE_CXX11=ON -DOPENCV_EXTRA_MODULES_PATH=/home/arxwn/Repositories/opencv/opencv_contrib/modules -DCPU_BASELINE="SSE3"  -DCPU_DISPATCH="SSE4_1;SSE4_2;AVX;FP16;AVX2;AVX512_SKX" -DOPENCV_ENABLE_NONFREE=ON

 

Netron an awesome neural networks visualization tool

I've been working recently on optimizing models from tensorflow to onnx and finally to tensorrt and got introduced to Netron an excellent tool for all ml engineers / data scientists working with different flavors and formats of neural networks. 

It is a huge improvement over the keras and tensorboard visualizer. 

Check it out here

Compiling Tensorflow 1.15 from source

Following my previous post Compiling Tensorflow under Debian Linux with GPU support and CPU extensions I was trying to build version r1.15 when i stumbled to another peculiar bug. When reaching the end of the compilation you get a lovely error in the form of

from keras.preprocessing import image as image_utils
ImportError: No module named keras.preprocessing

To fix this you you just need to install the following packages prior to compiling




pip install keras_applications==1.0.4 --no-deps
pip install keras_preprocessing==1.0.2 --no-deps
pip install h5py==2.8.0

And that's it you can build again.


bazel build -c opt --config=v1 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Software Commodification

Something that people outside the software industry don't fully grasp is the increasing pace of commodification. 

You had a Phd in CS 20 years ago ? congrats all your knowledge is a library. 

You were an excellent physicist 10 years ago ? Congrats we can import and run everything you knew without even understanding half of it. 

You studied Artificial Intelligence in depth 15 years ago ? A 17 year old with a weekend in keras can provide better solutions than you. Everything you know is getting obsolete in an increasing rate. 

As software people we are are used to packaging up our knowledge and re-learning new skills (tools, theories, frameworks) every couple of years, because the old have been completely commoditized. 

It is an exciting but tiresome journey that it shouldn't be required by all. However as everything is becoming software there may not be a choice by most people. 

Personally the increasing pace of commodification allows me to cut through the noise and add depth to my knowledge. Knowing that the low hanging fruits have been picked will clear up the space of Artificial Intelligence and Machine Learning.

What if you could push your AI models to be 10 times faster?

Deceptive Easy

Modern Machine Learning (ML) and especially Deep Learning (DL) have become deceptively easy. It is almost trivial to have something up and running with presentable results as the the tools hide most of the complexity and hard decisions. Whereas that might be good enough for a Proof of Concept (POC) or a Minimum Viable Product (MVP), ensuring stability, high performance and scalability is a whole different ball game.

High Sunk Cost 

Many CXO’s and seniors managers find themselves trapped into subpar solutions that cannot be used effectively because they lack the technical know-how to productionalize them.

At Electi we have just release a new brochure listing the services we can offer to companies already implementing Deep Learning. Check it out here.

Performance Measures for Machine Learning

Summary

In this blog spot I'm presenting a few performance measures for machine learning tasks. These performance measures come up a lot in the marketing domain.

Take at home lessons:

• the measure you optimize to makes a difference
• the measure you report makes a difference
• use measure appropriate for problem/community
• accuracy often is not sufficient/appropriate
• only accuracy generalizes to >2 classes
• this is not an exhaustive list of performance measures

 

Confusion Matrix

First we construct a confusion matrix for a binary classification problem. Given a classification function f(x)->R and a threshold T that can split the outcomes into {0, 1} we can create a confusion matrix that counts the occurrences of the predicted class given the true label.

 

Accuracy

Accuracy is then measured as the percentage of correct responses (True Positives + True Negatives) over the total amount of responses.

 

Problems with Accuracy

This measure is commonly used but it can be misleading. The problems arise from the domain we are modelling. If one of the class for example is poorly represented into the metric is meaningless as we could predict the same class always and still have a good results.

• Assumes equal cost for both kinds of errors 

• cost(b-type-error) = cost (c-type-error)

• is 99% accuracy good?

• can be excellent, good, mediocre, poor, terrible
• depends on problem

• Base Rate = accuracy of predicting predominant class

 

Weighted (Cost sensitive) Accuracy

A modified version of accuracy is "Weighted Accuracy" were we count the cost of misclassification.

In this scenario we aiming for a model and a threshold that can minimize the total cost.

 

Lift

If we are not interested in the accuracy on the entire dataset but want accurate predictions for 5%, 10% or 20% of the dataset then we can use the lift measure.

Lift measures how much better than random prediction on the fraction of the dataset predicted true (f(x) > threshold).

 

Precision / Recall

• The Precision measure counts how many of the interest class are correct.
• The Recall measure counts how many of the interest class does the model return.

In the case below the interest class is a(1). 

We can change the sweep over the threshold calculate Precision/Recall multiple times and graph out what is called the Precision/Recall curve.

At each different threshold we can see a different tradeoff between the two metrics.

• When the threshold is too high then c (everything is predicted as class 0) becomes zero and then the precision becomes zero.
• When the threshold is too low then b (everything is predicted as class 1) becomes zero and then the recall becomes zero.

Both of these metrics are flawed in isolation and it is the eye of the modeller on which one better represents the problem.

F-Measure

The F-Measure is an attempt to merge the two measures to construct a more meaningful performance measure.

Receiver Operating Characteristic (ROC)

• Developed in WWII to statistically model false positive and false negative detections of radar operators
• Better statistical foundations than most other measures
• Standard measure in medicine and biology
• Becoming more popular in ML 

Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice.

One of the earliest adopters of ROC graphs in machine learning was Spackman (1989), who demonstrated the value of ROC curves in evaluating and comparing algorithms.

ROC graphs are conceptually simple, but there are some non-obvious complexities that arise when they are used in research. 

ROC Plot

 

• Sweep threshold and plot

• TPR vs. FPR
• Sensitivity vs. 1-Specificity
• P(true|true) vs. P(true|false)

• Sensitivity = a/(a+b) = Recall = LIFT numerator
• 1 - Specificity = 1 - d/(c+d)

A ROC graph depicts relative trade-offs between benefits (true positives) and costs (false positives).

• The lower left point (0,0) represents the strategy of never issuing a positive classiffication. 
• The opposite strategy is represented by the upper right point (1,1).
• The point (0,1) represents perfect classiffication.
• The diagonal line y = x represents the strategy of randomly guessing a class. 
• A random classifier will produce an ROC point that "slides" back and forth on the diagonal based on the frequency with which it guesses the positive class. In order to get away from this diagonal into the upper triangular region, the classifier must exploit some information in the data. 
• Any classifier that appears in the lower right triangle performs worse than random guessing. This triangle is therefore usually empty in ROC graphs.
• ROC curves have an attractive property: they are insensitive to changes in class distribution.
• Any performance metric that uses values from both columns of theconfusion matrix will be inherently sensitive to class skews. Metrics such as accuracy, precision, lift and F scores use values from both
  columns of the confusion matrix. 
• ROC graphs are based upon TP rate and FP rate, in which each dimension is a strict columnar ratio, so do not depend on class distributions.