Train A Deep Learning Model, Implement F# Generic Forward Propagation, Integrate It In Xamarin Forms Mobile App

In our previous post, we saw how to develop a full deep learning model from scratch in F#, and we trained it to detect Cat Vs NonCat images. In this post, we will see how to integrate a train model in a .Net application. More specifically, a Xamarin Forms cross platform app that allow user to select a picture from the gallery and perform a cat detection on it. It may sounds a little bit like Jin Yeng hot dog detector app from Sillicon Valley TV show :

 

This how our app will actually work :

 

Here are the steps we will follow to achieve our POC :

  1. Develop and train our model using TensorFlow in python.
  2. Export trained weights and bias to a JSON file.
  3. Develop a .NetStandard 2.0 F# forward propagation library to be able to make predictions (not coupled with Cat Vs NonCat, would work with any kind of model structure).
  4. Develop a cross-platform Xamarin Forms 3 application that reference our F# library, and send it pictures to make predictions on.

The full code for the xamarin solution can be downloaded here, the notebooks behind this post and corresponding datasets can be found here.

Let’s get started !

1 – Train our model using TensorFlow

1.1 – Packages

1
2
3
4
5
6
7
8
9
10
11
import math
import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf

import numpy as np
import codecs, json

%matplotlib inline
np.random.seed(1)

1.12 – Dataset

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def load_data():
    train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")
    train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features
    train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels

    test_dataset = h5py.File('datasets/test_catvnoncat.h5', "r")
    test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features
    test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels

    classes = np.array(test_dataset["list_classes"][:]) # the list of classes
   
    train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
    test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
   
    return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
1
train_x_orig, train_y, test_x_orig, test_y, classes = load_data()
1
2
3
4
# Example of a picture
index = 3
plt.imshow(test_x_orig[index])
print ("y = " + str(test_y[0,index]) + ". It's a " + classes[test_y[0,index]].decode("utf-8") +  " picture.")

y = 1. It’s a cat picture.

As usual, you reshape and standardize the images before feeding them to the network. The code is given in the cell below.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Reshape the training and test examples
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T   # The "-1" makes reshape flatten the remaining dimensions
test_x_flatten = test_x_orig.reshape(test_x_orig.shape[0], -1).T

# Standardize data to have feature values between 0 and 1.
train_x = train_x_flatten/255.
test_x = test_x_flatten/255.

print ("train_x's shape: " + str(train_x.shape))
print ("test_x's shape: " + str(test_x.shape))

print ("train_y's shape: " + str(train_y.shape))
print ("test_y's shape: " + str(test_y.shape))

train_x’s shape: (12288, 209)
test_x’s shape: (12288, 50)
train_y’s shape: (1, 209)
test_y’s shape: (1, 50)

1.3 – Define and train our model

We will reuse the same neural network architecture as in the previous article :

  • Input layer of 12288 neurons
  • Hidden layer of 20 neurons
  • Hidden layer of 5 neurons
  • Output layer of 1 neuron (binary classifier)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
class TensorFlowNNModel :
   
    def random_mini_batches(self, X, Y, mini_batch_size = 64, seed = 0):
       
        m = X.shape[1]                  # number of training examples
        mini_batches = []
        np.random.seed(seed)

        # Step 1: Shuffle (X, Y)
        permutation = list(np.random.permutation(m))
        shuffled_X = X[:, permutation]
        shuffled_Y = Y[:, permutation].reshape((Y.shape[0],m))

        # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
        num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
        for k in range(0, num_complete_minibatches):
            mini_batch_X = shuffled_X[:, k * mini_batch_size : k * mini_batch_size + mini_batch_size]
            mini_batch_Y = shuffled_Y[:, k * mini_batch_size : k * mini_batch_size + mini_batch_size]
            mini_batch = (mini_batch_X, mini_batch_Y)
            mini_batches.append(mini_batch)

        # Handling the end case (last mini-batch < mini_batch_size)
        if m % mini_batch_size != 0:
            mini_batch_X = shuffled_X[:, num_complete_minibatches * mini_batch_size : m]
            mini_batch_Y = shuffled_Y[:, num_complete_minibatches * mini_batch_size : m]
            mini_batch = (mini_batch_X, mini_batch_Y)
            mini_batches.append(mini_batch)

        return mini_batches

    def compute_cost(self, Z4, Y):
        logits = tf.transpose(Z4)
        labels = tf.transpose(Y)
        cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = logits, labels = labels))
        return cost
   
    def forward_propagation(self, X, parameters):
        W1 = parameters['W1']
        b1 = parameters['b1']
        W2 = parameters['W2']
        b2 = parameters['b2']
        W3 = parameters['W3']
        b3 = parameters['b3']

        Z1 = tf.add(tf.matmul(W1, X), b1)
        A1 = tf.nn.relu(Z1)
        Z2 = tf.add(tf.matmul(W2, A1), b2)
        A2 = tf.nn.relu(Z2)
        Z3 = tf.add(tf.matmul(W3, A2), b3)

        return Z3
   
    def predict(self, X_pred, parameters):

        tf.reset_default_graph()
       
        X = tf.placeholder(tf.float32, [X_pred.shape[0], None], name="X")
       
        W1 = tf.convert_to_tensor(parameters[0]["W"])
        b1 = tf.convert_to_tensor(parameters[0]["b"])
        W2 = tf.convert_to_tensor(parameters[1]["W"])
        b2 = tf.convert_to_tensor(parameters[1]["b"])
        W3 = tf.convert_to_tensor(parameters[2]["W"])
        b3 = tf.convert_to_tensor(parameters[2]["b"])

        Z3 = tf.add(tf.matmul(W1, X), b1)
       
        params = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2,
                  "W3": W3,
                  "b3": b3}

        with tf.Session() as sess:
            sess.run(tf.global_variables_initializer())
            p = sess.run(tf.sigmoid(self.forward_propagation(X, params)), feed_dict = {X: X_pred})
            p = (p > 0.5) * 1

        return p
   
    def train(self, X_train, Y_train, X_test, Y_test, learning_rate = 0.001,
          num_epochs = 1500, minibatch_size = 32, print_cost = True):
       
        tf.reset_default_graph()
       
        m = X_train.shape[1]
        seed = 1
        costs = []  
       
        #X and Y placeholders for mini batches
        X = tf.placeholder(tf.float32, [X_train.shape[0], None], name="X")
        Y = tf.placeholder(tf.float32, [Y_train.shape[0], None], name="Y")
       
        #Weights and bias init
        W1 = tf.get_variable("W1", [20,12288], initializer = tf.contrib.layers.xavier_initializer())
        b1 = tf.get_variable("b1", [20,1], initializer = tf.zeros_initializer())
        W2 = tf.get_variable("W2", [5,20], initializer = tf.contrib.layers.xavier_initializer())
        b2 = tf.get_variable("b2", [5,1], initializer = tf.zeros_initializer())
        W3 = tf.get_variable("W3", [1,5], initializer = tf.contrib.layers.xavier_initializer())
        b3 = tf.get_variable("b3", [1,1], initializer = tf.zeros_initializer())

        parameters = {"W1": W1,
                      "b1": b1,
                      "W2": W2,
                      "b2": b2,
                      "W3": W3,
                      "b3": b3}
   
        Z3 = self.forward_propagation(X, parameters)
        cost = self.compute_cost(Z3, Y)
        optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost)

        with tf.Session() as sess:
           
            sess.run(tf.global_variables_initializer())
           
            # Do the training loop
            for epoch in range(num_epochs):
                epoch_cost = 0.
                num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set
                seed = seed + 1
                minibatches = self.random_mini_batches(X_train, Y_train, minibatch_size, seed)

                for minibatch in minibatches:
                    # Select a minibatch
                    (minibatch_X, minibatch_Y) = minibatch
                    _ , minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})
                    epoch_cost += minibatch_cost / num_minibatches

                # Print the cost every epoch
                if print_cost == True and epoch % 100 == 0:
                    print ("Cost after epoch %i: %f" % (epoch, epoch_cost))
                if print_cost == True and epoch % 5 == 0:
                    costs.append(epoch_cost)
           
            y_pred = sess.run(tf.sigmoid(self.forward_propagation(X, parameters)), feed_dict = {X: X_train})
            y_pred = (y_pred > 0.5) * 1
            print("train accuracy: {} %".format(100 - np.mean(np.abs(y_pred - Y_train)) * 100))
           
            y_pred = sess.run(tf.sigmoid(self.forward_propagation(X, parameters)), feed_dict = {X: X_test})
            y_pred = (y_pred > 0.5) * 1
            print("test accuracy: {} %".format(100 - np.mean(np.abs(y_pred - Y_test)) * 100))
           
            learn_parameters = [
                                    { "W": sess.run(W1).tolist(), "b": sess.run(b1).tolist(), "A": "relu"},
                                    { "W": sess.run(W2).tolist(), "b": sess.run(b2).tolist(), "A": "relu"},
                                    { "W": sess.run(W3).tolist(), "b": sess.run(b3).tolist(), "A": "sigmoid"}
                                ]
           
            json.dump(learn_parameters, codecs.open('export/weights.json', 'w', encoding='utf-8'), separators=(',', ':'), sort_keys=False, indent=4)
            return (learn_parameters, costs)
1
2
3
learning_rate = 0.0001
first_model = TensorFlowNNModel()
learned_params, first_model_costs = first_model.train(train_x, train_y, test_x, test_y, learning_rate, 1000, 32)

Cost after epoch 0: 0.807752
Cost after epoch 100: 0.255916
Cost after epoch 200: 0.073060
Cost after epoch 300: 0.022296
Cost after epoch 400: 0.009248
Cost after epoch 500: 0.004866
Cost after epoch 600: 0.002655
Cost after epoch 700: 0.001602
Cost after epoch 800: 0.000898
Cost after epoch 900: 0.000508
train accuracy: 100.0 %
test accuracy: 72.0 %

1
2
3
4
5
6
# plot the cost
plt.plot(np.squeeze(first_model_costs))
plt.ylabel('cost')
plt.xlabel('iterations (per tens)')
plt.title("Learning rate =" + str(learning_rate))
plt.show()

2 – Implement F# Forward Propagation

In this section we will load the trained weights in an F# function, write a simple forward propagation function, then include it in a Xamarin compatible F# dll so we can use our trained model in a mobile app.

2.1 – Forward Propagation Parameters

1
2
3
4
5
 type public DenseNnParameters = {
    W: Matrix<float>;
    b: Vector<float>;
    A: Matrix<float> -> Matrix<float>
}

2.2 – Basic Matrix / Vector Operations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
let AddVector (m:Matrix<float>) (v:Vector<float>) =
    let newColumns = seq {
                            for i in 0..(m.ColumnCount) - 1 do
                                yield m.Column(i).Map(fun x -> x + v.[i])
                        }
    (newColumns |> DenseMatrix.OfRowVectors).Clone()


let MultiplyByMatrix (m1:Matrix<float>) (m2:Matrix<float>) =
    let newColumns = seq {
                            for i in 0..(m2.ColumnCount) - 1 do
                                yield m1.LeftMultiply (m2.Column(i))
                        }
    (newColumns |> DenseMatrix.OfRowVectors).Clone()

2.3 – Activation Functions

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module public ActivationFunctions =

        let sigmoid (z:Matrix<float>) =

            printfn "prediction z value is  %s" (string z)

            z.Map(fun x -> -x)
                |> (fun x -> x.PointwiseExp())
                |> (fun x -> 1.0 + x)
                |> (fun x -> 1.0 / x)


        let relu (z:Matrix<float>) =
            let other_matrix = DenseMatrix.Build.Dense(fst (MatrixVector.Matrix z).shape, snd (MatrixVector.Matrix z).shape)
            z.Map2((fun x y -> Math.Max(x, y)), other_matrix)

2.4 – Put it together : Forward Model

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
type public DenseNnModel (parameters:list<DenseNnParameters>) =

    member val parameters = parameters

    member self.Predict(x:Matrix<float>) =

        let rec forwardProp (outputPrev:Matrix<float>) (parameters:list<DenseNnParameters>) =
            match parameters with
                    | [] -> outputPrev.Map (fun x -> if x > 0.5 then 1.0 else 0.0)
                    | hd::tail ->
                                let Z = (MultiplyByMatrix hd.W outputPrev) |> (fun x -> AddVector x hd.b)  
                                let A = hd.A Z
                                forwardProp A tail

        forwardProp x self.parameters

2.5 – Helper Function : Load Trained Weights

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
let CreateModelFromTrainedWeights jsonFileStream =

            let trainedValues = JsonValue.Load(stream = jsonFileStream).AsArray()

            let getValues values =
                seq {
                    for param in values do
                        yield {
                                W = param?W.AsArray()
                                        |> Array.map (fun x -> (x.AsArray() |> Array.map (fun y ->  y.AsFloat())))
                                        |> DenseMatrix.OfColumnArrays

                                b = param?b.AsArray()
                                        |> Array.map (fun x -> (x.AsArray() |> Array.map (fun y ->  y.AsFloat() )))
                                        |> DenseMatrix.OfColumnArrays
                                        |> (fun x -> DenseVector.OfVector(x.Transpose().Column(0)))

                                A = match param?A.AsString() with
                                        | "sigmoid" ->  ActivationFunctions.sigmoid
                                        | "relu" ->  ActivationFunctions.relu
                                        | _ -> failwith "unknown activation function %s." (param?A.AsString())
                                }
                    }

            let parameters = getValues trainedValues
            new DenseNnModel(parameters |> List.ofSeq)

3 – Use our Model In Xamarin Forms App

This section describes the key part only on how to develop a Xamarin Forms app that will work with our F# model

The full solution code can be found on my personal github here: https://github.com/mathieu-clerici/Blog_Articles_Code/tree/master/04_TensorflowModel_Forward_Prop_Xamarin/xamarin_code

3.1 – Solution setup

Our solution will consist of a Xamarin forms 3 project composed of a .NetStandard 2.0 shared core project, plus corresponding Xamarin iOS and Xamarin Android projects. In addition, we will add the F# forward propagation Library we developed in the previous section.

Our core project will contain a Models folder where we will put our json exported model developped in the first section, the XAML pages definitions, and a Dependencies folder where we will define a service interface to be implemented by platform to extract RGB values of an image, and represent it as normalized vector.

As we want to embed the weights.json file within our dll, we have to make sure it is has its build properties set to embed resource :

3.2 – Dependency Service Definition And Implementations

We need to define a generic service interface that would take the picture file and format it in the expected input format for our Cat vs NonCat model. Then we need to implement for each supported platform using the native APIs.

1
2
3
4
public interface IImageToVectorService
{
    double[] ConvertImageToVector(string imagePath);
}

Android Implementation :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[assembly: Dependency(typeof(ImageToVectorService))]
namespace CatVsNonCat.Droid.Dependencies
{
    public class ImageToVectorService : IImageToVectorService
    {
        public ImageToVectorService()
        {
        }
       
        public double[] ConvertImageToVector(string imagePath)
        {
            using (var bitmap = Android.Graphics.BitmapFactory.DecodeFile(imagePath))
            {
                var pixels = new List<double>();

                for (var i = 0; i < 64; i++)
                {
                    for (var j = 0; j < 64; j++)
                    {
                        var colour = bitmap.GetPixel(i, j);                  
                        var r = colour >> 16 & 0xff;
                        var g = colour >> 8 & 0xff;
                        var b = colour & 0xff;

                        pixels.Add(r / 255);
                        pixels.Add(g / 255);
                        pixels.Add(b / 255);
                    }
                }
                return pixels.ToArray();
            }
        }
    }
}

iOS Implementation :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
[assembly: Dependency(typeof(ImageToVectorService))]
namespace CatVsNonCat.Droid.Dependencies
{
    public class ImageToVectorService : IImageToVectorService
    {
        public ImageToVectorService()
        {
        }
       
        public double[] ConvertImageToVector(string imagePath)
        {        
            using (var image = UIImage.FromFile(imagePath))
            {            
                var pixels = new List<double>();

                for (var i = 0; i < 64; i++)
                {
                    for (var j = 0; j < 64; j++)
                    {
                        GetPixelColor(new PointF((float)i, (float)j), image, pixels);
                    }
                }
                return pixels.ToArray();
            }
        }
       
        private void GetPixelColor(PointF myPoint, UIImage myImage, List<double> pixels)
        {
            var rawData = new byte[3];
            var handle = GCHandle.Alloc(rawData);
            try
            {
                using (var colorSpace = CGColorSpace.CreateDeviceRGB())
                {
                    using (var context = new CGBitmapContext(rawData, 1, 1, 8, 4, colorSpace, CGImageAlphaInfo.PremultipliedLast))
                    {
                        context.DrawImage(new RectangleF(-myPoint.X, (float)(myPoint.Y - myImage.Size.Height), (float)myImage.Size.Width, (float)myImage.Size.Height), myImage.CGImage);
                        pixels.Add( (rawData[0]) / 255.0f);
                        pixels.Add( (rawData[1]) / 255.0f);
                        pixels.Add( (rawData[2]) / 255.0f);
                    }
                }
            }
            finally
            {
                handle.Free();
            }
        }
    }
}

3.3 – Instantiate our F# model from C# and use it

We instantiante the model by providing the json model oponed stream to our helper defined in the previous section, and we keep the instance as a readonly static property in our application class, for ease of use.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[assembly: XamlCompilation(XamlCompilationOptions.Compile)]
namespace CatVsNonCat
{
    public partial class App : Application
    {
        public static DenseNnModel CatVsNonCatModel { get; private set; }

        public App()
        {
            InitializeComponent();

            MainPage = new Pages.ImageRecognitionPage();
        }
       
        protected override async void OnStart()
        {        
            var assembly = Assembly.GetExecutingAssembly();
            var resourceName = "CatVsNonCat.Models.weights.json";

            using (var stream = assembly.GetManifestResourceStream(resourceName))
            {
                CatVsNonCatModel = DenseNnHelper.CreateModelFromTrainedWeights(stream);
            }

            await CrossMedia.Current.Initialize();
        }

        protected override void OnSleep()
        {
            // Handle when your app sleeps
        }

        protected override void OnResume()
        {
            // Handle when your app resumes
        }
    }
}

Now that we have our service to format a picture to the correct input for our model, and our model instance, we are ready to use them to make prediction on pictures ! Here is how we do it :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
private async Task PickPhotoAsync()
{
    var file = await CrossMedia.Current.PickPhotoAsync();
    PicturePath = file.Path;

    var toVectorService = DependencyService.Get<Dependencies.IImageToVectorService>();
    var vector = toVectorService.ConvertImageToVector(PicturePath);
    var toPredict = Matrix<double>.Build.DenseOfColumnArrays(vector);
    var preditionResult = App.CatVsNonCatModel.Predict(toPredict);
   
    if (preditionResult[0,0] == ((double)1.0f))
    {
        await _page.DisplayAlert("Forward Prop Result", "it's a cat !", "OK");
    }
    else
    {
        await _page.DisplayAlert("Forward Prop Result", "it's not a cat !", "OK");
    }
}

Leave a Reply

Your email address will not be published. Required fields are marked *